This technical report describes the UNITN system (a Part-Of-Speech Context Counter) that partici­pated to Semeval 2010 Task 8: Multi-Way Clas­sification of Semantic Relations Between Pairs of Nominals (see Hendrickx et al., 2009). A different version of this system based on Part-Of-Speech counting has been previously used for the auto­matic annotation of three general and separable se­mantic relation classes (taxonomy, location, asso­ciation) obtaining an average Fl-measure of 0.789 for english and 0.781 for italian, see Celli 2010 for details. The organizers of Semeval 2010 Task 8 provided ten different semantic relation types in context, namely:

• Cause-Effect (CE). An event or object leads to an effect. Example: Smoking causes can­cer.

• Instrument-Agency (IA). An agent uses an instrument. Example: Laser printer.

• Product-Producer (PP). A producer causes a product to exist. Example: The growth hor­mone produced by the pituitary gland.

• Content-Container (CC). An object is phys­ically stored in a delineated area of space, the container. Example: The boxes contained books.

• Entity-Origin (EO). An entity is coming or is derived from an origin (e.g., position or material). Example: Letters from foreign countries.

• Entity-Destination (ED). An entity is mov­ing towards a destination. Example: The boy went to bed.

• Component-Whole (CW). An object is a component of a larger whole. Example: My apartment has a large kitchen.

• Member-Collection (MC). A member forms a nonfunctional part of a collection. Exam­ple: There are many trees in the forest.

• Message-Topic (CT). An act of communica­tion, whether written or spoken, is about a topic. Example: The lecture was about se­mantics.

• Other. The entities are related in a way that do not fall under any of the previous men­tioned classes. Example: Batteries stored in a discharged state are susceptible to freezing.

UNITN is a Part-Of-Speech Context Counter. Given as input a plain text with Part-Of-Speech and end-of-sentence markers annotated it outputs a numerical feature vector that gives a representa­tion of a sentence. For Part-Of-Speech and end-of-sentence annotation I used Textpro, a tool for NLP that showed state-of-the-art performance for POS tagging (see Pianta et al, 2008). The POS tagset is the one used in the BNC, described at http : //pie.usna.edu/POScodes.html. Features in the vector can be tailored for specific tasks, in this case 20 features were used in total. They are:

1. Number of prepositions in sentence.

2. Number of nouns and proper names in sen­tence.

3. Number of lexical verbs in sentence.

4. Number of "be" verbs in sentence.

5. Number of "have" verbs in sentence.

6. Number of "do" verbs in sentence.

7. Number of modal verbs in sentence.

8. Number of conjunctions in sentence.

9. Number of adjectives in sentence.

10. Number of determiners in sentence.

11. Number of pronouns in sentence.

12. Number of punctuations in sentence.

13. Number of negative particles in sentence.

14. Number of words in the context between the first and the second entity.

15. Number of verbs in the context between the first and the second entity.

16. patterns (from, in, on, by, of, to).

17. POS of entity 1 (noun, adjective, other).

18. POS of entity 2 (noun, adjective, other).

19. Estimate of entities' position in the relation (el-e2, e2-el,00).

20. Estimate of semantic relation (relations de­scribed in section 1 above).

Prepositional patterns in feature 16 were chosen for their high cooccurrence frequency with a se­mantic relation type and their low cooccurrence with the other ones.

The system works in three steps: in the first one features 1-18 are used for predicting feature 19, in the second one features 1-19 are used for pre­dicting feature 20. In the third step, after the ap­plication of Hall 1998's attribute selection filter (that evaluates the worth of a subset of attributes by considering the individual predictive ability of each feature along with the degree of redundancy between them) features 12, 14, 16, 19 and 20 are used for the prediction of semantic relation plus entities' position (19 relations in total). For all the steps I used C4.5 decision trees (see Quinlan 1993) and Cohen 1995's RIPPER algo­rithm (Repeated Incremental Pruning to Produce Error Reduction). Evaluation for steps 1, 2 and 3 have been run on the training set, with a 10-fold cross-validation, since the test set was relased in a second time. Results of evaluation of step 1, 2 and 3 are reported in table 1 below, chance values (100/number of classes) are taken as baselines, all experiments have been run in Weka (see Witten and Frank, 2005).

I also inverted step 1 and 2 for predicting semanPrediction Baseline average Fl tic relation estimate before entities' position esti­mate and the average Fl-measure is even worse (0.271), demonstrating that entities' position esti­mate has a positive weight on semantic relation es­timate. There are instead some problems with step 2, and I will return on this later in the discussion (section 4).

As it was requested by the task, the system has been run 4 times in the testing phase: the first time (rl) using 1000 examples from the training set for building the model, the second time (r2) 2000 ex­amples, the third (r3) 4000 example and the last one (r4) using the entire training set. The results obtained by UNITN in the competi­tion are not good, overall performance is poor, es­pecially for some relations, in particular Product-Producer and Message-Topic. The best perfor­mance is achieved by the Member-Collection re­lation (47.30% ), that changed from 0% in the first run to 42.71% in the second one. Scores are re­ported, relation by relation, in table 2 below, the discussion follows in section 4.

On the one hand the POSCo system showed an high performance in step 1 (entities' position detection), indicating that the numerical sentence representation obtained by means of Part-Of-Speech can be a good way for extracting syntactic information.

On the other hand the POSCo system proved not to be good for the classification of semantic relations. This clearly indicates that lexical and semantic information is essential in relation extraction. This fact is highlighted also by the attribute selection filter algorithm that choosed, among others, feature 16 (prepositional patterns), which was the only attribute providing lexical information in the system. It is interesting to note that it chose feature 12 (punctuation) and 14 (number of words in the context between the first and the second entity). Punctuation can be used to provide, to a certain level, information about how much the sentence is complex (the higher the number of the punctuation, the higher the subordinated phrases), while feature 14 provides information about the distance between the related entities and this could be useful for the classification between presence or absence of a semantic relation (the longer the distance, the lower the probability to have a relation between entities) but it is useless for a multi-way classification with many semantic relations, like in this case.

In this report we have seen that Part-Of-Speech Counting does not yield good performances in re­lation extraction. Despite this it provides some information about the complexity of the sentence and this can be useful for predicting the position of the entities in the relation. The results confirm the fact that lexical and semantic information is essential in relation extraction, but also that there are some useful non-lexical features, like the com­plexity of the sentence and the distance between the first and the second related entities, that can be used as a complement for systems based on lexical and semantic resources.