The "Multi-Way Classification of Semantic Re­lations Between Pairs of Nominals" task at SemEval-2010 (Hendrickx et al., 2010) consists in: i) selecting from an inventory of nine possi­ble relations the one that most likely holds be­tween two annotated nominals appearing in the in­put sentence, and ii) specifying the order of the nominals as the arguments of the relation. In con­trast with the semantic relations classification task (Task #4) at SemEval-2007 (Girju et al., 2007), which treated each semantic relation separately as a single two-class (positive vs. negative) classifi­cation task, this year's edition of the challenge pre­sented participating systems with a more difficult and realistic multi-way setup, where the relation Other can also be assigned if none of the nine re­lations is suitable for a given sentence. Examples of the possible markable relations are reported in Table l.

The objective of our experiments with the pro­posed task is to develop a Relation Extraction sys­tem based on shallow linguistic processing, taking the most from available thesauri and ontologies. As a first step in this direction, our submitted runs have been obtained by processing the input sen­tences only to lemmatize their terms, and by using WordNet as the sole source of knowledge.

Similar to other approaches (Moldovan and Badulescu, 2009; Beamer et al., 2009), our sys­tem makes use of semantic boundaries extracted from the WordNet IS-A backbone. Such bound­aries {i.e. divisions in the WordNet hierarchy that best generalize over the training examples) are used to define pairs of high-level synsets with high correlation with specific relations. For in­stance, <microorganism#l, happening#l> and <writing#l, consequence#l> are extracted from the training data as valid high-level collocations respectively for the relations Cause-Effect and Message-Topic. Besides exploiting the Word-Net IS-A hierarchy, the system also uses the holo-/meronymy relations, and information de­rived from the WordNet glosses to capture specific relations such as Member-Collection and Product-Producer. In addition, the context surrounding the annotated nominals is represented as a bag-of-words/synonyms to enhance the relation extraction process. Several experiments have been carried out encoding all the information as large sets of binary features (up to ^6200) to train a Bayesian Network classifier available in the Weka toolkit. To capture both the relations and the order of their arguments, training sentences having oppo­site argument directions for the same relation have been handled separately, and assigned to different classes (thus obtaining 18 classes for the nine tar­get relations, plus one for the Other relation).

The following sections overview our experi­ments, describing the features used by the sys­tem (Section 2), and the submitted runs with the achieved results (Section 3). A concluding discus­sion on the results is provided in Section 4.

The system uses two types of boolean features: WordNet features, and context features.

Our participation to the SemEval-2010 Task #8 consisted in four runs, with the best one (FBK_NK-RES1) achieving a Macro-averaged Fl of 68.02% on the test data. For this submis­sion, the overall training and test running times are about 12'30" and 1'30" respectively, on an Intel Core2 Quad 2.66GHz with 4GB RAM.

FBK_NK-RES1. This run has been obtained adopting a conservative approach, trying to min­imize the risk of overfitting the training data. The features used can be summarized as follows:

• Semantic boundary collocations: all the col­locations of <el> and <e2> ancestors oc­curring at least 10 times in the training set (m param.), for at most 3 relations (n param.);

• Holo-/meronymy relations between the anno­tated nominals;

• Glosses: handled at the level of tokens;

• Context features: left, between, and right context windows of size 3-ALL-3 words re­spectively. Number of occurrences: 25 {left), 10 {between), 25 {right).

On the training set, the Bayesian Network classi­fier (trained with 2239 features, and evaluated with 10-fold cross-validation) achieves an Accuracy of 65.62% (5249 correctly classified instances out of 8000), and a Macro Fl of 78.15%.

FBK_NK-RES2. Similar to the first run, but:

• Semantic boundary collocations: m=9, n=3;

• Glosses: handled at the level of lemmas;

• Context features: left, between, and right context windows of size 4-ALL-l words re­spectively (occurrences: 25-10-25).

Based on the observation of system's behaviour on the training data, the objectives of this run were to: i) add more collocations as features, ii) increase the importance of terms appearing in the left con­text, Hi) reduce the importance of terms appearing in the right context, and iv) increase the possibil­ity of matching the nominals with gloss terms by considering their respective lemmas. On the train­ing set, the classifier (trained with 2998 features) achieves 66.92% Accuracy (5353 correctly classi­fied instances), and a Macro Fl of 79.56%.

FBKJVK-RES3. Similar to the second run, but considering the synonyms of the most frequent sense of the words between <el> and <e2>.

The goal of this run was to generalize the con­text between nominals, by considering word lem­mas. On the training set, the classifier (trained with 2998 features) achieves an Accuracy of 64.94% (5195 correctly classified instances), and a Macro Fl of 77.38%.

FBK_NK-RES4. Similar to the second run, but considering semantic boundary collocations oc­curring at least 7 times in the training set (m param.), for at most 3 relations {n param.).

The goal of this run was to further increase the number of collocations used as features. On the training set, the classifier (trained with 6233 fea­tures) achieves achieves 68.12% Accuracy (5449 correct classifications), and 82.24% Macro Fl.

As regards the results on the test set, Table 2 re­ports the scores achieved by each run using differ­ent portions of the training set (1000, 2000, 4000, 8000 examples), while Figure 1 shows the learn­ing curves for each relation of our best run.

As can be seen from Table 2, the results contra­dict our expectations about the effectiveness of our less conservative configurations and, in particular, about the utility of using larger amounts of se­mantic boundary collocations. The performance

Number of Training Examples

decrease from Run2 to Run4 clearly indicates an overfitting problem. Though suitable to model the training data, the additional collocations were not encountered in the test set. This caused a bias to­wards the Other relation, which reduced the over­all performance of the system.

Regarding our best run, Figure 1 shows dif­ferent system's behaviours with the different tar­get relations. For some of them {e.g. Entity-Destination, Cause-Effect) better results are mo­tivated by the fact that they are often triggered by frequent unambiguous word patterns {e.g. "<el>has been moved to a <e2>", "<el> causes <e2>"). Such relations are effectively handled by the context features which, in contrast, are inadequate for those expressed with high lex­ical variability. This is particularly evident with the Other relation, for which the acquired context features poorly discriminate positive from nega­tive examples even on the training set.

For some relations additional evidence is suc­cessfully brought by the WordNet features. For instance, the good results for Member-Collection demonstrate the usefulness of the holo-/meronymy features.

As regards semantic boundary collocations, to check their effectiveness we performed a post-hoc analysis of those used in our best run. Such anal­ysis was done in two ways: i) by counting the number of collocations acquired on the training set for each relation Ri, and ii) by calculating the ambiguity of each Ri's collocation on the training set {i.e. the average number of other relations activated by the collocation). The analysis re­vealed that the top performing relations {Member-Collection, Entity-Destination, Cause-Effect, and Content-Container) are those for which we ac­quired lots of unambiguous collocations. These findings also explain the poor performance on the Instrument-Agency and the Other relation. For Instrument-Agency we extracted the lowest num­ber of collocations, which were also the most am­biguous ones. For the Other relation the high am­biguity of the collocations extracted is not com­pensated by their huge number (around 50% of the total collocations acquired).

In conclusion, considering i) the level of pro­cessing required (only lemmatization), ii) the fact that WordNet is used as the sole source of knowl­edge, and Hi) the many possible solutions left unexplored due to time constraints, our results demonstrate the validity of our approach, de­spite its simplicity. Future research will focus on a better use of semantic boundary colloca­tions, on more refined ways to extract knowledge from WordNet, and on integrating other knowl­edge sources {e.g. SUMO, YAGO, Cyc).

The research leading to these results has received funding from the European Community's Sev­enth Framework Programme (FP7/2007-2013) un­der Grant Agreement n. 248531 (CoSyne project).