We interpret the task of determining semantic rela­tions between nominals as a classification problem that can be solved, per relation, by machine learning algorithms. We aim at using straightforward features that are easy to compute and relevant to preferably all of the seven relations central to the task.

The starting conditions of the task provide us with a very small amount of training data, which further stresses the need for robust, generalizable features, that generalize beyond surface words. We there­fore hypothesize that generic information on the lex­ical semantics of the entities involved in the rela­tion is crucial. We developed two systems, based on two sources of semantic information. Since the entities in the provided data were word-sense dis-ambiguated, an obvious way to model their lexical semantics was by utilizing WordNet3.0 (Fellbaum, 1998) (WN). One of the systems followed this route.

We also entered a second system, which did not rely on WN but instead made use of automatically generated semantic clusters (Decadt and Daelemans, 2004) to model the semantic classes of the entities.

For both systems we trained seven binary clas­sifiers; one for each relation. From a pool of eas­ily computable features, we selected feature subsets for each classifier in a number of wrapper exper­iments, i.e. repeated cross-validation experiments on the training set to test out subset selections sys­tematically. Along with feature subsets we also chose the machine-learning method independently for each classifier.

Section 2 presents the system description, Sec­tion 3, the results, and Section 4, the conclusions.

The development of the system consists of a prepro­cessing phase to extract the features, and the classi­fication phase.

In Table 3 we first present the best results computed on the training set using 10-fold CV for the clusterbased system (A) and the WN-based system (B). These results are generally higher than the official test set results, shown in Tables 4 and 5, possibly showing a certain amount of overfitting on the train­ing sets.

Table 3: Average accuracy on the training set com­puted in 10-fold CV experiments of the cluster-based system (A) and the WN-based system (B).

The official scores on the test set are computed by the task organizers: accuracy, precision, recall and Fi score. Table 4 presents the results of the cluster-based system. Table 5 presents the results of the WN-based system. (The column Total shows the number of instances in the test set.) Markable is the high accuracy for the part-whole relation as the classifier was only trained on two features cod­ing the WN classes.

Table 4: Test scores for the seven relations of the cluster-based system trained on 140 examples (A4).

The system using all training data with WordNet features, B4 (Table 5), performs better in terms of F-score on six out of the seven subtasks as compared to the system that does not use the WordNet features but the semantic cluster information instead, A4 (Ta­ble 4). This is largely due to a lower precision of the A4 system. The WordNet features appear to be di­rectly responsible for a relatively higher precision.

In contrast, the semantic cluster features of sys­tem A sometimes boost recall. A4's recall on the

cause-effect relation is 97.6% (the classifier pre­dicts the class 'true' for 75 of the 80 examples), and on content-container the system attains 78.9%, markedly better than B4.

We have shown that a machine learning approach us­ing shallow and easily computable features performs quite well on this task. The system using Word­Net features based on the provided disambiguated word senses outperforms the cluster-based system. It would be interesting to compare both systems to a more realistic WN-based system that uses predicted word senses by a Word Sense Disambiguation sys­tem.

However we end by noting that the amount of training and test data in this shared task should be considered too small to base any reliable conclu­sions on. In a realistic scenario (e.g. when high-precision relation classification would be needed as a component of a question-answering system), more training material would have been gathered, and the examples would not have been seeded by a limited number of queries - especially the negative exam­ples are very artificial now due to their similarity to the positive cases, and the fact that they are down-sampled very unrealistically. Rather, the focus of the task should be on detecting positive instances of the relations in vast amounts of text (i.e. vast amounts of implicit negative examples). Positive training exam­ples should be as randomly sampled from raw text as possible. The seven relations are common enough to warrant a focused effort to annotate a reasonable amount of randomly selected text, gathering several hundreds of positive cases of each relation.