The theme of Task 4 is the classification of semantic relations between simple nominals (nouns or base noun phrases) other than named entities - honey bee, for example, shows an instance of the Product-Producer relation. The classification occurs in the context of a sentence in a written English text. Al­gorithms for classifying semantic relations can be applied in information retrieval, information extrac­tion, text summarization, question answering and so on. The recognition of textual entailment (Tatu and Moldovan, 2005) is an example of successful use of this type of deeper analysis in high-end NLP appli­cations.

The literature shows a wide variety of methods of nominal relation classification. They depend as much on the training data as on the domain of ap­plication and the available resources. Rosario and

Hearst (2001) classify noun compounds from the domain of medicine, using 13 classes that describe the semantic relation between the head noun and the modifier in a given noun compound. Rosario et al. (2002) classify noun compounds using the MeSH hierarchy and a multi-level hierarchy of se­mantic relations, with 15 classes at the top level. Nastase and Szpakowicz (2003) present a two-level hierarchy for classifying noun-modifier relations in base noun phrases from general text, with 5 classes at the top and 30 classes at the bottom; other re­searchers (Turney and Littman, 2005; Turney, 2005; Nastase et al., 2006) have used their class scheme and data set. Moldovan et al. (2004) propose a 35-class scheme to classify relations in various phrases; the same scheme has been applied to noun com­pounds and other noun phrases (Girju et al., 2005). Chklovski and Pantel (2004) introduce a 5-class set, designed specifically for characterizing verb-verb semantic relations. Stephens et al. (2001) propose 17 classes targeted to relations between genes. La-pata (2002) presents a binary classification of rela­tions in nominalizations.

There is little consensus on the relation sets and algorithms for analyzing semantic relations, and it seems unlikely that any single scheme could work for all applications. For example, the gene-gene re­lation scheme of Stephens et al. (2001), with rela­tions like Xphosphorylates Y, is unlikely to be trans­ferred easily to general text.

We have created a benchmark data set to allow the evaluation of different semantic relation classifica­tion algorithms. We do not presume to propose a sin­gle classification scheme, however alluring it would

Table 1 : Data set statistics be to try to design a unified standard - it would be likely to have shortcomings just as any of the others we have just reviewed. Instead, we have decided to focus on separate semantic relations that many re­searchers list in their relation sets. We have built an­notated data sets for seven such relations. Every data set supports a separate binary classification task.

Ours is a new evaluation task, so we began with data set creation and annotation guidelines. The data set that Nastase and Szpakowicz (2003) created had re­lation labels and part-of-speech and WordNet sense annotations, to facilitate classification. (Moldovan et al., 2004; Girju et al., 2005) gave the annotators an example of each phrase in a sentence along with WordNet senses and position of arguments. Our annotations include all these, to support a variety of methods (since we work with relations between nominals, the part of speech is always noun). We have used WordNet 3.0 on the Web and sense index tags.

We chose the following semantic relations: Cause-Effect, Content-Container, Instrument-Agency, Origin-Entity, Part-Whole, Product-Producer and Theme-Tool. We wrote seven detailed definitions, including restrictions and conventions, plus prototypical positive and near-miss negative examples. For each relation separately, we based data collection on wild-card search patterns that Google allows. We built the patterns manually, following Hearst (1992) and Nakov and Hearst (2006). Instances of the relation Content-Container, for example, come up in response to queries such as "* contains *", "* holds *", "the * in the *". Fol­lowing the model of the Senseval-3 English Lexical Sample Task, we set out to collect 140 training and at least 70 test examples per relation, so we had a number of different patterns to ensure variety. We also aimed to collect a balanced number of positive and negative examples. The use of heuristic patterns to search for both positive and negative examples should naturally result in negative examples that are near misses. We believe that near misses are more useful for supervised learning than negative examples that are generated randomly.

Figure 1 illustrates the annotations. We tag the nominals, so parsing or chunking is not necessary. For Task 4, we define a nominal as a noun or base noun phrase, excluding names entities. A base noun phrase, e.g., lawn or lawn mower, is a noun with pre-modifiers. We also exclude complex noun phrases (e.g., with attached prepositional phrases - the en­gine of the lawn mower).

The procedure was the same for each relation. One person gathered the sample sentences (aim­ing approximately for a similar number of positive and negative examples) and tagged the entities; two other people annotated the sentences with WordNet senses and classified the relations. The detailed re­lation definitions and the preliminary discussions of positive and negative examples served to maximize the agreement between the annotators. They first classified the data independently, then discussed ev­ery disagreement and looked for consensus. Only the agreed-upon examples went into the data sets. Next, we split each data set into 140 training and no fewer than 70 test examples. (We published the training set for the Content-Container relation as de­velopment data two months before the test set.) Ta­ble 1 shows the number of positive and negative ex"Among the contents of the <e1>vessel</e1> were a set of carpenter's <e2>tools</e2>, sev­eral large storage jars, ceramic utensils, ropes and remnants of food, as well as a heavy load of ballast stones." amples for each relation.

The average inter-annotator agreement on rela­tions (true/false) after the independent annotation step was 70.3%, and the average agreement on WordNet sense labels was 71.9%. In the process of arriving at a consensus between annotators, the def­inition of each relation was revised to cover explic­itly cases where there had been disagreement. We expect that these revised definitions would lead to much higher levels of agreement than the original definitions did.

The task of classifying semantic relations between nominals has attracted the participation of 14 teams who submitted 15 systems. Table 4 lists the sys­tems, the authors and their affiliations, and brief de­scriptions. The systems' performance information in terms of precision, recall, F-measure and accu­racy, macroaveraged over all relations, appears in Table 3. We computed these measures as described in Lewis (1991).

We distinguish four categories of systems based on the type of information used - WordNet senses and/or Google queries:

A - WordNet = NO & Query = NO; B - WordNet = YES & Query = NO; C - WordNet = NO & Query = YES; D - WordNet = YES & Query = YES.

only whether a system uses the WordNet sense la­bels in the data sets. A system may use WordNet internally for varied purposes, but ignore our sense labels; such a system would be in category A or C. Based on the input variation, each submitted system may have up to 4 variations - A,B,C,D.

Table 2 presents three baselines for a relation. Majority always guesses either "true" or "false", whichever is the majority in the test set (maximizes accuracy). Alltrue always guesses "true" (maxi­mizes recall). Probmatch randomly guesses "true" ("false") with the probability matching the distribu­tion of "true" ("false") in the test dataset (balances precision and recall).

We present the results in Table 3 grouped by cat­egory, to facilitate system comparison.

Table 2: Baselines: precision, recall, F-measure and accuracy averaged over the 7 binary classifications.

Systems tagged with * have a Task 4 organizer as part of the team.

Table 3: System performance grouped by category. Precision, recall, F-measure and accuracy macro-averaged over each system's performance on all 7 relations.

The highest average accuracy on Task 4 was 76.3%. Therefore, the average initial agreement between an-notators (70.3%), before revising the definitions, is not an upper bound on the accuracy that can be achieved. That the initial agreement between anno-tators is not a good indicator ofthe accuracy that can be achieved is also supported by the low correlation

Systems tagged with * have a Task 4 organizer as part of the team.

We thank Eneko Agirre, Lluis Marquez and Richard Wicentowski, the organizers of SemEval 2007, for their guidance and prompt support in all organiza­tional matters. We thank Marti Hearst for valu­able advice throughout the task description and de­bates on semantic relation definitions. We thank the anonymous reviewers for their helpful comments.