The English lexical sample task is the result of a coordinated effort between the University of Pennsylvania, which provided training/test data for the verbs, and Adam Kilgarriff at Brighton, who provided the training/test data for the nouns and adjectives (see Kilgarriff, this issue). In addition, we provided the test data for the English all-words task. The pre-release version of WordNet 1.7 from Princeton was used as the sense inventory. Most of the revisions of sense definitions relevant to the English tasks were done prior to the bulk of the tagging.

The manual annotation for both the English all-words and verb lexical sample tasks was done by researchers and students in linguistics and computational linguistics at the University of Pennsylvania. All of the verbs in both the lex­ical sample and all-words tasks were annotated using a graphical tagging interface that allowed the annotators to tag instances by verb type and view the sentences surrounding the instances. Well over 1000 person hours went into the tag­ging tasks.

The test data for the English all-words task con­sisted of 5,000 words of running text from three Wall Street Journal articles representing varied domains from the Penn Treebank II. Annota­tors preparing the data were allowed to indi-

Christiane Fellbaum is at Princeton University, fell-baum@clarity.princeton.edu

Table 1: System performance on English all-words task (fine-grained scores); (*) indicates system results that were submitted after the senseval-2 workshop and official deadline.

cate at most one multi-word construction for each content word to be tagged, but could give multiple senses for the construction. In some cases, a multi-word construction was annotated with senses associated with just the head word of the phrase in addition to more specific senses based on the entire phrase. The annotations were done under a double-blind scheme by two linguistics students, and were then adjudicated and corrected by a different person.

Task participants were supplied with test data only, in the standard all-words format for senseval-2, as well as the original syntactic and part-of-speech annotations from the Tree-bank. Table 1 shows the system performance on the task. Most of the systems tagged al­most all the content words. This included not only indicating the appropriate sense from the WordNet 1.7 pre-release (as it stood at the time of annotation), but also marking multi-word constructions appropriate to the corresponding sense tags. If given a perfect lemmatizer, a sim­ple baseline strategy which does not attempt to find the satellite words in multi-word construc­tions, but which simply tags each head word with the first WordNet sense for the correspond­ing Treebank part-of-speech tag, would result in precision and recall of about 0.57.

The data for the verb lexical sample task came primarily from the Penn Treebank II Wall Street Journal corpus. However, where that did not supply enough samples to approximate 75+15*n instances per verb, where n is the num­ber of senses for the verb, we supplemented with British National Corpus instances. We did not find sentences for every sense of every word we tagged. We also sometimes found sentences for which none of the available senses were appro­priate, and these were discarded. The instances for each verb were partitioned into training/test data using a ratio of 2:1.

We also grouped the nouns, adjectives and verbs for the lexical sample task, attempting to be explicit about the criteria for each grouping. In particular, the criteria for grouping verbs included differences in semantic classes of ar­guments, differences in the number and type of arguments, whether an argument refers to a created entity or a resultant state, whether an event involves concrete or abstract entities or constitutes a mental act, whether there is a specialized subject domain, etc. All of the verbs were grouped by two or more people, with differences being reconciled. In some cases the groupings of the verbs are identical to the ex­isting WordNet groupings; in some cases they are quite different. The nouns and adjectives were grouped by the primary annotator in the project; WordNet does not have comparable groups for nouns and adjectives.

These groupings were used for coarse-grained scoring, under the framework of S ense val-1.

After the senseval-2 workshop, participant* were invited to retrain their systems on th( groups; only a handful of participants chose tc do this, and in the end the results were uni­formly only slightly better than training on the fine-grained senses with coarse-grained scoring.

Table 2 shows the system performance on just the verbs of the lexical sampk task. For comparison we ran several sim­ple baseline algorithms that had been used in Senseval-1, including RANDOM, COMMON­EST, LESK, LESK-DEFINITION, and LESK-CORPUS (Kilgarriff and Rosenzweig, 2000). In contrast to Senseval-1, in which none of the competing systems performed significantly bet­ter than the highest baseline (LESK-CORPUS), the best-performing systems this time per­formed well above the highest baseline.

Overall, the performance of the systems was much lower than in senseval-1. Several fac­tors may have contributed to this. In addi­tion to the use of fine-grained WordNet senses instead of the smaller Hector sense inventory from senseval-1, most of the verbs included in this task were chosen specifically because we expected them to be difficult to tag. There was also generally less training data made available to the systems (ignoring outliers, there were on average twice as many training samples for each verb in Senseval-1 as there were in Senseval-2). Table 3 shows the correspondence between test data size (half of training data size), en­tropy, and system performance for each verb.

The annotators made every effort to match the target word to a WordNet sense both syntacti­cally and semantically, but sometimes this could not be done. Given a conflict between syntax and semantics, the annotators opted to match semantics. For example, the word "train" has an intransitive sense ("undergo training or in­struction in preparation for a particular role, function, or profession") as well as a related (causative) transitive sense ( "create by training and teaching"). Instances of "train" that were interpreted as having a dropped object were tagged with the transitive sense even though the overt syntax did not match the sense definition.

Some sentences seemed to fit equally well with two different senses, often because of am-

Table 2: System precision (P) and recall (R) for English verb lexical sample task (fine-grained scores); (*) indicates system results that were submitted after the Senseval-2 workshop and official deadline.

biguous context; others did not fit well under any sense. One of the solutions employed in these cases was the assignment of multiple sense tags. The taggers would choose two senses (on rare occasions, even three) that they felt made an approximation of the correct sense when used in combination. Sometimes this strategy was also used in arbitration, when it was decided that neither tagger's tag was better than the other. The taggers tried to use this strategy sparingly and chose single tags whenever possi­ble.

Often, a particular verb yielded multiple in-

Table 3: Test corpus size, entropy (base 2) of tagged data, and average system recall for each verb, using fine-grained and coarse-grained scor­ing.

stances of what was clearly a salient sense, but one not found in WordNet. One of the results was that sentences that should have received a clear sense tag ended up with something rather ad hoc, and often inconsistent. One of the most notorious examples was "call," which had no sense that fit sentences like "The restau­rant is called Marrakesh." WordNet contains some senses related to this one. One sense refers to the bestowing of a name; another to informal designations; another to greetings and vocatives. But there is no sense in WordNet for simply stating something's name without additional connotations, and the gap possibly caused some inconsistencies in the annotation. All these senses belonged to the same group, and if the annotators had been allowed to tag with the more general group sense, there may have been less inconsistency.

It has been well-established that sense-tagging is a very difficult task (Kilgarriff, 1997; Hanks, 2000), even for experienced human tag­gers. If the sense inventory has gaps or redun­dancies, or if some of the sense glosses have ambiguous wordings, choosing the correct sense can be all but impossible. Even if the annotator is working with a very good entry, unforeseen instances of the word always arise.

The degree of polysemy does not affect the relative difficulty of tagging, at least not in the way it is often thought. Very polysemous words, such as "drive," are not necessarily harder to tag than less polysemous words like "replace." The difficulty of tagging depends much more on other aspects of the entry and of the word itself. Often very polysemous words are quite difficult to tag, because they are more likely to be un-derspecified or occur in novel uses; however, "re­place," with four senses, proved a difficult verb to tag, while "play," with thirty-five senses, was relatively straightforward.

In many ways, the grouped senses are very helpful for the sense-tagger. Grouping similar senses allows the sense-tagger to study side-by-side the senses that are perhaps most likely to be confused, which is helpful when the differences between the senses are very subtle. However, it would be a poor idea to attempt to tag a corpus using only the groups, and not the finer sense distinctions, because often some of the senses included in a group will have some properties that the others do not; it is always better to make the finest distinction possible and not just assign the same tag to everything that seems close.

Inter-annotator agreement figures for the hu­man taggers are quite low. However, in some respects they are not quite as low as they seem. Some of the apparent discrepancies were sim­ply the result of a technical error: the annota­tor accidentally picked the wrong tag, perhaps choosing one of its neighbors. Other differences resulted from the sense inventories themselves. Sometimes the taggers interpreted the wording of a given sense definition in different ways, which caused them to choose different tags, but does not entail that they had interpreted the instances differently; in fact, discussion of such cases usually revealed that the taggers had interpreted the instances themselves in the sam way. Additional apparent discrepancies resulte< from the various strategies for dealing with case in which there was no single proper sense h WordNet. This was the case when an instance in the corpus was underspecified so as to al low multiple appropriate interpretations. Thii resulted in (a) multiple tags by one or botl taggers, and (b) each tagger making a differ ent choice. Here, again, the taggers often hac the same interpretation of the instance itself but because the sense inventory was insufficienl for their needs, they were forced to find différent strategies. Sometimes, in fact, one tagger would double-tag a particular instance while the sec­ond tagger chose a single sense that matched one of the two selected by the first annotator. This is considered a discrepancy for statistical purposes, but clearly reflects similar interpreta­tions on the part of the annotators.

In the most recent evaluation, with two new annotators tagging against the Gold Standard, the best fine-grained agreement figures for verbs were in the 70's, similar to Semcor figures. How­ever, when we used the groupings to do a more coarse-grained evaluation, and counted a match between a single tag and a member of a double tag as correct, the human annotator agreement figures rose to 90%.

Support for this work was provided by the National Science Foundation (grants NSF-9800658 and NSF-9910603), DARPA (grant 535626), and the CIA (contract number 2000*SO53100*000). We would also like to thank Joseph Rosenzweig for building the anno­tation tools, and Susanne Wolff for contribution to the manual annotation.