Word sense disambiguation (wsd) has been de­scribed as a task in need of an application. Whilst researchers believe that it will ultimately prove use­ful for applications which need some degree of se­mantic interpretation, the jury is still out on this point. One problem is that wsd systems have been tested on fine-grained inventories, rendering the task harder than it need be for many applications (Ide and Wilks, 2006). Another significant problem is that there is no clear choice of inventory for any given task (other than the use of a parallel corpus for a specific language pair for a machine translation application).

The lexical substitution task follows on from some previous ideas (McCarthy, 2002) to exam­ine the capabilities of wsd systems built by re­searchers on a task which has potential for NLP applications. Finding alternative words that can occur in given contexts would potentially be useful to many applications such as question answer­ing, summarisation, paraphrase acquisition (Dagan et al., 2006), text simplification and lexical acquisi­tion (McCarthy, 2002). Crucially this task does not specify the inventory for use beforehand to avoid bias to one predefined inventory and makes it eas­ier for those using automatically acquired resources to enter the arena. Indeed, since the systems in SemEval did not know the candidate substitutes for a word before hand, the lexical resource is evaluated as much as the context based disambiguation com­ponent.

The task involves a lexical sample of nouns, verbs, adjectives and adverbs. Both annotators and sys­tems select one or more substitutes for the target word in the context of a sentence. The data was selected from the English Internet Corpus of En­glish produced by Sharoff (2006) from the Inter­net (http://corpus.leeds.ac.uk/internet.html). This is a balanced corpus similar in flavour to the BNC, though with less bias to British English, obtained by sampling data from the web. Annotators are not provided with the PoS (noun, verb, adjective or ad­verb) but the systems are. Annotators can provide up to three substitutes but all should be equally as good. They are instructed that they can provide a phrase if they can't think of a good single word sub­stitute. They can also use a slightly more general word if that is close in meaning. There is a "name" response if the target is part of a proper name and "N IL" response if annotators cannot think of a good substitute. The subjects are also asked to identify if they feel the target word is an integral part of a phrase, and what that phrase was. This option was envisaged for evaluation of multiword detec­tion. Annotators did sometimes use it for paraphras­ing a phrase with another phrase. However, for an item to be considered a constituent of a multiword, a majority of at least 2 annotators had to identify the same multiword.

The annotators were 5 native English speakers from the UK. They each annotated the entire dataset. All annotations were semi-automatically lemma-tised (substitutes and identified multiwords) unless the lemmatised version would change the meaning of the substitute or if it was not obvious what the canonical version of the multiword should be.

We have 3 separate subtasks 1) best 2) oot and 3) mw which we describe below. In the equations and results tables that follow we use P for precision, R for recall, and Mode P and Mode R where we calculate precision and recall against the substitute chosen by the majority of annotators, provided that there is a majority.

Let H be the set of annotators, T be the set of test items with 2 or more responses (non NIL or NAME) and hi be the set of responses for an item i G T for annotator h g H.

For each i g T we calculate the mode (mi) i.e. the most frequent response provided that there is a response more frequent than the others. The set of items where there is such a mode is referred to as TM. Let A (and AM) be the set of items from T (or TM) where the system provides at least one sub­stitute. Let ai : i g A (or ai : i g AM) be the set of guesses from the system for item i. For each i we calculate the multiset union (Hi) for all hi for all h g H and for each unique type (res) in Hi will have an associated frequency (freqres) for the num­ber of times it appears in Hi.

For example: Given an item (id 9999) for happy;a supposing the annotators had supplied answers as follows:

annotator responses

mw measures For this measure, a system must identify items where the target is part ofa multiword and what the multiword is. The annotators do not all have linguistics background, they are simply asked if the target is an integral part of a phrase, and if so what the phrase is. Sometimes this option is used by the subjects for paraphrasing a phrase of the sen­tence, but typically it is used when there is a mul­tiword. For scoring, a multiword item is one with a majority vote for the same multiword with more than 1 annotator identifying the multiword.

Let MW be the subset of T for which there is such a multiword response from a majority of at least 2 annotators. Let mwi g MW be the multiword identified by majority vote for item i. Let MWsys be the subset of T for which there is a multiword response from the system and mwsysi be a multiword specified by the system for item i.

detection P =

^mwsysieMWsys 1 mwi CXÏStS at %

detection R =

^2mwsy.Si^MW 1 mwi CXlStS at %

\MW\ identification P =

^mwsysjeMWsys 1 if rnwsysj = mWj \MWsys\

identification R

■imwsysidMW

9 teams registered and 8 participated, and two of these teams (swag and irst) each entered two sys­tems, we distinguish the first and second systems with a 1 and 2 suffix respectively.

The systems all used 1 or more predefined inven­tories. Most used web queries (hit, melb, unt) or web data (Brants and Franz, 2006) (irst2, ku,

swag 1, swag2, us yd, unt) to obtain counts for disambiguation, with some using algorithms to de­rive domain (irst1) or co-occurrence (tor) infor­mation from the BNC. Most systems did not use sense tagged data for disambiguation though melb did use SemCor (Miller et al., 1993b) for filtering in­frequent synonyms and unt used a semi-supervised word sense disambiguation combined with a host of other techniques, including machine translation en­gines.

in tables 1 to 3 we have ordered systems accord­ing to R on the best task, and in tables 4 to 6 ac­cording to R on oot. We show all scores as per­centages i.e. we multiply the scores in section 3 by 100. in tables 3 and 6 we show results using the subset of items which were i) NOT identified as multiwords (nmwt) ii) scored only on non multi­word substitutes from both annotators and systems (i.e. no spaces) (nmws). Unfortunately we do not have space to show the analysis for the man and rand subsets here. Please refer to the task website for these results. We retain the same ordering for the further analysis tables when we look at subsets of the data. Although there are further differences in the systems which would warrant reranking on an individual analysis, since we combined the subanal­yses in one table we keep the order as for 1 and 4 respectively for ease of comparison.

There is some variation in rank order of the sys­tems depending on which measures are used. ku is highest ranking on R for best. unt is best at find­ing the mode, particularly on oot, though it is the most complicated system exploiting a great many knowledge sources and components. irst2 does well at finding the mode in best. The irst2 best R score is lower because it supplied many answers for each item however it achieves the best R score on the oot task. The baselines are outperformed by most systems. The WordNet baseline outperforms those derived from distributional methods. The dis­tributional methods, especially lin, show promising results given that these methods are automatic and

Table 1: best results

Table 2: best baseline results don't require hand-crafted inventories. As yet we haven't combined the baselines with disambiguation methods.

Only HIT attempted the mw task. It outperforms all baselines from WordNet.

We think this task is an interesting one in which to evaluate automatic approaches of capturing lexical meaning. There is an inherent variation in the task because several substitutes may be possible for a given context. This makes the task hard and scoring is less straightforward than a task which has fixed choices. On the other hand, we believe the task taps into human understanding of word meaning and we hope that computers that perform well on this task will have potential in NLP applications. Since a pre-defined inventory is not used, the task allows us to compare lexical resources as well as disambigua­tion techniques without a bias to any predefined in­ventory. it is possible for those interested in disam­biguation to focus on this, rather than the choice of substitutes, by using the union of responses from the annotators in future experiments.

We acknowledge support from the Royal Society UK for fund­ing the annotation for the project, and for a Dorothy Hodgkin

Fellowship to the first author. We also acknowledge support to the second author from INTEROP NoE (508011, 6th EU FP). We thank the annotators for their hard work. We thank Serge Sharoff for the use of his Internet corpus, Julie Weeds for the software we used for producing the distributional similarity baselines and Suzanne Stevenson for suggesting the oot task .