For the most part, the British National Cor­pus (New edition) was used. (The new edition has the advantage that it is available world­wide, so all participants had the opportunity of obtaining it for system training.) Our goal was to match this source, containing British En­glish, with another, of American English. In the event, only limited quantities of corpus data for American English were available without copy­right complications, so the lion's share of the data was from the BNC with a limited quantity from the Wall Street Journal.

In accordance with standard SENSEVAL pro­cedure, the goal was to have 75 + 15n + 6m in­stances for each lexical-sample word, where n is the number of senses the word has and m is the number of multiword expressions that the word is part of (both, of course, relative to a specific lexicon). In practice numbers varied slightly, as instances were deleted because they had the wrong part of speech or were otherwise unusable. See Table 1 for actual numbers of senses, multiwords expressions and instances.

Here lay the biggest contrast with the SENSEVAL-1 task, which had used Oxford University Press's experimental HECTOR lexi­con. This time, in response to popular acclaim, WordNet was used.

Since SENSEVAL was first mooted, in 1997, WordNet-or-not-WordNet has been a recurring theme. In favour was the argument that it was already very widely used, almost a de facto stan­dard. The argument against concerned its sense distinctions. WordNet, like thesauruses but un­like standard dictionaries, is organised around groups of words of similar meanings (synsets), not around words (with their various meanings). This means that the priority for the lexicogra­pher is building coherent synsets rather than the coherent analysis of the various meanings of a particular word. The writer of a thesaurus does not need to pay as much attention to the distinc­tion between two senses of a word, as the writer of a dictionary. Word sense disambiguation is a task which needs clear and well-motivated sense distinctions. In English SENSEVAL-1, Word-Net was not used because of concerns that it did not provide clean enough sense distinctions.

While HECTOR provided good sense distinc­tions, it was unsatisfactory in that it did not cover the whole lexicon so there was no pos­sibility of scaling up. The case for WordNet - that it was already integrated into so much NLP and WSD work - still stood, so the de­cision was made to use WordNet. To guard against cases where WordNet made a distinc­tion between two meanings, but it was not clear what the distinction was, all the words in the lexical sample had their entries reviewed by a

Table 1: Lexical sample: rubric for column headers: Ss=number of fine-grained senses; Mwe = number of multi-word expressions which the word participates in (as bear participates in WordNet headword polar bear); inst = number of instances tagged; ITA = inter-tagger agree­ment (fine-grained).

lexicographer, with a view particularly to merg­ing insufficiently-distinct senses. It was initially unclear how these revisions would relate to the publicly available version of WordNet (at that time, WordNet 1.6). We are very grateful to the Princeton WordNet team (George Miller, Chris­tiane Fellbaum and Randee Tengi) for their help at this point; they agreed to incorporate our proposed revisions into a new version of Word-Net (1.7) which was then made available in time (despite some very tight deadlines) for the SEN­SEVAL competition.

WordNet 1.7 was not available as a complete object at the time of the gold standard pro­duction, in Spring 2001, but the entries for the lexical sample words were fixed at that point. For each lexical sample entry, we produced an HTML version for the lexicographers to work from. In addition to all the relevant infor­mation in WordNet, this had a mnemonic for each sense, so that taggers could use mnemon­ics when doing the tagging, rather than easily-forgotten, easily-confused sense numbers. The mnemonics were selected by a lexicographer.

Once the corpus sources and lexical entries were fixed, work could proceed with the Gold-Standard tagging.

First, a team of three professional lexicogra­phers and fourteen students and others was re­cruited. Recruitment proceeded as follows: an aptitude test was set up on the web. The test involved sense-tagging some corpus instances (taken from SENSEVAL-1, so the gold-standard answers were known). Email postings were made asking interested people to visit the web­site and take the test. All applicants scoring sufficiently well on the test were then offered work, on a piecework basis.

An HTML version of the corpus for a word was prepared. This comprised a series of ten-sentence stretches of text, with one word in the last of the sentences highlighted; that was the word to be sense-tagged. The files were HTML versions of the XML files used for test and train­ing data.

A tagger was emailed the lexical entry and corpus for a word. They then tagged it, and

lrThe tagging was supported by a grant from EPSRC, the UK funding council, under GR/R02337/01 (MATS).

As in all exercises where a gold standard corpus is the goal, it was necessary to have all data tagged by more than one person. The question then arises, how many taggings does each item need? The algorithm adopted here was:

Thus, in simple cases, a minimum of effort was used, but in difficult cases, more opinions were obtained. The number of taggings per items is shown below. Note that the algorithm stops at step 2 if both taggers agree on one tag, or if both taggers agree on two or more tags.

Table 2: Patterns of (dis)agreement for 3-tagger cases. GS = gold standard tagging arising from these human taggings. ";" used as separator where a tagger (or the gold standard) gave mul­tiple tags.

Of the 5032 two-tagger items, in 4688 cases, the taggers agreed on one tag; in 340 cases, on two tags; and in 4 cases, on three tags.

For the 2446 cases which were tagged three times, 136 were cases where all three taggers agreed perfectly (so, had the algorithm been followed to the letter, the item would not have been tagged a third time; such cases were caused by delays in taggers returning answers.) The common patterns amongst the remainder are shown in Table 2.

For the 86 cases with four taggers, half the cases were {A, A, B, C} taggings.

Fine-grained inter-tagger agreement (ITA) figures was calculated using the same scoring algorithm as for the systems. For each pair of taggers tagging an instance, two scores were calculated, one with the one answer as the key, the other with the other. For each instance, scores were normalised so that the maximum score for each corpus instance was one, however many times it had been tagged. The overall ITA was 85.5%. A breakdown by word and by word class is given in Table l.

As argued in (Kilgarriff and Rosenzweig, 2000) (also (Kilgarriff, 1999)) the inter-tagger agreement figure for a gold standard is of less interest than the replicability figure: if a com­pletely different team of taggers used the same methodology to do the same task, what would the agreement level between the two teams' out­puts be? It is the replicability figure, rather than ITA, which defines an upper bound for the task. We have not yet had time to conduct such a study.

The organisation followed standard SENSEVAL procedure. The data was prepared in XML us­ing SENSEVAL DTDs, with the data for each word split in a ration of 2:1 between training and test data. Data distribution, results up­loads, baselines and scoring were handled at UPenn (see paper by Cotton and Edmonds).

Results are presented in the table below. Owing to space constraints, where a team submitted multiple systems with similar results, only the best result is shown. Full results are available at the SENSEVAL website, as are decodings of system names. At the SENSEVAL workshop (5-6 July 2001) it was agreed that there should also be a later deadline (end July 2001) so that 'egregious bugs' could be fixed. In order to hon­our both standard practice in evaluation exer­cises (eg, no extension of deadlines) and also the agreement made at the workshop, both results sets are presented, with later-deadline results marked with (R) as a suffix to the name.

There has not yet been time for an analy­sis of the results. The one comment that does seem pertinent is the contrast with the English-lexical-sample task in SENSEVAL-1. The tasks were organised in similar ways, and some of the systems were improved versions of systems par­ticipating in 1998. Yet the performance of the best systems has, apparently, dropped around 14%. We may well ask, why?

We believe the drop is due to the choice of lexicon. As discussed above, using Word-Net for SENSEVAL has drawbacks. High-

the figures shown, unlike kappa figures, have the merit of being directly comparable with system performance scores.

Table 3: PR=system precision; ATT= percent­age of cases for which an answer was returned ("attempted").

accuracy word sense disambiguation is only pos­sible where the lexicon makes clear and well-motivated sense distinctions, and provides suf­ficient information about the distinctions for the disambiguation algorithm to build on. An im­plication for future WSD research is that it is time to turn our attention from algorithms, to sense distinctions.