Each task organiser chose the lexicon for their task. Notably, WordNet was used for the first time in SENSEVAL. Version 1.7 for the English tasks, and versions of Euro WordNet for Spanish, Italian, and Estonian.

For the lexical sample tasks, the guidelines suggests that words be chosen from different parts of speech, different frequencies in the corpus, and different polysemies (i.e., number of senses). The number of words depended on the available resources. The sample words were kept secret from the wider community until the training data was released; however, the organisers consulted each other so that translations of some of the sample words could be used across tasks.

For the all-words tasks, the guidelines suggest that at least 5000 words of running text be selected, and that all content words be tagged.

For the lexical sample tasks, it was suggested that for each sample word, at least 75+15n corpus instances be chosen, where n is the number of senses of the word. Again, lack of resources might have precluded this much tagged data.

The Gold Standard corpus must be replicable; the goal is to have human taggers agree at least 90% of the time. Thus, at least two human taggers were required to tag every instance of a word. Taggers are allowed to tag with multiple tags and to use special tags for proper names, and unassignable senses. See the papers in this proceedings for more details.

For the evaluation, the corpus had to be divided into a training set and a test set. The training set is a random subset of the Gold Standard corpus, which allows supervised systems to train. Not all tasks supplied training data, so only 'unsupervised' systems could participate (e.g., in the English all-words task -although many systems trained on other corpora such as Semcor). The test set is the rest of the corpus, with tags removed, on which the systems would be evaluated. It was suggested that a 2:1 ratio of training to test data be used. Although somewhat different from what is normally used in machine learning, the committee felt that having more test data would give a more realistic indication of a system's performance (since more varied contexts per word would be tested), and, moreover, unsupervised systems would be less 'short-changed'.

All data sets are now in the public domain (on the SENSEVAL website).

Since some sense inventories are two fine­grained for plausible sense disambiguation, the scoring program can take into account sense hierarchies or sense groupings. Optionally, a task could provide such a grouping of senses, so that choosing any sense within the group or higher in the hierarchy would count towards a system's overall score. For example, the WordNet hierarchy was used for English nouns, whereas a separate 'grouping' was specially constructed for the English verbs (since the verbs do not have a useful hierarchy in WordNet for scoring purposes). See the paper on the English tasks for more detail.

All tasks used a specially defined common data format for encoding the tagged and untagged corpus examples. Specifically, it accommodated the multi-lingual nature of the data by using an XML document type definition which allowed for a flexible mapping from lexical items to their textual instances. Using XML also allowed for arbitrary character encodings in the corpora. The structure was designed so that individual instances of lexical items could be associated with multiple sense tags, and allowed for discontinuous phrasal lexical items. It did not, however allow for multiple phrasal items with overlapping portions in the surface string.

Another requirement was simplicity. This quality would not only facilitate the logistics of designing a task, but would also ease any hand annotation that may have been necessary. As a result, a standoff annotation system was not feasible. This restricted the format in such a way as to limit the feasibility of embedding extant annotation of the corpora and to require that participants use standoff annotation in submitting their answers for reasons of space efficiency.

The use of the common data format simplified many system's participation in multiple tasks, consequently furthering research into the comparison of WSD in different languages.

A schedule was set up for task organisers to prepare and submit their data to the central website, while participants followed a separate, more rigid (and in the end very tight), schedule for downloads and submissions.

Task organisers started preparing their data as far back as September 2000, but the real push occurred in the three months proceeding the competition period.

The competition period ran April 17 - June 18. Within this period, each task had a critical window defined to be the period from when the training data was first made available to the last day for answer submissions to that task. The critical window had to be a minimum of 21 days.

Participants could download and submit answers at any time during the critical window of a particular task, subject to the following constraints. A submission of answers must:

■ not have occurred more than 7 days after downloading the test data,

Data for the tasks was distributed via a website at University of Pennsylvania. Participants were required to register for tasks in order to download the trial, training, and test data for the tasks, and to upload their answers. Each of these operations required authentication via a password chosen at the time of registration. Additionally, timestamps were recorded for each of these operations in order to enforce the timing constraints on a per-participant basis. The system was not secure, as a participant could register multiple times under different names and use the data from the first registration to perform the task at hand. However, there were no signs of security problems in the use of the website.

Use of the distribution center was recommended, not required, of the task organizers. All the tasks with the exception of the Japanese tasks used the distribution center. A nice by-product of this process in concert with the common data format was the development an overarching organization of all the SENSEVAL data, which is evident in the data available to the public domain.

The same answer format and scoring program was used for SENSEVAL-2 as was used in the first SENSEVAL.

Systems were allowed to tag a word with as many senses as appropriate, giving probabilities, if desired. If the task had a sense hierarchy or grouping, then fine- and coarse-grained scoring was done. In fine-grained scoring, a system had to give at least one of the Gold Standard senses.

In coarse-grained scoring, all senses in the answer key and in system output are collapsed to their highest parent or group identifier. For sense hierarchies, mixed-grained scoring was also done: a system is given partial credit for choosing a sense that is a parent of the required sense according to Melamed and Resnik's (1997) scheme.

Systems were not required to tag all instances of a word, or even all words, thus, as in SENSEVAL-1, we used precision and recall to score the systems, although the metrics are not completely analogous to IR evaluation. Recall (percentage of right answers on all instances in the test set) is the basic measurement of accuracy in this task, because it shows how many correct disambiguations the system achieved overall. Precision (percentage of right answers in the set of answered instances) favours systems that are very accurate if only on a small subset of cases that the system chose to give answers to; the cases might be particularly easy to disambiguate, but this can be determined by comparing the answers to the baseline on the same subset (a type of analysis that has yet to be done). Coverage, the percentage of instances that a system gives any answer to, is also reported. Where available, baseline and inter-tagger agreement numbers are given.

No further data analysis was done. Thus, the question of who 'won' depends on your perspective, but, in fact, that is not the relevant question. The important thing is to examine how each system achieved the performance that it shows. Some of this analysis is given in the papers of this proceedings. (Note that in the results, where appropriate, we distinguished between supervised and unsupervised systems.)

When the results were unveiled at the workshop, it soon became apparent that bugs in the scoring software had potentially affected the results. It was decided by everyone present (on the first day) that all systems should be rescored. Also, owing to the tight schedule, some teams had made serious inadvertent errors in formatting their answers. Thus, it was also agreed that any team could resubmit their (corrected) answers before 31 July 2001. In so doing, the team would have to include an explanation about the modifications and only reasons of 'egregious' bugs would be allowed.