Word sense ambiguity is a potential source for errors in human language technology applications, such as Machine Translation, and it is considered as the great open problem at the lexical level of Natural Language Processing (NLP). There are, however, several computer programs for automatically determining which sense of a word is being used in a given context, according to a variety of semantic, or defining dictionaries as demonstrated in the SENSEVAL-1 exercise; (Kilgarriff and Palmer, 2000). The purpose of SENSEVAL is to be able to say which programs and methods perform better, which worse, which words, or varieties of language, present particular problems to which programs; when modifications improve performance of systems, and how much and what combinations of modifications are optimal. Specifically for Swedish, we would also like to investigate to what extent sense disambiguation can be accomplished and the potential resources available for the task. We would thus be creating a framework that can be shared both within the Yvonne CEDERHOLM

University Box 200, SE-405 30 Göteborg, Sweden Yvonne.Cederholm@ svenska.gu.se exercise and for future evaluation exercises of similar kind, national and international.

Three tasks were identified for SENSEVAL-2, namely: the lexical-sample, the all-words and the 'in a system' tasks. In the lexical sample task, first, we sample the lexicon, then we find instances in context of the sample words and the evaluation is carried out on the sampled instances. In the all-word task a system will be evaluated on its disambiguation performance on every word in the test collection. Finally, in the third type of task, a word sense disambiguation (WSD) system is evaluated on how well it improves the performance of a NL system (MT, IR etc). The reasons we chose the lexical-sample task for Swedish are summarised below:

1. Cost-effectiveness of annotation: it is easier and quicker for the human annotators to sense-tag multiple occurrences of one word at a time, particularly when robust interactive means are utilized (Section 3);

2. The lexical-sample reduces the work of preparing training data since only a subset of the sense inventory is used;

4. The all-words task requires access to a full dictionary, which is problematic from the copyright point of view, since industrial partners were also allowed to participate; and, as Kilgarriff and Palmer (2000) noted:

5. Provided that the sample is well chosen, the lexical sample strategy would be more informative about the current strengths and failings of sense disambiguation research than the all-words task.

In this section we will give a concise description of how the whole exercise (for Swedish) was set up, putting more emphasis on some of the main ingredients of the work, i.e. sampling, resources, annotation and scoring.

A number of likely participants were invited to express their interest and participate in the Swedish SENSEVAL (summer, 2000). A plan for selecting the evaluation material was agreed in Sprâkdata, and human annotators were set on the task of generating the training and testing material. The material was released to the participants at the end of April 2001 and during the second week of June, 2001 the results were returned for scoring. The Swedish SENSEVAL material was divided into three parts and released in stages:

• Trial data: freezing and showing the data formatting conventions (lexicon & corpus);

• Training data: the finalised sense inventory and portion of the 'gold standard';

• Evaluation data: the rest of the 'gold standard', untagged.

The annotation was carried out interactively using a concordance-based interface (developed in SemTag) and which interacts with the corpus and the dictionary; (see

http://svenska.gu.se/-svedk/SENSEVAL/images/semt ag.gif for a Screenshot of this tool). Due to our limited financial resources only two professional lexicographers and a trained Phd student were involved in the tagging process, which was preferred to (untrained) students doing the annotation. High replicability between the human annotators was observed (>95%). The uncertain cases were not used in the training or testing material, while the provided dictionary descriptions for the 40 lemmas were revised (extended and/or modified) prior to their release.

Prior to SENSEVAL, evaluating WSD performance was based solely on the exact match criterion, which is not consider a "fair" metric, and has a lot of drawbacks (e.g. it does not account for the semantic distance between senses when assigning penalties for incorrect labels, and it does not offer a mechanism to offer partial credit; cf. Resnik & Yarowsky (2000)) Instead, in SENSEVAL-2 three scoring policies are adopted:

2. Coarse-grained: answers are mapped to coarse-grained senses and compared to the gold standard tags, also mapped to coarse­grained ones (sense map is required; see below)

3. Mixed-grained: if a sense subsumption hierarchy is available, then the mixedgrained scoring gives some credit to choosing a more coarse-grained sense than the gold standard tag, but not full credit (also using a sense map; see below).

A "sense map" containing a complete list of all sense-ids involved in the evaluation was provided in order to perform the two last types of scoring policies. Each line in the sense map included sense subsumption information and contained a list of the subsumer senses and branching factors.

Five groups showed interest in participating in the Swedish task (eight systems in total). Table 2 provides information for the participating systems, while their average performance is given in Table 3, the score in parenthesis concerns: Verbs/Noun/ Adjectives. All systems returned answers for all instances, thus precision equals recall, all used supervised methods and all systems scored lower on the adjectives and higher on the nouns.

Table2. Participants

System JHU PWE-Vote Sprakdata-ML PWE-Simple PWE-Disj Sprâkdata-

Mixed-Grained Conclusion

The process of WSD is a complex, controversial matter, but relevant for a number of NLP applications. Our contribution to the exercise will eventually sharpen the focus of WSD in Sweden; the material developed in SENSEVAL-2 can be used as benchmark for other researchers that need to measure their system's WSD performance against a concrete reference point (although the dictionary is limited). We think that WSD opens up exciting opportunities for linguistic analysis, contributing with very important information for the assignment of lexical semantic knowledge to polysemous and homonymous content words. The existence of sense ambiguity (polysemy and homonymy) is one of the major problems affecting the usefulness of basic corpus exploration tools. In this respect, we regard WSD as a very important process when it is seen in the context of a wider and deeper NLP system.

We would like to thank the Swedish Council for Research in the Humanities and Social Sciences (HSFR) for providing financial support for the coordination of the task.