Word Sense Disambiguation, the task of se­lecting the correct sense of an ambiguous word in a given context, is a well-researched NLP problem (see for example Agirre and Edmonds (2006) and Navigli (2009)), largely boosted by the various Senseval and SemEval editions. The SemEval-2010 Cross-lingual Word Sense Disambiguation task focuses on two bottle­necks in current WSD research, namely the scarcity of sense inventories and sense-tagged corpora (especially for languages other than English) and the growing tendency to eval­uate the performance of WSD systems in a real application such as machine translation and cross-language information retrieval (see for example Agirre et al. (2007)).

The Cross-lingual WSD task aims at the de­velopment of a multilingual data set to test the feasibility of multilingual WSD. Many studies have already shown the validity of this cross-lingual evidence idea (Gale et al., 1993; Ide et al., 2002; Ng et al., 2003; Apidianaki, 2009), but until now no benchmark data sets have been available. For the SemEval-2010 compe­tition we developed (i) a sense inventory in which the sense distinctions were extracted from the multilingual corpus Europarl and (ii) a data set in which the ambiguous words were annotated with the senses from the mul­tilingual sense inventory. The Cross-Lingual WSD task is a lexical sample task for English nouns, in which the word senses are made up of the translations in five languages, viz. Dutch, French, Italian, Spanish and German. Both the sense inventory and the annotated data set were constructed for a sample of 25 nouns. The data set was divided into a trial set of 5 ambiguous nouns and a test set of 20 nouns. The participants had to automatically deter­mine the contextually appropriate translation for a given English noun in each or a subset of the five target languages. Only translations present in Europarl were considered as valid translations.

The remainder of this article is organized as follows. Section 2 focuses on the task descrip­tion and gives a short overview of the construc­tion of the sense inventory and the annotation of the benchmark data set with the senses from the multilingual sense inventory. Section 3 clarifies the scoring metrics and presents two frequency-based baselines. The participating systems are presented in Section 4, while the results of the task are discussed in Section 5. Section 6 concludes this paper.

Two types of data sets were used in the Cross-lingual WSD task: (a) a parallel corpus on the basis of which the gold standard sense inventory was created and (b) a collection of English sentences containing the lexical sam­ple words annotated with their contextually appropriate translations in five languages.

xhttp://www.statmt.org/europarl/

Below, we provide a short summary of the complete data construction process. For a more detailed description, we refer to Lefever and Hoste (2009; 2010).

The gold standard sense inventory was derived from the Europarl parallel corpus, which is extracted from the proceedings of the European Parliament (Koehn, 2005). We se­lected 6 languages from the 11 European lan­guages represented in the corpus, viz. English (our target language), Dutch, French, Ger­man, Italian and Spanish. All data were al­ready sentence-aligned using a tool based on the Gale and Church (1991) algorithm, which was part of the Europarl corpus. We only con­sidered the 1-1 sentence alignments between English and the five other languages. These sentence alignments were made available to the task participants for the five trial words. The sense inventory extracted from the paral­lel data set (Section 2.2) was used to annotate the sentences in the trial set and the test set, which were extracted from the JRC-ACQUIS Multilingual Parallel Corpus and BNC.

To score the participating systems, we use an evaluation scheme which is inspired by the English lexical substitution task in SemEval 2007 (McCarthy and Navigli, 2007). We per­form both a best result evaluation and a more relaxed evaluation for the top five results. The evaluation is performed using precision and re­call (Prec and Ree in the equations below), and Mode precision (Mp) and Mode recall (Mr), where we calculate precision and re­call against the translation that is preferred by the majority of annotators, provided that one translation is more frequent than the others.

For the precision and recall formula we use the following variables. Let H be the set of annotators, T the set of test items and hi the set of responses for an item % G T for annota-tor h G H. For each ieTwe calculate the mode (m,) which corresponds to the transla­tion with the highest frequency weight. For a detailed overview of the Mp and Mr cal­culations, we refer to McCarthy and Navigli (2007). Let A be the set of items from T (and TM) where the system provides at least one answer and a» : i e A the set of guesses from the system for item i. For each i, we calculate the multiset union (Hi) for all hi for all h £ H and for each unique type (res) in Hi that has an associated frequency (freqres). In order to assign frequency weights to our gold standard translations, we asked our human annotators to indicate their top 3 translations, which en­ables us to also obtain meaningful associated frequencies (freqres) viz. "1" in case a transla­tion is picked by 1 annotator, "2" if picked by two annotators and "3" if chosen by all three annotators.

Best result evaluation For the best re­sult evaluation, systems can propose as many guesses as the system believes are correct, but the resulting score is divided by the number of guesses. In this way, systems that output a lot of guesses are not favoured.

E resea^regres

Out-of-five (Oof) evaluation For the more relaxed evaluation, systems can propose up to five guesses. For this evaluation, the resulting score is not divided by the number of guesses.

E J2 resea.freg res av-itA \W~\ , x

E J2resea.fr eg r

We received sixteen submissions from five dif­ferent participating teams. One group tack­led all five target languages, whereas the other groups focused on four (one team), two (one team) or one (two teams) target language(s). For both the best and the Out-of-five evalua­tion tasks, there were between three and seven participating systems per language.

The OWNS system identifies the nearest neighbors of the test instances from the train­ing data using a pairwise similarity measure (weighted sum of the word overlap and se­mantic overlap between two sentences). They use WordNet similarity measures as an ad­ditional information source, while the other teams merely rely on parallel corpora to ex­tract all lexical information. The UvT-WSD systems use a k-nearest neighbour classifier in the form of one word expert per lemma-Part-of-Speech pair to be disambiguated. The classifier takes as input a variety of local and global context features. Both the FCC-WSD and T3-COLEUR systems use bilingual translation probability tables that are derived from the Europarl corpus. The FCC-WSD system uses a Naive Bayes classifier, while the T3-COLEUR system uses an unsupervised graph-based method. Finally, the UHD sys­tems build for each target word a multilin­gual co-occurrence graph based on the target word's aligned contexts found in parallel cor­pora. The cross-lingual nodes are first linked by translation edges, that are labeled with the translations of the target word in the corre­sponding contexts. The graph is transformed into a minimum spanning tree which is used to select the most relevant words in context to disambiguate a given test instance.

For the system evaluation results, we show precision (Prec), recall (Ree), Mode precision (Mp) and Mode recall (Mr). We ranked all system results according to recall, as was done for the Lexical Substitution task. Table 3 shows the system ranking on the best task, while Table 4 shows the results for the Oof task.

Table 3: Best System Results

Beating the baseline seems to be quite chal­lenging for this WSD task. While the best sys­tems outperform the baseline for the best task,

Table 4: Out-of-five System Results this is not always the case for the Out-of-five task. This is not surprising though, as the Oof baseline contains the five most frequent Eu­roparl translations. As a consequence, these translations usually contain the most frequent translations from different sense clusters, and in addition they also contain the most generic translation that often covers multiple senses of the target word.

The best results are achieved by the UvT-WSD (Spanish, Dutch) and ColEur (French, Italian and German) systems. An interest­ing feature that these systems have in com­mon, is that they extract all lexical informa­tion from the parallel corpus at hand, and do not need any additional data sources. As a consequence, the systems can easily be applied to other languages as well. This is clearly il­lustrated by the ColEur system, that partici­pated for all supported languages, and outper­formed the other systems for three of the five languages.

In general, we notice that Spanish and French have the highest scores, followed by Italian, whereas Dutch and German seem to be more challenging. The same observation can be made for both the Oof and Best results, except for Italian that performs worse than Dutch for the latter. However, given the low participation rate for Italian, we do not have sufficient information to explain this different behaviour on the two tasks. The discrepancy between the performance figures for Spanish and French on the one hand, and German and Dutch on the other hand, seems more readily explicable. A likely explanation could be the number of classes (or translations) the systems have to choose from. As both Dutch and Ger­man are characterized by a rich compound­ing system, these compound translations also result in a higher number of different trans­lations. Figure 1 illustrates this by listing the number of different translations (or classes in the context of WSD) for all trial and test words. As a result, the broader set of trans­lations makes the WSD task, that consists in choosing the most appropriate translation from all possible translations for a given in­stance, more complicated for Dutch and Ger­man.

We believe that the Cross-lingual Word Sense Disambiguation task is an interesting contri­bution to the domain, as it attempts to ad­dress two WSD problems which have received a lot of attention lately, namely (1) the scarcity of hand-crafted sense inventories and sense-tagged corpora and (2) the need to make WSD more suited for practical applications.

The system results lead to the following ob­servations. Firstly, languages which make ex­tensive use of single word compounds seem harder to tackle, which is also reflected in the baseline scores. A possible explanation for this phenomenon could lie in the number of translations the systems have to choose from. Secondly, it is striking that the systems with the highest performance solely rely on paral­lel corpora as a source of information. This would seem very promising for future multi­lingual WSD research; by eliminating the need

- ♦Dutch H French

)K A Spanish X Italian

X German

for external information sources, these sys­tems present a more flexible and language-independent approach to WSD.