In the Cross-Lingual Lexical Substitution task, an-notators and systems had to find an alternative substitute word or phrase in Spanish for an En­glish target word in context. The task is based on the English Lexical Substitution task run at SemEval-2007, where both target words and sub­stitutes were in English.

An automatic system for cross-lingual lexical substitution would be useful for a number of ap­plications. For instance, such a system could be used to assist human translators in their work, by providing a number of correct translations that the human translator can choose from. Similarly, the system could be used to assist language learners, by providing them with the interpretation of the unknown words in a text written in the language they are learning. Last but not least, the output of a cross-lingual lexical substitution system could be used as input to existing systems for cross-language information retrieval or automatic ma­chine translation.

While there has been a lot of discussion on the rel­evant sense distinctions for monolingual WSD sys­tems, for machine translation applications there is a consensus that the relevant sense distinctions are those that reflect different translations. One early and notable work was the Senseval-2 Japanese Translation task (Kurohashi, 2001) that obtained alternative translation records of typical usages of a test word, also referred to as a translation mem­ory. Systems could either select the most appro­priate translation memory record for each instance and were scored against a gold-standard set of an­notations, or they could provide a translation that was scored by translation experts after the results were submitted. In contrast to this work, in our task we provided actual translations for target in­stances in advance, rather than predetermine trans­lations using lexicographers or rely on post-hoc evaluation, which does not permit evaluation of new systems after the competition.

Previous standalone WSD tasks based on par­allel data have obtained distinct translations for senses as listed in a dictionary (Ng and Chan, 2007). In this way fine-grained senses with the same translations can be lumped together, how­ever this does not fully allow for the fact that some senses for the same words may have some transla­tions in common but also others that are not (Sinha et al., 2009).

In our task, we collected a dataset which al­lows instances of the same word to have some translations in common, while not necessitating a clustering of translations from a specific re­source into senses (in comparison to Lefever and Hoste (2010)). Resnik and Yarowsky (2000) also

'Though in that task note that it is possible for a transla­tion to occur in more than one cluster. It will be interesting to

conducted experiments using words in context, rather than a predefined sense-inventory however in these experiments the annotators were asked for a single preferred translation. In our case, we al­lowed annotators to supply as many translations as they felt were equally valid. This allows us to examine more subtle relationships between us­ages and to allow partial credit to systems that get a close approximation to the annotators' trans­lations. Unlike a full blown machine translation task (Carpuat and Wu, 2007), annotators and sys­tems are not required to translate the whole context but just the target word.

The English Lexical substitution task (hereafter referred to as lexsub) was run at SemEval-2007 (McCarthy andNavigli, 2007; McCarthy and Navigli, 2009). lexsub was proposed as a task which, while requiring contextual disambiguation, did not presuppose a specific sense inventory. In fact, it is quite possible to use alternative rep­resentations of meaning, such as those proposed by Schütze (1998) and Pantel and Lin (2002).

The motivation for a substitution task was that it would reflect capabilities that might be useful for natural language processing tasks such as para­phrasing and textual entailment, while not requir­ing a complete system that might mask system ca­pabilities at a lexical level and make participation in the task difficult for small research teams.

The task required systems to produce a substi­tute word for a word in context. The data was collected for 201 words from open class parts-of-speech (PoS) (i.e. nouns, verbs, adjectives and ad­verbs). Words were selected that have more than one meaning with at least one near synonym. Ten sentences for each word were extracted from the English Internet Corpus (Sharoff, 2006). There were five annotators who annotated each target word as it occurred in the context of a sentence. The annotators were each allowed to provide up to three substitutes, though they could also provide a NIL response if they could not come up with a substitute. They had to indicate if the target word was an integral part of a multiword.

see the extent that this actually occurred in their data and the extent that the translations that our annotators provided might be clustered.

The Cross-Lingual Lexical Substitution task fol­lows lexsub except that the annotations are translations rather than paraphrases. Given a tar­get word in context, the task is to provide several correct translations for that word in a given lan­guage. We used English as the source language and Spanish as the target language.

We provided both development and test sets, but no training data. As for lexsub, any systems re­quiring training data had to obtain it from other sources. We included nouns, verbs, adjectives and adverbs in both development and test data. We used the same set of 30 development words as in lexsub, and a subset of 100 words from the lex­sub test set, selected so that they exhibit a wide variety of substitutes. For each word, the same ex­ample sentences were used as in lexsub.

To place results in perspective, several baselines as well as the upper bound were calculated.

Nine teams participated in the task, and several of them entered two systems. The systems used various resources, including bilingual dictionar­ies, parallel corpora such as Europarl or corpora built from Wikipedia, monolingual corpora such as Web IT or newswire collections, and transla­tion software such as Moses, GIZA or Google. Some systems attempted to select the substitutes on the English side, using a lexical substitu­tion framework or word sense disambiguation, whereas some systems made the selection on the Spanish side using lexical substitution in Spanish.

In the following, we briefly describe each par­ticipating system.

cu-smt relies on a phrase-based statistical ma­chine translation system, trained on the Europarl English-Spanish parallel corpora.

The UvT-v and UvT-g systems make use of k-nearest neighbour classifiers to build one word ex­pert for each target word, and select translations on the basis of a GIZA alignment of the Europarl parallel corpus.

The uba-t and uba-w systems both use can­didates from Google dictionary, SpanishDict.com and Babylon, which are then confirmed using par­allel texts, uba-t relies on the automatic trans­lation of the source sentence using the Google Translation API, combined with several heuristics. The uba-w system uses a parallel corpus auto­matically constructed from DBpedia.

swat-E and swat-S use a lexical substitution framework applied to either English or Spanish. The swat-E system first performs lexical substitution in English, and then each substitute is translated into Spanish, swat-S translates the source sentences into Spanish, identifies the Span­ish word corresponding to the target word, and then it performs lexical substitution in Spanish.

TYO uses an English monolingual substitution module, and then it translates the substitution can­didates into Spanish using the Freedict and the Google English-Spanish dictionary.

FCC-LS uses the probability of a word to be translated into a candidate based on estimates ob­tained from the GIZA alignment of the Europarl corpus. These translations are subsequently fil­tered to include only those that appear in a trans­lation of the target word using Google translate.

WLVusp determines candidates using the best N translations of the test sentences obtained with the Moses system, which are further filtered us­ing an English-Spanish dictionary. uspWLV uses candidates from an alignment of Europarl, which are then selected using various features and a clas­sifier tuned on the development data.

IRST-1 generates the best substitute using a PoS constrained alignment of Moses translations of the source sentences, with a back-off to a bilingual dictionary. For out-of-ten, dictionary translations are filtered using the LSA similarity between can­didates and the sentence translation into Spanish. iRSTbs is intended as a baseline, and it uses only the PoS constrained Moses translation for best, and the dictionary translations for out-of-ten.

colEur and colslm use a supervised word sense disambiguation algorithm to distinguish between senses in the English source sentences. Trans­lations are then assigned by using GIZA align­ments from a parallel corpus, collected for the word senses of interest.

Tables 1 and 2 show the precision P and recall R for the best and out-of-ten tasks respectively, for normal and mode. The rows are ordered by R. The out-of-ten systems were allowed to pro­vide up to 10 substitutes and did not have any ad­vantage by providing less. Since duplicates were allowed so that a system can put more emphasis on items it is more confident of, this means that out-of-ten R and P scores might exceed 100% because the credit for each of the human answers is used for each of the duplicates (McCarthy and Navigli, 2009). Duplicates will not help the mode scores, and can be detrimental as valuable guesses which would not be penalised are taken up with duplicates. In table 2, in the column marked dups, we display the number of test items for which at least one duplicate answer was provided. Al­though systems were perfectly free to use dupli­cates, some may not have realised this. Dupli­cates help when a system is fairly confident of a subset of its 10 answers.

We had anticipated a practical issue to come up with all participants, which is the issue of different character encodings, especially when using bilin­gual dictionaries from the Web. While we were counting on the participants to clean their files and provide us with clean characters only, we ended up with result files following different encodings (e.g, UTF-8, ANSI), some of them including diacrit­ics, and some of them containing malformed char­acters. We were able to perform a basic cleaning of the files, and transform the diacritics into their diacriticless counterparts, however it was not pos­sible to clean all the malformed characters without a significant manual effort that was not possible due to time constraints. As a result, a few of the participants ended up losing a few points because their translations, while being correct, contained an invalid, malformed character that was not rec­ognized as correct by the scorer.

There is some variation in rank order of the sys­tems depending on which measures are used.

Table 2: out-of-ten results uba-t has the highest ranking on R for best. us­pWLV is best at finding the mode, for best how­ever the uba-w and uba-t systems (particularly the former) both have exceptional performance for finding the mode in the out-of-ten task, though note that swat-S performs competitively given that its duplicate responses will reduce its chances on this metric. swat-E is the best system for out-of-ten, as several of the items that were empha­sized through duplication were also correct.

The results are much higher than for lex­sub (McCarthy and Navigli, 2007). There are sev­eral possible causes for this. It is perhaps easier for humans, and machines to come up with trans­lations compared to paraphrases. Though the ITA figures are comparable on both tasks, our task con­tained only a subset of the data in lexsub and we specifically avoided data where the lexsub an­notators had not been able to come up with a sub­stitute or had labelled the instance as a name e.g. measurements such as pound, yard or terms such as mad in mad cow disease. Another reason for this difference may be that there are many parallel corpora available for training a system for this task whereas that was not the case for lexsub.

In this paper we described the SemEval-2010 cross-lingual lexical substitution task, including the motivation behind the task, the annotation pro­cess and the scoring system, as well as the partic­ipating systems. Nine different teams with a total

systems typically supplied answers for most items. However, IRST-1 and IRSTbs did considerably better on precision com­pared to recall since they did not cover all test items.

of 15 different systems participated in the task, us­ing a variety of resources and approaches. Com­parative evaluations using different metrics helped determine what works well for the selection of cross-lingual lexical substitutes.

The work of the first and second authors has been partially supported by a National Science Foundation CAREER award #0747340. The work of the third author has been supported by a Royal Society UK Dorothy Hodgkin Fellowship. The authors are grateful to Samer Hassan for his help with the annotation interface.