Word Sense Disambiguation (WSD) competitions have focused on general domain texts, as attested in previous Senseval and SemEval competitions (Kilgarriff, 2001; Mihalcea et al., 2004; Snyder and Palmer, 2004; Pradhan et al., 2007). Specific domains pose fresh challenges to WSD sys­tems: the context in which the senses occur might change, different domains involve different sense distributions and predominant senses, some words tend to occur in fewer senses in specific domains, the context of the senses might change, and new senses and terms might be involved. Both super­vised and knowledge-based systems are affected by these issues: while the first suffer from differ­ent context and sense priors, the later suffer from lack of coverage of domain-related words and in­formation.

The main goal of this task is to provide a mul­tilingual testbed to evaluate WSD systems when faced with full-texts from a specific domain. All datasets and related information are publicly avail­able from the task websites.

This task was designed in the context of Ky­oto (Vossen et al., 2008), an Asian-European project that develops a community platform for modeling knowledge and finding facts across lan­guages and cultures. The platform operates as a Wiki system with an ontological support that so­cial communities can use to agree on the mean­ing of terms in specific domains of their interest. Kyoto focuses on the environmental domain be­cause it poses interesting challenges for informa­tion sharing, but the techniques and platforms are

!http://xmlgroup.iit.cnr.it/SemEval2 010/ and http://semeval2.fbk.eu/

independent of the application domain.

The paper is structured as follows. We first present the preparation of the data. Section 3 re­views participant systems and Section 4 the re­sults. Finally, Section 5 presents the conclusions.

The data made available to the participants in­cluded the test set proper, and background texts. Participants had one week to work on the test set, but the background texts where provided months earlier.

Eleven participants submitted more than thirty runs (cf. Table 3). The authors classified their runs into supervised (S in the tables, three runs), weakly supervised (WS, four runs), unsupervised (no runs) and knowledge-based (KB, the rest of runs). Only one group used hand-tagged data from the domain, which they produced on their own. We will briefly review each of the participant groups, ordered fol­lowing the rank obtained for English. They all par­ticipated on the English task, with one exception as noted below, so we report their rank in the En­glish task. Please refer to their respective paper in these proceedings for more details.

CFILT: They participated with a domain-specific knowledge-based method based on Hop-field networks (Khapra et al., 2010). They first identify domain-dependant words using the back­ground texts, use a graph based on hyponyms in WordNet, and a breadth-first search to select the most representative synsets within domain. In ad­dition they added manually disambiguated around one hundred examples from the domain as seeds.

The evaluation has been carried out using the stan­dard Senseval/SemEval scorer s cor er 2 as in­cluded in the trial dataset, which computes preci­sion and recall. Table 3 shows the results in each dataset. Note that the main evaluation measure is recall (R). In addition we also report precision (P) and the recall for nouns and verbs. Recall mea­sures are accompanied by a 95% confidence interval calculated using bootstrap resampling pro­cedure (Noreen, 1989). The difference between two systems is deemed to be statistically signifi­cant if there is no overlap between the confidence intervals. We show graphically the results in Fig­ure 1. For instance, the differences between the highest scoring system and the following four sys­tems are not statistically significant. Note that this method of estimating statistical significance might be more strict than other pairwise methods.

We also include the results of two baselines. The random baseline was calculated analytically. The first sense baseline for each language was taken from each wordnet. The first sense baseline in English and Chinese corresponds to the most frequent sense, as estimated from out-of-domain corpora. In Dutch and Italian, it followed the in­tuitions of the lexicographer. Note that we don't have the most frequent sense baseline from the do­main texts, which would surely show higher re­sults (Koeling et al., 2005).

Domain portability and adaptation of NLP com­ponents and Word Sense Disambiguation systems present new challenges. The difficulties found by supervised systems to adapt might change the way we assess the strengths and weaknesses of super­vised and knowledge-based WSD systems. With this paper we have motivated the creation of an all-words test dataset for WSD on the environ­ment domain in several languages, and presented the overall design of this SemEval task.

One of the goals of the exercise was to show that WSD systems could make use of unannotated background corpora to adapt to the domain and improve their results. Although it's early to reach hard conclusions, the results show that in each of the datasets, knowledge-based systems are able to improve their results using background text, and in two datasets the adaptation of knowledge-based systems leads to results over the MFS baseline. The evidence of domain adaptation of supervised systems is weaker, as only one team tried, and the differences with respect to MFS are very small. The best results for English are obtained by a sys­tem that combines a knowledge-based system with some targeted hand-tagging. Regarding the tech­niques used, graph-based methods over WordNet and distributional thesaurus acquisition methods have been used by several teams.

We thank the collaboration of Lawrence Jones-Walters, Amor Torre-Marin (ECNC) and Karin de Boom (WWF), com­piling the test and background documents. This work task is partially funded by the European Commission (KY­OTO ICT-2007-211423), the Spanish Research Department (KNOW-2 TIN2009-14715-C04-01) and the Basque Govern­ment (BERBATEK IE09-262).