It is commonly thought that one of the major obsta­cles to high-performance Word Sense Disambigua­tion (WSD) is the fine granularity of sense inven­tories. State-of-the-art systems attained a disam­biguation accuracy around 65% in the Senseval-3 all-words task (Snyder and Palmer, 2004), where WordNet (Fellbaum, 1998) was adopted as a ref­erence sense inventory. Unfortunately, WordNet is a fine-grained resource, encoding sense distinctions that are difficult to recognize even for human an-notators (Edmonds and Kilgarriff, 2002). Making WSD an enabling technique for end-to-end applica­tions clearly depends on the ability to deal with rea­sonable sense distinctions.

The aim of this task was to explicitly tackle the granularity issue and study the performance of WSD systems on an all-words basis when a coarser set of senses is provided for the target words. Given the need of the NLP community to work on freely available resources, the solution of adopting a dif­ferent computational lexicon is not viable. On the other hand, the production of a coarse-grained sense inventory is not a simple task. The main issue is certainly the subjectivity of sense clusters. To overcome this problem, different strategies can be adopted. For instance, in the OntoNotes project (Hovy et al., 2006) senses are grouped until a 90% inter-annotator agreement is achieved. In contrast, as we describe in this paper, our approach is based on a mapping to a previously existing inventory which encodes sense distinctions at different levels of granularity, thus allowing to induce a sense clus­tering for the mapped senses.

We would like to mention that another SemEval-2007 task dealt with the issue of sense granularity for WSD, namely Task 17 (subtask #1): Coarse­grained English Lexical Sample WSD. In this paper, we report our experience in organizing Task 07.

The task required participating systems to annotate open-class words (i.e. nouns, verbs, adjectives, and adverbs) in a test corpus with the most appropriate sense from a coarse-grained version of the WordNet sense inventory.

We calculated two baselines for the test corpus: a random baseline, in which senses are chosen at random, and the most frequent baseline (MFS), in which we assign the first WordNet sense to each word in the dataset.

Formally, the accuracy of the random baseline was calculated as follows:

1 \T\ 1BLRand it| \CoarseSenses(wi)\

where T is our test corpus, wi is the i-th word instance in T, and CoarseSenses(wi) is the set of coarse senses for wi according to the sense cluster­ing we produced as described in Section 2.2.

The accuracy of the MFS baseline was calculated as:

where ö(wi,k) equals 1 when the k-th sense of word wi belongs to the cluster(s) manually associ­ated by the lexicographer to word wi (0 otherwise). Notice that our calculation of the MFS is based on the frequencies in the SemCor corpus (Miller et al., 1993), as we exploit WordNet sense rankings.

12 teams submitted 14 systems overall (plus two systems from a 13th withdrawn team that we will not report). According to the SemEval policy for task organizers, we remark that the system labelled as UoR-SSl was submitted by the first author (the system is based on the Structural Semantic Inter­connections algorithm (Navigli and Velardi, 2005) with a lexical knowledge base composed by Word­Net and approximately 70,000 relatedness edges). Even though we did not specifically enrich the al­gorithm's knowledge base on the task at hand, we list the system separately from the overall ranking.

The results are shown in Table 3. We calcu­lated a MFS baseline of 78.89% and a random base­line of 52.43%. In Table 4 we report the F1 mea­sures for all systems where we used the MFS as a backoff strategy when no sense assignment was at­tempted (this possibly reranked 6 systems - marked in bold in the table - which did not assign a sense to all word instances in the test set). Compared to previous results on fine-grained evaluation exer­cises (Edmonds and Kilgarriff, 2002; Snyder and palmer, 2004), the systems' results are much higher. on the other hand, the difference in performance between the MFS baseline and state-of-the-art sys­tems (around 5%) on coarse-grained disambiguation is comparable to that of the Senseval-3 all-words ex­ercise. However, given the novelty of the task we believe that systems can achieve even better perfor-

Table 3: System scores sorted by F1 measure (A = attempted, P = precision, R = recall, F1 = F1 mea­sure, t: system from one of the task organizers).

mance by heavily exploiting the coarse nature of the sense inventory.

In Table 5 we report the results for each of the five articles. The interesting aspect of the table is that documents from some domains seem to have predominant senses different from those in Sem-Cor. Specifically, the MFS baseline performs more poorly on documents d004 and d005, from the Computer Science and Biography domains respectively. We believe this is due to the fact that these documents have specific predominant senses, which correspond less often to the most frequent sense in SemCor than for the other three documents. It is also interesting to observe that different systems perform differently on the five documents (we high­light in bold the best performing systems on each article).

Finally, we calculated the systems' performance by part of speech. The results are shown in Table 6. Again, we note that different systems show dif­ferent performance depending on the part-of-speech tag. Another interesting aspect is that the perfor­mance of the MFS baseline is very close to state-of-the-art systems for adjectives and adverbs, whereas it is more than 3 points below for verbs, and around 5 for nouns.

Table 4: System scores sorted by F1 measure with MFS adopted as a backoff strategy when no sense assignment is attempted (t : system from one of the task organizers). Systems affected are marked in bold.

Table 6: System scores by part-of-speech tag (N = nouns, V = verbs, A = adjectives, R = adverbs) sorted by overall F1 measure (best scores are marked in bold,t: system from one of the task organizers).

In order to allow for a critical and comparative in­spection of the system results, we asked the partici­pants to answer some questions about their systems. These included information about whether:

1. the system used semantically-annotated and unannotated resources;

3. the system used the coarse senses provided by the organizers; 4. the system was trained on some corpus.

We believe that this gives interesting information to provide a deeper understanding of the results. We summarize the participants' answers to the question­naires in Table 7. We report about the use of seman­tic resources as well as semantically annotated cor­pora (SC = SemCor, DSo = Defence Science organ­isation Corpus, SE = Senseval corpora, OMWE = Open Mind Word Expert, XWN = eXtended Word­Net, WN = WordNet glosses and/or relations, WND = WordNet Domains), as well as information about the use of unannotated corpora (UC), training (TR), MFS (based on the SemCor sense frequencies), and the coarse senses provided by the organizers (CS). As expected, several systems used lexico-semantic information from the WordNet semantic network and/or were trained on the SemCor semantically-annotated corpus.

Finally, we point out that all the systems perform­ing better than the MFS baseline adopted it as a backoff strategy when they were not able to output a sense assignment.

It is commonly agreed that Word Sense Disambigua­tion needs emerge and show its usefulness in end-to-end applications: after decades of research in the field it is still unclear whether WSD can provide a relevant contribution to real-world applications, such as Information Retrieval, Question Answering, etc. In previous Senseval evaluation exercises, state-of-the-art systems achieved performance far below 70% and even the agreement between human anno-tators was discouraging. As a result of the discus­sion at the Senseval-3 workshop in 2004, one of the aims of SemEval-2007 was to tackle the problems at the roots of WSD. In this task, we dealt with the granularity issue which is a major obstacle to both system and human annotators. In the hope of over­coming the current performance upper bounds, we

Table 7: Information about participating systems (SC = SemCor, DSO = Defence Science Organisation Corpus, SE = Senseval corpora, OMWE = Open Mind Word Expert, XWN = eXtended WordNet, WN = WordNet glosses and/or relations, WND = WordNet Domains, UC = use of unannotated corpora, TR = use of training, MFS = most frequent sense backoff strategy, CS = use of coarse senses from the organizers, t: system from one of the task organizers).

proposed the adoption of a coarse-grained sense in­ventory. We found the results of participating sys­tems interesting and stimulating. However, some questions arise. First, it is unclear whether, given the novelty of the task, systems really achieved the state of the art or can still improve their performance based on a heavier exploitation of coarse- and fine­grained information from the adopted sense inven­tory. We observe that, on a technical domain such as computer science, most supervised systems per­formed worse due to the nature of their training set. Second, we still need to show that coarse senses can be useful in real applications. Third, a full coarse sense inventory is not yet available: this is a major obstacle to large-scale in vivo evaluations. We be­lieve that these aspects deserve further investigation in the years to come.

This work was partially funded by the Interop NoE (508011), 6th European Union FP. We would like to thank Martha Palmer for providing us the first three texts of the test corpus.