Using large-scale knowledge bases, such as Word­Net (Fellbaum, 1998), has become a usual, often necessary, practice for most current Natural Lan­guage Processing (NLP) systems. Even now, build­ing large and rich enough knowledge bases for broad-coverage semantic processing takes a great deal of expensive manual effort involving large re­search groups during long periods of development. In fact, dozens of person-years have been invested in the development of wordnets for various languages (Vossen, 1998). For example, in more than ten years of manual construction (from version 1.5 to 2.1), WordNet passed from 103,445 semantic relations to 245,509 semantic relations. That is, around one thousand new relations per month. But this data does not seems to be rich enough to support ad­vanced concept-based NLP applications directly. It seems that applications will not scale up to work­ing in open domains without more detailed and rich general-purpose (and also domain-specific) seman­tic knowledge built by automatic means.

Fortunately, during the last years, the research community has devised a large set of innovative methods and tools for large-scale automatic acqui­sition of lexical knowledge from structured and un­structured corpora. Among others we can men­tion extended WordNet (Mihalcea and Moldovan, 2001), large collections of semantic preferences ac­quired from SemCor (Agirre and Martinez, 2001; Agirre and Martinez, 2002) or acquired from British National Corpus (BNC) (McCarthy, 2001), large-scale Topic Signatures for each synset acquired from the web (Agirre and de la Calle, 2004) or acquired from the BNC (Cuadros et al., 2005). Obviously, these semantic resources have been acquired using a very different set of methods, tools and corpora, re­sulting on a different set of new semantic relations between synsets (or between synsets and words).

Many international research groups are working on knowledge-based WSD using a wide range of ap­proaches (Mihalcea, 2006). However, less attention has been devoted on analysing the quality of each semantic resource. In fact, each resource presents different volume and accuracy figures (Cuadros et al., 2006).

In this paper, we evaluate those resources on the

SemEval-2007 English Lexical Sample task. For comparison purposes, we also include the results of the same resources on the Senseval-3 English Lex­ical sample task. In both cases, we used only the nominal part of both data sets and we also included some basic baselines.

In order to compare the knowledge resources, all the resources are evaluated as Topic Signatures (TS). That is, word vectors with weights associated to a particular synset. Normally, these word vectors are obtained by collecting from the resource under study the word senses appearing as direct relatives. This simple representation tries to be as neutral as possi­ble with respect to the resources studied.

A common WSD method has been applied to all knowledge resources on the test examples of Senseval-3 and SemEval-2007 English lexical sam­ple tasks. A simple word overlapping counting is performed between the Topic Signature and the test example. The synset having higher overlapping word counts is selected. In fact, this is a very sim­ple WSD method which only considers the topical information around the word to be disambiguated. Finally, we should remark that the results are not skewed (for instance, for resolving ties) by the most frequent sense in WN or any other statistically pre­dicted knowledge.

As an example, table 1 shows a test example of SemEval-2007 corresponding to the first sense of the noun capital. In bold there are the words that appear in its corresponding Topic Signature acquired from the web.

Note that although there are several important related words, the WSD process implements ex­act word form matching (no preprocessing is per­formed).

The evaluation presented here covers a wide range of large-scale semantic resources: WordNet (WN) (Fellbaum, 1998), extended WordNet (Mihalcea and Moldovan, 2001), large collections of seman­tic preferences acquired from SemCor (Agirre and Martinez, 2001; Agirre and Martinez, 2002) or ac­quired from the BNC (McCarthy, 2001), large-scale Topic Signatures for each synset acquired from the web (Agirre and de la Calle, 2004) or SemCor (Lan­des et al., 2006).

Although these resources have been derived us­ing different WN versions, using the technology for the automatic alignment of wordnets (Daude et al., 2003), most of these resources have been integrated into a common resource called Multilingual Cen­tral Repository (MCR) (Atserias et al., 2004) main­taining the compatibility among all the knowledge resources which use a particular WN version as a sense repository. Furthermore, these mappings allow to port the knowledge associated to a particular WN version to the rest of WN versions.

The current version of the MCR contains 934,771 semantic relations between synsets, most of them acquired by automatic means. This represents al­most four times larger than the Princeton WordNet (245,509 unique semantic relations in WordNet 2.1).

Hereinafter we will refer to each semantic re­source as follows:

WN (Fellbaum, 1998): This resource uses the direct relations encoded in WN1.6 or WN2.0 (for instance, tree#n#1-hyponym->teak#n#2). We also tested WN (using relations at distances 1 and 2), WN (using relations at distances 1 to 3) and WN(using relations at distances 1 to 4).

XWN (Mihalcea and Moldovan, 2001): This re­source uses the direct relations encoded in extended WN (for instance, teak#n#2-gloss->wood#n#1).

WN+XWN: This resource uses the direct rela­tions included in WN and xWN. We also tested (WN+XWN) (using either WN or XWN relations at distances 1 and 2, for instance, tree#n# 1-related->wood#n#1).

spBNC (McCarthy, 2001): This resource contains 707,618 selectional preferences acquired for sub­jects and objects from BNC.

spSemCor (Agirre and Martinez, 2002): This re­source contains the selectional preferences acquired for subjects and objects from SemCor (for instance, read#v#1-tobj->book#n#1).

MCR (Atserias et al., 2004): This resource uses the direct relations included in MCR but ex­cluding spBNC because of its poor performance. Thus, MCR contains the direct relations from WN (as tree#n#1-hyponym->teak#n#2), XWN (as teak#n#2-gloss->wood#n#1), and spSemCor (as read#v#1-tobj->book#n#1) but not the indirect relations of (WN+XWN) (tree#n#1-related->wood#n#1). We also tested MCR (using rela­tions at distances 1 and 2), which also integrates (WN+XWN) relations.

Table 6 presents ordered by F1 measure, the perfor­mance of each knowledge resource on Senseval-3 and the average size of the TS per word-sense. The average size of the TS per word-sense is the number of words associated to a synset on average. Obvi­ously, the best resources would be those obtaining better performances with a smaller number of asso­ciated words per synset. The best results for preci­sion, recall and F1 measures are shown in bold. We also mark in italics those resources using non-direct relations.

Surprisingly, the best results are obtained by TSSEM (with F1 of 52.4). The lowest result is ob­tained by the knowledge directly gathered from WN mainly because of its poor coverage (R of 18.4 and F1 of 26.1). Also interesting, is that the knowledge integrated in the MCR although partly derived by automatic means performs much better in terms of precision, recall and F1 measures than using them separately (F1 with 18.4 points higher than WN, 9.1 than XWN and 3.7 than spSemCor).

Despite its small size, the resources derived from SemCor obtain better results than its counterparts using much larger corpora (TSSEM vs. TSWEB and spSemCor vs. spBNC).

Regarding the basic baselines, all knowledge re­sources surpass RANDOM, but none achieves nei­ther WN-MFS, TRAIN-MFS nor TRAIN. Only

Table 6: P, R and F1 fine-grained results for the resources evaluated individually at Senseval-03 En­glish Lexical Sample Task.

TSSEM obtains better results than SEMCOR-MFS and is very close to the most frequent sense of WN (WN-MFS) and the training (TRAIN-MFS).

Table 7 presents ordered by F1 measure, the per­formance of each knowledge resource on SemEval-2007 and its average size of the TS per word-sense. The best results for precision, recall and F1 mea­sures are shown in bold. We also mark in italics those resources using non-direct relations.

Interestingly, on SemEval-2007, all the knowl­edge resources behave differently. Now, the best results are obtained by (WN+XWN) (with F1 of 52.9), followed by TSWEB (with F1 of 51.0). The lowest result is obtained by the knowledge encoded in spBNC mainly because of its poor precision (P of 24.4 and F1 of 20.8).

Regarding the basic baselines, spBNC, WN (and also WN and WN) and spSemCor do not sur­pass RANDOM, and none achieves neither WN-

MFS, TRAIN-MFS nor TRAIN. Now, WN+XWN,

XWN, TSWEB and (WN+XWN) obtain better re­sults than SEMCOR-MFS but far below the most frequent sense of WN (WN-MFS) and the training (TRAIN-MFS).

In order to evaluate deeply the contribution of each knowledge resource, we also provide some results of the combined outcomes of several resources. The

Table 7: P, R and F1 fine-grained results for the resources evaluated individually at SemEval-2007, English Lexical Sample Task .

combinations are performed following a very basic strategy (Brody et al., 2006).

Rank-Based Combination (Rank): Each se­mantic resource provides a ranking of senses of the word to be disambiguated. For each sense, its place­ments according to each of the methods are summed and the sense with the lowest total placement (clos­est to first place) is selected.

Table 8 presents the F1 measure result with re­spect this method when combining four different se­mantic resources on the Senseval-3 test set.

Regarding the basic baselines, this combination outperforms the most frequent sense of SemCor (SEMCOR-MFS with F1 of 49.1), WN (WN-MFS with F1 of 53.0) and, the training data (TRAIN-MFS with F1 of54.5).

Table 9 presents the F1 measure result with re­spect the rank mthod when combining the same four different semantic resources on the SemEval-2007 test set.

Although this task had no participants, we provide the performances of a large set of knowledge re­sources on two different test sets: Senseval-3 and SemEval-2007 English Lexical Sample task. We also provide the results of a system combination of four large-scale semantic resources. When evalu­ated on Senseval-3, the combination of knowledge sources surpass the most-frequent classifiers. How­ever, a completely different behaviour is observed on SemEval-2007 data test. In fact, both corpora present very different characteristics. The results show that some resources seems to be less depen­dant than others to corpus changes.

Obviously, these results suggest that much more research on acquiring, evaluating and using large-scale semantic resources should be addressed.

We want to thank the valuable comments of the anonymous reviewers. This work has been partially

E. Agirre and O. Lopez de la Calle. 2004. Publicly avail­able topic signatures for all wordnet nominal senses. In Proceedings ofLREC, Lisbon, Portugal.

E. Agirre and D. Martinez. 2001. Learning class-to-class selectional preferences. In Proceedings of CoNLL, Toulouse, France.

E. Agirre and D. Martinez. 2002. Integrating selectional preferences in wordnet. In Proceedings of GWC, Mysore, India.

J. Atserias, L. Villarejo, G. Rigau, E. Agirre, J. Car­roll, B. Magnini, and Piek Vossen. 2004. The mean­ing multilingual central repository. In Proceedings of GWC, Brno, Czech Republic.

S. Brody, R. Navigli, and M. Lapata. 2006. Ensem­ble methods for unsupervised wsd. In Proceedings of COLING-ACL, pages 97-104.

M. Cuadros, L. Padro, and G. Rigau. 2005. Comparing methods for automatic acquisition of topic signatures.

In Proceedings ofRANLP, Borovets, Bulgaria.

M. Cuadros, L. Padro, and G. Rigau. 2006. An empirical study for automatic acquisition of topic signatures. In Proceedings of GWC, pages 51-59.

J. Daude, L. Padro, and G. Rigau. 2003. Validation and Tuning of Wordnet Mapping Techniques. In Proceed­ings ofRANLP, Borovets, Bulgaria.

C. Fellbaum, editor. 1998. WordNet. An Electronic Lexi­cal Database. The MIT Press.

S. Landes, C. Leacock, and R. Tengi. 2006. Build­ing a semantic concordance of english. In WordNet: An electronic lexical database and some applications. MIT Press, Cambridge,MA., 1998, pages 97-104.

C. Leacock, M. Chodorow, and G. Miller. 1998. Us­ing Corpus Statistics and WordNet Relations for Sense Identification. Computational Linguistics, 24(1):147166.

C. Lin and E. Hovy. 2000. The automated acquisition of topic signatures for text summarization. In Proceed­ings ofCOLING. Strasbourg, France.

D. McCarthy. 2001. Lexical Acquisition at the Syntax-Semantics Interface: Diathesis Aternations, Subcate-gorization Frames and Selectional Preferences. Ph.D. thesis, University ofSussex.

R. Mihalcea and D. Moldovan. 2001. extended wordnet: Progress report. In Proceedings ofNAACL Workshop on WordNet and Other Lexical Resources, Pittsburgh,

PA.

R. Mihalcea. 2006. Knowledge based methods for word sense disambiguation. In E. Agirre and P. Edmonds (Eds.) Word Sense Disambiguation: Algorithms and applications. , volume 33 of Text, Speech and Lan­guage Technology. Springer.

P. Vossen, editor. 1998. EuroWordNet: A Multilingual Database with Lexical Semantic Networks . Kluwer Academic Publishers .