Topic signatures (Agirre and de Lacalle, 2004) are lists of words related to a particular sense. They can be built from a variety of sources, and be used di­rectly to perform WSD. Cuadros and Rigau (2006) present a detailed evaluation of topic signatures built from a variety of knowledge sources. In this work we built those coming from the following:

• the relations in the Multilingual Central Repos­itory (TS-MCR)

• the relations in the Extended WordNet (TS-XWN)

In order to apply this resource for WSD, we sim­ply measured the word-overlap between the target context and each of the senses of the target word. The sense with highest overlap is chosen as the cor­rect sense.

This is an unsupervised method presented in Mar­tinez et al. (2006). This algorithm makes use of the WordNet relatives of the target word for disam­biguation. The process is carried out in these steps: (i) obtain a set of close relatives from WordNet for each sense (the relatives can be polysemous); (ii) for each test instance define all possible word sequences that include the target word; (iii) for each word se­quence, substitute the target word with each relative and query a web search engine; (iv) rank queries ac­cording to the following factors: length of the query, distance of the relative to the target word, and num­ber of hits; and (v) select the sense associated with the highest ranked query.

The intuition behind this system is that we can find related words that can be substituted for the tar­get word in a given context, which are indicative of its sense. The close relatives that can form more common phrases from the target context determine the target sense.

This heuristic has been motivated as a way of identi­fying rare senses of a word. An important disadvan­tage of unsupervised systems is that rare senses can be over-represented in the models, while supervised systems are able to discard them because they have access to token-level word sense distributions.

This simple algorithm relies on the number of close relatives found in WordNet for each sense of the word. The senses are ranked according to the number of synonyms, direct hypernyms, and di­rect hyponyms they have in WordNet. The highest ranked sense is taken to be the most important for the target word, and all occurrences of the target word are tagged with that sense.

As our supervised system, we relied on kNN. This is a memory-based learning method where the neigh­bours are the k most similar contexts, represented by feature vectors (ci) of the test vector (/). The sim­ilarity among instances is measured by the cosine of their vectors. The test instance is labeled with the sense that obtains the maximum sum of the weighted votes of the k most similar contexts. Each vote is weighted depending on its (neighbour) position in the ordered rank, with the closest being first. Equa­tion 1 formalizes kNN, where Ci corresponds to the sense label of the i-th closest neighbour.

The UBC group used a combination of kNN clas­sifiers trained over a large set of features, and en­hanced this method using Singular Value Decompo­sition (SVD) for their supervised submission (UBC-ALM) to the lexical-sample and all-words subtasks (Agirre and Lopez de Lacalle, 2007). However, we only used the basic implementation in this work, due to time constraints.

We explored two approaches to combine the stan­dalone systems. The first consisted simply of adding up the normalized weights that each system would give to each sense. We tested this voting approach both for the unsupervised and supervised settings.

The second method could only be applied in com­bination with the supervised kNN system. The idea was to include the unsupervised predictions as weighted features for the supervised system. We re­fer to this method as "stacking", and it has been pre­viously used to integrate heterogeneous knowledge sources for WSD (Stevenson and Wilks, 2001).

For the first evaluation of our unsupervised systems, we relied on Semcor, and tagged 43,063 instances of the 329 word types occurring in SemEval-2007

Table 1: Evaluation of standalone and combined unsupervised systems over 43,063 instances from Semcor

Table 2: Evaluation of standalone and combined unsupervised systems over 8,518 instances from S07LS training all-words. Due to time constraints, we were not able to test the RIC algorithm on this dataset. The re­sults are shown in Table 1. We can see that the RNB heuristic performs poorly, and that the best configu­ration consists of applying the single TS-MCR algo­rithm. From this experiment, we decided to remove the RNB heuristic and focus on the topic signatures and RIC.

We also used S07LS for extra experiments in the unsupervised setting. From the training part of the S07LS dataset, we extracted 8,518 instances of words also occurring in SemEval-2007 all-words. As S07LS used senses from OntoNotes, we relied on the mapping provided by the task organisers to link them to WordNet senses. We left RNB out of this experiment due to its low performance in Sem-cor, and regarding RIC, we only evaluated a sample of 68 instances. Results are shown in Table 2. The best scores are achieved when combining both sets of topic signatures. The few cases that have been disambiguated with RIC improve the overall perfor­mance slightly.

We could not rely on Semcor in the supervised set­ting (we used it for training), and therefore tried to use as much data as possible from the training com­ponent of S07LS, wherein all the instances avail­able (22,281) were disambiguated. We tested first

System Recall

Table 4: Evaluation of "stacking" the unsupervised systems on kNN over 8,518 instances from S07LS training the voting combination by adding the normalized weights from the output of each system. Due to time constraints we only evaluated the combination of kNN with TS-MCR and TS-XWN. Results are shown in Table 3, where we can see that combin­ing the unsupervised systems with voting hurts the performance of the kNN method.

Finally, we applied the second combination ap­proach, consisting of including the predictions of the unsupervised systems as features for kNN ("stack­ing"). We performed this experiment on the training part of S07LS, but only for the 8,518 instances of the words occurring on the all-words dataset. The results of this experiment are given in Table 4. We observed a slight improvement in this case.