We advocate researching unsupervised tech­niques for Word Sense Disambiguation (WSD). Supervised techniques offer better results in general but the setbacks, such as the problem of developing reliable training data, are very considerable. Also there's probably more to WSD than blind machine learning (a typical ap­proach, although such systems produce interest­ing baselines).

Within the unsupervised paradigm, we are in­terested in performing in-depth measures of the disambiguation potential of different sources of information. We have previously investigated the informational value of semantic distance measures in (Fernandez-Amorös et aL, ). For Senseval-2, we have turned to investigate pure coocurrence information as a source of disam­biguation evidence. In essence, our system com­putes a matrix of mutual information for a fixed vocabulary and applies it to weight coocurrence counting between sense and context character­istic vectors.

In the next section we describe the process of constructing the relevance matrix. In section 3 we present the particular heuristics used for the competing systems. In section 4 we show the results by system and heuristic and some base­lines for comparison. Finally in the last sections we draw some conclusions about the exercise.

We have developed a very simple language in order to systematize the experiments. This lan­guage allows the construction of WSD systems composed of different heuristics that are ap­plied in cascade so that each word to be disam­biguated is presented to the first heuristic, and if it fails to disambiguate, then the word is passed on to the second heuristic and so on. We can have several such systems running in parallel for efficiency reasons (the matrix has high memory requirements). Next we show the heuristics we have considered to build the systems• Monosemous expressions.

Monosemous expressions are simply unam­biguous words in the case of the all words English task. In the case of the lexical sample English task, however, the annota­tions include multiword expressions. We have implemented a multiword term de­tector that considers the multiword terms from WordNet's index.sense file and detects them in the test file using a multilevel back­tracking algorithm that takes account of the inflected and base forms of the compo­nents of a particular multiword in order to maximize multiword detection. We tested this algorithm against the PG and found millions of these multiword terms.

We restricted ourselves to the multiwords already present in the training file since there are, apparently, multiword expres­sions that where overlooked during manual tagging (for instance the WordNet expres­sion 'the..good_old_days' is not hand-tagged as such in the test files)• Statistical filter

WordNet comes with a file, cntlist, literally 'file listing number of times each tagged sense occurs in a semantic concordance' so we use this to compute the relative prob­ability of a sense given a word (approxi­mate in the case of collections other than SemCor). Using this information, we elimi­nated the senses that had a probability un­der 10% and if only one sense remains we choose it. Otherwise we go on to the next heuristic. In other words, we didn't apply complex techniques with words which are highly skewed in meaning .

• Relevance filter

This heuristic makes use of the relevance matrix. In order to assign a score to a sense, we count the coocurrences of words in the context of the word to be dis­ambiguated with the words in the defini­tion of the senses (the WordNet gloss to-kenized, lemmatized and stripped out of stop words and punctuation signs) weight­ing each coocurrence by the entry in the relevance matrix for the word to be disam­biguated and the word whose coocurrences are being counted, i.e., if s is a sense of the word a whose definition is S and C is the context in which a is to be disambiguated, then the score for s would be:

Where idf(iu,a) = logj^-, with N being the number of senses for word a and dw the number of sense glosses in which w appears. freq(iu, C) is the frequency of word w in the context C and freq(i(/, S) is the frequency of w in the sense gloss S.

The idea is to prime the occurrences of words that are relevant to the word being disambiguated and give low credit (possi­bly none) to the words that are incidentally used in the context.

Also, in the all words task (where POS tags from the TreeBank are provided) we have considered only the context words that have a POS tag compatible with that of the word being disambiguated. By com­patible we mean nouns and nouns, nouns and verbs, nouns and adjectives, verbs and verbs, verbs and adverbs and vice versa. Roughly speaking, words that can have an intra-phrase relation.

We also filtered out senses with low values in the cntlist file, and in any case we only considered at most the first six senses of a word.

Enriching sense characteristic vectors

The relevance filter provided very good re­sults in our experiments with SemCor and Senseval-1 data as far as precision is concerned, but the problem is that there is little overlapping between the defini­tions of the senses and the contexts in terms of coocurrence (after removing stop words and computing idf) which means that the previous heuristic didn't disam­biguate many words.

To overcome this problem, we enrich the senses characteristic vectors adding for each word in the vector the words related to it via the relevance matrix weights. This corresponds to the algebraic notion of mul­tiplying the matrix and the characteristic vector. In other words, if R is the relevance matrix and v our characteristic vector we would finally use Rv + v. This should increase the number of words disambiguated provided we eliminate the idf factor (which would be zero in most cases because now the sense characteristics vectors are not as sparse as before). When we also discard senses with low relative fre­quency in SemCor we call this heuristic mixed filter.

back off strategies

For those cases that couldn't be covered by other heuristics we employed the first sense heuristic. In the case of the supervised sys­tem for the English lexical sample task we thought of using the most frequent sense but didn't implement it due to lack of time. Systems and Results UNED-AW-U2

We won't delve into UNED-AW-U system as it is very similar to this one. This is an (arguably) unsupervised system for the English all words task. The heuristics we used and the results obtained for each of them are shown in Table 1.

If the individual heuristics are used as stan­dalone WSD systems we would obtain the results in Table 2.

In the lexical sample task, we weren't able to multiply by the relevance matrix due to time constraints, so in order to increase the coverage for the relevance filter heuristic we expanded the definitions of the senses with those of the first 5 levels of hyponyms. Also, we selected the ra­dius of the context to be considered depending on the POS of the word being disambiguated. For nouns and verbs we used 25 words radius neighbourhood and for adjectives 5 words at each side.

• UNED-LS-U This is essentially the same system as UNED-AW-U2, in this case ap­plied to the lexical sample task. The results are displayed in Table 3.

We've put a lot of effort into making the rele­vance matrix but its performance in the WSD task is striking. The matrix is interesting and its application in the relevance filter heuristic is slightly better than simple coocurrence count­ing, which proves that it doesn't discard rele­vant words. The problem seems to lie in the fact that irrelevant words (with respect to the word to be disambiguated) rarely occur both in the context of the word and in the definition of the senses (if they appeared in the definition they wouldn't be so irrelevant) so the direct im­pact of the information in the matrix is very weak. Likewise, relevant (via the matrix) words with respect to the word to be disambiguated occur often both in the context and in the defi­nitions so the final result is very similar to sim­ple coocurrence counting.

This problem only showed up in the lexical sample task systems. In the all words systems we were to enrich the sense definitions to make a more advantageous use of the matrix.

We were very confident that the relevance filter would yield good results as we have already evaluated it against the SensevaL-1 and SemCor data. We felt however that we could improve the coverage of the heuristic enrich­ing the definitions multiplying by the matrix. A similar approach was used by Yarowsky (Yarowsky, 1992) and Schütze (Schütze and Pedersen, 1995) and it worked for them. This wasn't the case for us; still, we think the re­source is well worth researching other ways of using it.

As for the overall scores, the unsupervised lexical sample obtained the highest recall of the unsupervised systems, which proves that care­fully implementing simple techniques still pays off. In the all words task the UNED-WS-U2 had also the highest recall among the unsupervised systems (as characterized in the Senseval-2 web descriptions), and the fourth overall. We'll train it with the examples in Semcor 1.6 and see how much we can gain.