A typical word sense disambiguation system is trained on a corpus of manually sense tagged text. Machine learning algorithms are then employed to find the best sense for a word in a novel context by generalizing from the training examples. The training data is costly to generate and inter-annotator agreement is difficult to achieve. Thus there is very little training data available: the largest single cor­pus of sense tagged text, SemCor, has 41,497 sense tagged words. (Yuret, 2004) observed that approxi­mately half of the test instances do not match any of the contextual features learned from the training data for an all words disambiguation task. (Yarowsky and Florian, 2002) found that each successive doubling of the training data only leads to a 3-4% error reduc­tion within their experimental range.

Humans do not seem to be cursed with an expo­nential training data requirement to become profi­cient with the use of a word. Dictionaries typically contain a definition and one or two examples of us­age for each sense. This seems to be sufficient for a human to use the word correctly in contexts that share no surface features with the dictionary exam­ples. The 10 waking seconds it takes a person to become proficient in a language does not seem suf­ficient to master all the words and their different senses. We need models that do not require large amounts of annotated text to perform WSD.

What possible process can explain our proficiency without relying on a lot of labeled data? Let us look at a concrete example: The two most frequent senses of the word "board" according to WordNet 3.0 (Fell­baum, 1998) are the "committee" sense, and the "plank" sense. When we hear a sentence like "There was a board meeting", it is immediately obvious that the first sense is intended. One hypothesis is that a common sense inference engine in your brain rules out the second sense. Maybe you visualize pieces of timber sitting around a meeting table and decide that it is absurd. Another hypothesis is that the plank sense does not even occur to you because you hear this sentence in the middle of a conversation about corporate matters. Therefore the plank sense is not psychologically "primed". Finally, maybe you sub­consciously perform a substitution and the sentence "There was a plank meeting" just sounds bad to your linguistic "ear".

In this paper I will describe a system that judges potential substitutions in a given context using a sta­tistical language model as a surrogate for the linguis­tic "ear". The likelihoods of the various substitutes are used to select the best sense for a target word.

The use of substitutes for WSD is not new. (Lea-cock et al., 1998) demonstrated the use of related monosemous words (monosemous relatives) to col­lect examples for a given sense from the Internet. (Mihalcea, 2002) used the monosemous relatives technique for bootstrapping the automatic acquisi­tion of large sense tagged corpora. In both cases, the focus was on collecting more labeled examples to be subsequently used with supervised machine learn­ing techniques. (Martinez et al., 2006) extended the method to make use of polysemous relatives. More importantly, their method places these relatives in the context of the target word to query a search en­gine and uses the search results to predict the best sense in an unsupervised manner.

There are three areas that distinguish my system from the previous work: (i) The probabilities for substitutes in context are determined using a statisti­cal language model rather than search hits on heuris-tically constructed queries, (ii) The set ofsubstitutes are derived from multiple sources and optimized us­ing WSD performance as the objective function, and (iii) A probabilistic generative model is used to se­lect the best sense rather than typical machine learn­ing algorithms or heuristics. Each of these areas is explained further below.

Probabilities for substitutes: Statistical language modeling is the art of determining the probability of a sequence of words. According to the model used in this study, the sentence "There was a committee meeting" is 17,629 times more likely than the sen­tence "There was a plank meeting". Thus, a statis­tical language model can be used as a surrogate for your inner ear that decides what sounds good and what sounds bad. I used a language model based on the Web 1T 5-gram dataset (Brants and Franz, 2006) which gives the counts of 1 to 5-grams in a web cor­pus of 10 words. The details of the Web1T model are given in the Appendix.

Given that I criticize existing WSD algorithms for using too much data, it might seem hypocritical to employ a data source with 10 words. In my de­fense, from an engineering perspective, an unanno-tated 10 word corpus exists, whereas large sense tagged corpora do not. From a scientific perspective, it is clear that no human ever comes close to expe­riencing 10 words, but they do outperform simple n-gram language models based on that much data in predicting the likelihood of words in novel contexts (Shannon, 1951). So, even though we do not know how humans do it, we do know that they have the equivalent of a powerful statistical language model in their heads.

Selecting the best substitutes: Perhaps more im­portant for the performance ofthe system is the deci­sion of which substitutes to try. We never thought of using "monkey" as a potential substitute for "board". One possibility is to use the synonyms in Word­Net which were selected such that they can be in­terchanged in at least some contexts. However 54% of WordNet synsets do not have any synonyms. Be­sides, synonymous words would not always help if they share similar ambiguities in meaning. Substi­tutes that are not synonyms, on the other hand, may be very useful such as "hot" vs. "cold" or "car" vs. "truck". In general we are looking for potential substitutes that have a high likelihood of appearing in contexts that are associated with a specific sense of the target word. The substitute selection method used in this work is described in Section 3.

Selecting the best sense: Once we have a lan­guage model and a set of substitutes to try, we need a decision procedure that picks the best sense of a word in a given context. An unsupervised system can be designed to keep track of the sense associ­ated with each substitute based on the lexical re­source used. However since I used multiple lexical resources, and had training data available, I chose a supervised approach. For each instance in the train­ing set, the likelihood of each substitute is deter­mined. Then instances of a single sense are grouped together to yield a probability distribution over the substitutes for that sense. When a test instance is encountered its substitute distribution is compared to that of each sense to select the most appropriate one. Section 2 describes the sense selection proce­dure in detail.

Consider a target word w0 with n senses S = {s1,...,sn}. Let Cj = {cj1 ,Cj2,...} be the set of contexts in the training data where wo has been tagged with sense sj. The prior probability of a sense Sj will be defined as:

Suppose we decide to use m substitutes W = {w1,..., wm}. The selection of the possible sub­stitutes is discussed in Section 3. Let P(wi, c) de­note the probability of the context c where the target word has been replaced with wj. This probability is obtained from the Web1T language model. The con­ditional probability of a substitute wi in a particular context c is defined as:

The conditional probability of a substitute wi for a particular sense Sj is defined as:

'In all the tasks participated, the system submitted a unique answer for each instance. Therefore precision, recall, F-measure, and accuracy have the same value. I will use the term accuracy to represent them all.

Given a test context ct, we would like to find out which sense sj it is most likely to represent:

To calculate the likelihood of the test context P(ct|sj), we first find the conditional probability distribution of the substitutes P(wi |ct), as described above. Treating these probabilities as fractional counts we can express the likelihood as:

Thus we choose the sense that maximizes the pos­terior probability:

argmaxjP(sj) n P(w[sj)P(w|ct)wew

Potential substitutes for a word were selected from WordNet 3.0 (Fellbaum, 1998), and the Roget The­saurus (Thesaurus.com, 2007).

When selecting the WordNet substitutes, the pro­gram considered all synsets of the target word and neighboring synsets accessible following a single link. All words contained within these synsets and their glosses were considered as potential substi­tutes.

When selecting the Roget substitutes, the program considered all entries that included the target word. By default, the entries that included the target word as part of a multi word phrase and entries that had the wrong part of speech were excluded.

I observed that the particular set of substitutes used had a large impact on the disambiguation per­formance in cross validation. Therefore I spent a considerable amount of effort trying to optimize the substitute sets. The union of the WordNet and Ro-get substitutes were first sorted based on their dis­criminative power measured by the likelihood ratio of their best sense:

LR P(wi| sj)

The following optimization algorithms were then run to maximize the leave-one-out cross validation (loocv) accuracy on the lexical sample WSD train­ing data.

The English Lexical Substitution Task (McCarthy and Navigli, 2007), for both human annotators and systems is to replace a target word in a sentence with as close a word as possible. It is different from the standard WSD tasks in that there is no sense repos­itory used, and even the identification of a discrete sense is not necessary.

The task used a lexical sample of 171 words with 10 instances each. For each instance the human annotators selected several substitutes. There were three subtasks: best: scoring the best substitute for a given item, oot: scoring the best ten substitutes for a given item, and mw: detection and identification of multi-words. The details of the subtasks and scor­ing can be found in (McCarthy and Navigli, 2007). My system participated in the first two subtasks.

Because there is no training set, the supervised optimization of the substitute set using the algo­rithms described in Section 3 is not applicable. Based on the trial data, I found that the Roget substi­tutes work better than the WordNet substitutes most

Table 1: best and oot results: P is precision, R is recall, Mode indicates accuracy selecting the sin­gle preferred substitute when there is one, nmwt is the score without items identified as multi-words, nmws is the score using only single word substi­tutes, rand is the score for the items selected ran­domly, and man is the score for the items selected manually.

of the time. The antonyms in each entry and the entries that did not have the target word as the head were filtered out to improve the accuracy. Antonyms happen to be good substitutes for WSD, but not so good for lexical substitution.

For the final output of the system, the substitutes wi in a context c were simply sorted by P (wi, c) which is calculated based on the Web1T language model.

In the best subtask the system achieved 12.9% ac­curacy, which is the top score and 2.95% above the baseline. The system was able to find the mode (a single substitute preferred to the others by the anno-tators) in 20.65% of the cases when there was one, which is 5.37% above the baseline and 0.08% be­low the top score. The top part of Table 1 gives the breakdown of the best score, see (McCarthy and

Navigli, 2007) for details.

The low numbers here are partly a consequence of the scoring formula used. Specifically, the score for a single item is bounded by the frequency ofthe best substitute in the gold standard file. Therefore, the

0.8 I-,—........-,—........-,—.......

-0.4 I-h—........-■—........-■—.......

Number of training instances

highest achievable score was not 100%, but 45.76%. A more intuitive way to look at the result may be the following: Human annotators assigned 4.04 dis­tinct substitutes for each instance on average, and my system was able to guess one ofthese as the best in 33.73% of the cases.

In the oot subtask the system achieved 46.15% accuracy, which is 16.45% above the baseline and 22.88% below the top result. The system was able to find the mode as one of its 10 guesses in 61.30% of the cases when there was a mode, which is 20.73% above the best baseline and 4.96% below the top score. Unlike the best scores, 100% accuracy is pos­sible for oot. Each item had 1 to 9 distinct substi­tutes in the gold standard, so an ideal system could potentially cover them all with 10 guesses. The sec­ond part of Table 1 gives the breakdown of the oot score.

In conclusion, selecting substitutes based on a standard repository like Roget and ranking them us­ing the ngram language model gives a good base­line for this task. To improve the performance along these lines we need better language models, and bet­ter substitute selection procedures. Even the best language model will only tell us which words are most likely to replace our target word, not which ones preserve the meaning. Relying on reposito­ries like Roget for the purpose ofsubstitute selection seems ad-hoc and better methods are needed.

The Coarse-Grained English Lexical Sample WSD Task (Palmer et al., 2007), provided training and test data for sense disambiguation of 65 verbs and 35 nouns. On average there were 223 training and 49 testing instances for each word tagged with an OntoNote sense tag (Hovy et al., 2006). OntoNote sense tags are groupings of WordNet senses that are more coarse-grained than traditional WN entries, and which have achieved on average 90% inter-annotator agreement. The number of senses for a word ranged from 1 to 13 with an average of 3.6.

I used substitute sets optimized for each word as described in Section 3. Then a single best sense for each test instance was selected based on the model given in Section 2. The system achieved 85.05% ac­curacy, which is 6.39% above the baseline of pick­ing the most frequent sense and 3.65% below the top score.

These numbers seem higher than previous Sen-seval lexical sample tasks. The best system in Senseval-3 (Mihalcea et al., 2004; Grozea, 2004) achieved 72.9% fine grained, 79.3% coarse grained accuracy. Many factors may have played a role but the most important one is probably the sense inven­tory. The nouns and verbs in Senseval-3 had 6.1 fine grained and 4.5 coarse grained senses on average.

The leave-one-out cross-validation result of my system on the training set was 83.21% with the un-filtered union of Roget and WordNet substitutes, and 90.69% with the optimized subset. Clearly there is some over-fitting in the substitute optimization pro­cess which needs to be improved.

Table 2 details the performance on individual words. The accuracy is 88.67% on the nouns and 81.02% on the verbs. One can clearly see the rela­tion of the performance with the number of senses (decreasing) and the frequency of the first sense (in­creasing). Interestingly no clear relation exists be­tween the training set size and the accuracy above the baseline. Figure 1 plots the relationship between training set size vs. the accuracy gain above the most frequent sense baseline. This could indicate that the system peaks at a low training set size and general­izes well because of the language model. However, it should be noted that each point in the plot rep­resents a different word, not experiments with the same word at different training set sizes. Thus the difficulty of each word may be the overriding factor in determining performance. A more detailed study similar to (Yarowsky and Florian, 2002) is needed to explore the relationship in more detail.

The Word Sense Disambiguation of Prepositions Task (Litkowski and Hargraves, 2007), provided training and test data for sense disambiguation of 34 prepositions. On average there were 486 train­ing and 234 test instances for each preposition. The number of senses for a word ranged from 1 to 20 with an average of 7.4.

The system described in Sections 2 and 3 were applied to this task as well. WordNet does not have information about prepositions, so most of the can­didate substitutes were obtained from Roget and The Preposition Project (Litkowski, 2005). After opti­mizing the substitute sets the system achieved 54.7% accuracy which is 15.1% above the most frequent sense baseline and 14.6% below the top result. Un­fortunately there were only three teams that partic­ipated in this task. The detailed breakdown of the results can be seen in the second part of Table 2.

The loocv result on the training data with the ini­tial unfiltered set of substitutes was 51.70%. Opti­mizations described in Section 3 increased this to 59.71%. This increase is comparable to the one in the lexical substitution task. The final result of 54.7% shows signs of overfitting in the substitute se­lection process.

The average gain above the baseline for preposi­tions (39.6% to 54.7%) is significantly higher than the English lexical sample task (78.7% to 85.1%). However the preposition numbers are generally lower compared to the nouns and verbs because they are more ambiguous: the number of senses is higher and the first sense frequency is lower.

Good quality substitutes are difficult to find for prepositions. Unlike common nouns and verbs, common prepositions play unique roles in language and are difficult to replace. Open class words have synonyms, hypernyms, antonyms etc. that provide good substitutes: it is easy to come up with "I ate halibut" when you see "I ate fish". It is not as easy to replace "of in the phrase "the president of the company". Even when there is a good substitute, e.g. "over" vs. "under", the two prepositions usually share the exact same ambiguities: they can both ex­press a physical direction or a quantity comparison. Therefore the substitution based model presented in this work may not be a good match for preposition disambiguation.

A WSD method employing a statistical language model was introduced. The language model is used to evaluate the likelihood of possible substitutes for the target word in a given context. Each context is represented with its preferences for possible substi­tutes, thus contexts with no surface features in com­mon can nevertheless be related to each other.

The set of substitutes used for a word had a large effect on the performance of the resulting system. A substitute selection procedure that uses the language model itself rather than external lexical resources may work better.

I hypothesize that the model would be advanta­geous on tasks like "all words" WSD, where data sparseness is paramount, because it is able to link contexts with no surface features in common. It can be used in an unsupervised manner where the sub­stitutes and their associated senses can be obtained from a lexical resource. Work along these lines was not completed due to time limitations.

Finally, there are two failure modes for the algo­rithm: either there are no good substitutes that dif­ferentiate the various senses (as I suspect is the case for some prepositions), or the language model does not yield accurate preferences among the substitutes that correspond to our intuition. In the first case we have to fall back on other methods, as the substi­tutes obviously are of limited value. The correspon­dence between the language model and our intuition requires further study.