Lexical substitution is defined as the task of identify­ing the most likely alternatives (substitutes) for a tar­get word, given its context (McCarthy, 2002). Many natural language processing applications can bene­fit from the availability of such alternative words, including word sense disambiguation, lexical ac­quisition, machine translation, information retrieval, question answering, text simplification, and others.

The task is closely related to the problem of word sense disambiguation, with the substitutes acting as synonyms for the input word meaning. Unlike word sense disambiguation however, lexical substitution is not performed with respect to a given sense inven­tory, but instead candidate synonyms are generated "on the fly" for a given word occurrence. Thus, lexi­cal substitution can be regarded in a way as a hybrid task that combines word sense disambiguation and distributional similarity, targeting the identification of semantically similar words that fit the context.

SubFinder is a system able to provide the most likely set of substitutes for a word in a given context.

* Contact author.

In SubFinder, the lexical substitution task is car­ried out as a sequence of two steps. First, candidates are extracted from a variety of knowledge sources; so far, we experimented with WordNet (Fellbaum, 1998), Microsoft Encarta encyclopedia, Roget, as well as synonym sets generated from bilingual dic­tionaries, but additional knowledge sources can be integrated as well. Second, provided a list of candi­dates, a number of ranking methods are applied in a weighted combination, resulting in a final list of lexical substitutes ranked by their semantic fit with both the input target word and the context.

Candidates are extracted using several lexical re­sources, which are combined into a larger compre­hensive resource.

WordNet: WordNet is a large lexical database of English, with words grouped into synonym sets called synsets. A problem we encountered with this resource is that often times the only candidate in the synset is the target word itself. Thus, to enlarge the set of candidates, we use both the synonyms and the hypernyms of the target word. We also remove the target word from the synset, to ensure that only vi­able candidates are considered.

Microsoft Encarta encyclopedia: The Microsoft Encarta is an online encyclopedia and thesaurus re­source, which provides for each word the part of speech and a list of synonyms. Using the part of speech as identified in the context, we are able to ex­tract synsets for the target word. An important fea­ture in the Encarta Thesaurus is that the first word in the synset acts as a definition for the synset, and therefore disambiguates the target word. This defi­nition is maintained as a separate entry in the comprehensive resource, and it is also added to its corre­sponding synset.

Other Lexical Resources: We have also experi­mented with two other lexical resources, namely the Roget thesaurus and a thesaurus built using bilingual dictionaries. In evaluations carried out on the devel­opment data set, the best results were obtained using only WordNet and Encarta, and thus these are the resources used in the final SubFinder system.

All these resources entail different forms ofsynset clustering. In order to merge them, we use the largest overlap among them. It is important to note that the choice of the first resource considered has a bearing on the way the synsets are clustered. In experiments ran on the development data set, the best results were obtained using a lexical resource constructed starting with the Microsoft Encarta The­saurus and then mapping the WordNet synsets to it.

Several ranking methods are used to score the can­didate substitutes, as described below.

Lexical Baseline (LB): In this approach we use the pre-existing lexical resources to provide a rank­ing over the candidate substitutes. We rank the can­didates based on their occurrence in the two selected lexical resources WordNet and Encarta, with those occurring in both resources being assigned a higher ranking. This technique emphasizes the resources annotators' agreement that the candidates belong in­deed to the same synset.

Machine Translation (MT): We use machine translation to translate the test sentences back-and-forth between English and a second language. From the resulting English translation, we extract the re­placement that the machine translation engine pro­vides for the target word. To locate the translated word we scan the translation for any of the can­didates (and their inflections) as obtained from the comprehensive resource, and score the candidate synset accordingly.

We experimented with a range of languages such as French, Italian, Spanish, Simplified Chinese, and German, but the best results obtained on the devel­opment data were based on the French translations. This could be explained because French is part of the Romance languages family and synonyms to En­glish words often find their roots in Latin. If we consider again the word bright, it was translated into French as intelligent and then translated back into English as intelligent for obvious reasons. In one instance, intelligent was the best replacement for bright in the trial data. Despite the fact that we also used Italian and Spanish (which are both Latin-based) we can only assume that French worked bet­ter because translation engines are better trained on French. From the resulting English translation, we extract the replacement that the machine translation engine provides for the target word. To locate the translated word we scan the translation for any ofthe candidates (and their inflections) as obtained from the comprehensive resource, and score the candidate synset accordingly. The translation process was car­ried out using Google and AltaVista translation en­gines resulting in two systems MTG and MTA re­spectively. The translation systems feature high pre­cision when a candidate is found (about 20% of the time), at the cost of low recall. The lexical baseline method is therefore used when no candidates are re­turned by the translation method.

Most Common Sense (MCS): Another method we use for ranking candidates is to consider the first word appearing in the first synset returned by WordNet. When no words other than the target word are available in this synset, the method recur­sively searches the next synset available for the tar­get word. In order to guarantee a sufficient number of candidates, we use the lexical baseline method as a baseline.

Language Model (LM): We model the semantic fit of a candidate substitute within the given context using a language model, expressed using the condi­tional probability:

where c represents a possible candidate and g rep­resents the context. The probability P(g) of the context is the same for all the candidates, hence we can ignore it and estimate P(c| g) as the N-gram fre­quency of the context where the target word is re­placed by the proposed candidate. To avoid skewed counts that can arise from the different morpholog­ical inflections of the target word or the candidate and the bias that the context might have toward any specific inflection, we generalize P(c|g) to take into account all the inflections of the selected candidate as shown in equation 2.

where n is the number of possible inflections for the candidate c.

We use the Google N-gram dataset to calculate the term Count(ci g). The Google N-gram corpus is a collection of English N-grams, ranging from one to five N-grams, and their respective frequency counts observed on the Web (Brants and Franz, 2006). In order for the model to give high preference to the longer N-grams, while maintaining the relative fre­quencies of the shorter N-grams (typically more fre­quent), we augment the counts of the higher order N-grams with the maximum counts of the lower or­der N-grams, hence guaranteeing that the score as­signed to an N-gram of order N is higher than the the score of an N-gram of order N — 1.

Semantic Relatedness using Latent Semantic Analysis (LSA): We expect to find a strong se­mantic relationship between a good candidate and the target context. A relatively simple and efficient way to measure such a relatedness is the Latent Se­mantic Analysis (Landauer et al., 1998). Documents and terms are mapped into a 300 dimensional latent semantic space, providing the ability to measure the semantic relatedness between two words or a word and a context. We use the InfoMap package from Stanford University's Center for the Study of Lan­guage and Information, trained on a collection of approximately one million Wikipedia articles. The rank of a candidate is given by its semantic related-ness to the entire context sentence.

Information Retrieval (IR): Although the Lan­guage Model approach is successful in ranking the candidates, it suffers from the small N-gram size im­posed by using the Google N-grams corpus. Such a restriction is obvious in the following 5-gram ex­ample who was a bright boy in which the context is not sufficient to disambiguate between happy and smart as possible candidates. As a result, we adapt an information retrieval approach which uses all the content words available in the given context. Similar to the previous models, the target word in the con­text is replaced by all the generated inflections of the selected candidate and then queried using a web search engine. The resulting rank represents the sum of the total number of pages in which the candidate or any of its inflections occur together with the con­text. This also reflects the semantic relatedness or the relevance of the candidate to the context.

Word Sense Disambiguation (WSD): Since pre­vious work indicated the usefulness of word sense disambiguation systems in lexical substitution (Da-gan et al., 2006), we use the SenseLearner word sense disambiguation tool (Mihalcea and Csomai, 2005) to disambiguate the target word and, accord­ingly, to propose its synonyms as candidates.

Final System: Our candidate ranking methods are aimed at different aspects of what constitutes a good candidate. On one hand, we measure the semantic relatedness of a candidate with the original context (the LSA and WSD methods fall under this cate­gory). On the other hand, we also want to ensure that the candidate fits the context and leads to a well formed English sentence (e.g., the language model method). Given that the methods described earlier aim at orthogonal aspects of the problem, it is ex­pected that a combination of these will provide a better overall ranking.

We use a voting mechanism, where we consider the reciprocal of the rank of each candidates as given by one of the described methods. The final score of a candidate is given by the decreasing order of the weighted sum of the reciprocal ranks:

m£rankings c

To determine the weight X of each individual ranking we run a genetic algorithm on the develop­ment data, optimized for the mode precision and re­call. Separate sets of weights are obtained for the best and oot tasks. Table 1 shows the weights of the individual ranking methods. As expected, for the best task, the language model type of methods obtain higher weights, whereas for the oot task, the semantic methods seem to perform better.

The SubFinder system participated in the best and the oot tracks of the lexical substitution task. The best track calls for any number of best guesses, with the most promising one listed first. The credit for each correct guess is divided by the number of guesses. The oot track allows systems to make up to 10 guesses, without penalizing, and without being of any benefit if less than 10 substitutes are provided. The ordering of guesses in the oot metric is unim­portant.

For both tracks, the evaluation is carried out using precision and recall, calculated based on the number of matching responses between the system and the human annotators, respectively. A "mode" evalua­tion is also conducted, which measures the ability of the systems to capture the most frequent response (the "mode") from the gold standard annotations. For details, please refer to the official task descrip­tion document (McCarthy and Navigli, 2007).

Tables 2 and 3 show the results obtained by Sub-Finder in the best and oot tracks respectively. The tables also show a breakdown of the results based on: only target words that were not identified as multiwords (NMWT); only substitutes that were not identified as multiwords (nmws); only items with sentences randomly selected from the Internet cor­pus (rand); only items with sentences manually se­lected from the Internet corpus (MAN).

nmwt nmws rand man

Analysis

Baselines

Table 2: best results

Table 3: OOT results

Compared to other systems participating in this task, our system consistently ranks on the first or second place. SubFinder clearly outperforms all the other systems for the "mode" evaluation, show­ing the ability of the system to find the substitute most often preferred by the human annotators. In addition, the system exceeds by a large margin all the baselines calculated for the task, which select substitutes based on existing lexical resources (e.g., WordNet or Lin distributional similarity).

Separate from the "official" submission, we ran a second experiment where we optimized the com­bination weights targeting high precision and recall (rather than high mode). An evaluation ofthe system using this new set of weights yields a precision and recall of 13.34 with a mode of 21.71 for the best task, surpassing the best system according to the anony­mous results report. For the oot task, the precision and recall increased to 50.30, still maintaining sec­ond place.

The lexical substitution task goes beyond simple word sense disambiguation. To approach such a task, we first need a good comprehensive and precise lexical resource for candidate extraction. Secondly, we need to semantically filter the highly diverse and ambiguous set of candidates, while taking into ac­count their fitness in the context in order to form a proper linguistic expression. To accomplish this, we built a system that incorporates lexical, semantic, and probabilistic methods to capture both the seman­tic similarity with the target word and the semantic fit in the context. Compared to other systems partic­ipating in this task, our system consistently ranks on the first or second place. SubFinder clearly out­performs all the other systems for the "mode" eval­uation, proving its ability to find the substitute most often preferred by the human annotators.

This work was supported in part by the Texas Ad­vanced Research Program under Grant #003594. The authors are grateful to the Language and Infor­mation Technologies research group at the Univer­sity of North Texas for many useful discussions and feedback on this work.