The meaning of a word can be defined by the words that accompany it in the text. This is the principle often used in previous studies on Word Sense Disambiguation (WSD) (Ide and Véronis, 1998; Navigli, 2009). In general, the accompa­nying words form a context vector of the target word, or a probability distribution of the context words. For example, under the unigram bag-of-word assumption, this means building p(x\t) = count(xt) where count(xA) is the count of co-occurrences of word x with the target word t under a certain criterion. In most studies, x and t should co-occur within a window of up to k words or sentences. The bounds are usually selected as to maximize system performance. Occurrences in­side the window usually weight the same with­out regard to their position. This is counterintu­itive. Indeed, a word closer to the target word usu­ally has a greater semantic constraint on the tar­get word than a more distant word. Some studies have also proposed decaying factors to decrease the importance of more distant words in the con­text vector. However, the decaying functions are defined manually. It is unclear that the functions defined can capture the true impact of the con­text words on the target word. In this paper, we propose an unsupervised method to automatically learn the optimal weight of a word according to its distance to the target word. The general idea used to determine such weight is that, if we randomly determine two sets of texts containing the target word, the resulting probability distributions for its context words in the two sets should be similar. Therefore, the weights of context words at differ­ent distance are determined so as to maximize the mutual generation probabilities of two sets of sam­ples. Experimentation on Semeval-2007 English and Semeval-2010 lapanese lexical sample task data shows that improvements can automatically be attained on simple Naive Bayes (NB) systems in comparison to the best manually selected fixed window system.

The remainder of this paper is organized as fol­lows: example uses of text windows and related work are presented in Section 2. Our method is presented in Section 3. In Section 4 and 5, we show experimental results on English and lapanese WSD. We conclude in Section 6 with discussion and further possible extensions.

Modeling the distribution of words around one target word has many uses. For instance, the Xu&Croft co-occurrence-based stemmer (Xu and Croft, 1998) uses window co-occurrence statis­tics to calculate the best equivalence classes for a group of word forms. They suggest using win­dows of up to 100 words. Another example can be found in WSD systems, where a shorter window is preferred. In Semeval-2007, top performing sys­tems on WSD tasks, such as NUS-ML (Cai et al., 2007), made use of bag-of-word features around the target word. In this case, they found that the best results can be achieved using a window size of 3.

In this section, we present our method for choos­ing how much a word should count according to its distance to the target word. First, for some defini­tions, let C be a corpus, W a set of text windows, cw,i,x the count of occurrences of word x at dis­tance i in W, cw,i the sum of these counts, and the weight put on one word at distance i. Then,

J2j (XjCW,i,x J2i OLiCw.i

is the maximum likelihood estimator for x. To counter the zero-probability problem, we apply Dirichlet smoothing with the collection language model as a prior:

The pseudo-count ßw is found by using Newton's method via leave-one-out estimation. We follow the procedure shown in (Zhai and Lafferty, 2002), but since occurrences have different weights, the log-likelihood is changed to

To find the best weights for our model we pro­pose the following:

• Let T be the set of all windows containing the target word. We randomly split this set into two sets A and B.

• We want to find a* that maximizes the mu­tual generation of the two sets, by minimizing their cross-entropy:

In other words, we want to represent how much an occurrence at distance i models the con­text better than the collection language model, whose counts are controlled by the Dirichlet pseudo-count. We hypothesize that target words occurs in limited contexts, and as we get farther from them, the possibilities become greater, re­sulting in sparse and less related counts.

The Semeval workshop holds WSD tasks such as the English Lexical Sample (ELS) (Pradhan et al., 2007). It consists of a selected set of polysemous words, contained within passages where a sense taken from a sense inventory is manually anno­tated. The task is to create supervised classifiers maximizing accuracy on test data.

Since there are only 50 words and instances are few, we judged there was not enough data to com­pute weights. Instead, we used the AP Newswire corpus of the TREC collection (CD 1 & 2). Words were stemmed with the Porter stemmer and text windows were grouped for all words. For sim­plicity and efficiency, windows to the right and to the left were considered independent, and we only kept words with between 30 and 1000 windows. Also, only windows with a size of 100, which was considered big enough without any doubt, were kept. A stop list of the top 10 frequent words was used, but place holders were left in the windows to preserve the distances. Multiple consecutive stop words (ex: "of the") were merged, and the tar­get word, being the same for all samples of a set, was ignored. This results in 32,650 sets contain­ing 5,870,604 windows. In Figure 1, we can see the resulting weight curve.

Figure 1 : Weight curve for AP Newswire

Since the curve converges, words over the 100th distance were assigned the minimum weight found in the curve. From this we constructed NB models whose class priors used an absolute discounting of 0.5. The collection language model used the con­catenation of the AP collection and the Semeval data. As the unstemmed target word is an impor­tant feature it was added to the models. It's weight was chosen to be 0.7 by maximizing accuracy on one-held-out cross-validation of the training data. The results are listed in Table 1.

We used two baselines: most frequent sense (prior only), and the best uniform (except target word) fixed size window found from extensive search on the training data. The best settings were a window of size 4, with a weight of 4.4 on the target word and a Laplace smoothing of 2.9. The improvements seen using our system are substan­tial, beating most of the systems originally pro­posed for the task (Pradhan et al., 2007). Out of 15 systems, the best results had accuracies of 89.1*, 89.1*, 88.7, 86.9 and 86.4 (* indicates post-competition submissions). Notice that most were using Support Vector Machine (SVM) with bag-of-word features in a very small window, local col­locations and POS tags. In our future work, we will investigate the applications of SVM with our new term weighting scheme.

The Semeval-2010 lapanese WSD task (Okumura et al., 2010) consists of 50 polysemous words for which examples were taken from the BC-CWI tagged corpus. It was manually segmented, tagged, and annotated with senses taken from the Iwanami Kokugo dictionary. The task is identical to the ELS of the previous experiment.

Since the data was again insufficient to com­pute curves, we used the Mainichi-2005 corpus of NTCIR-8. We tried to reproduce the same kind of segmentation as the training data by using the Chasen parser with UniDic. For the corpus and Semeval data, conjugations (setsuzoku-to, jodô-shi, etc.), particles (all jo-shi), symbols (blanks, kigô, etc.), and numbers were stripped. When a base-form reading was present (for verbs and ad­jectives), the token was replaced by the Kanjis (Chinese characters) in the word writing concate­nated with the base-form reading. This treatment is somewhat equivalent to the stemming+stop list of the ELS tasks. The resulting curve can be seen in Figure 2.

The NB models are the same as in the previous experiments. Target words were again added the same way as in the ELS task. The best fixed win­dow model was found to have a window size of 1 with a target word weight of 0.6 and used manual Dirichlet smoothing with a pseudo-count of 110. We submited two systems with the following set­tings: RALI-1 used manual Dirichlet smoothing and 0.9 for the target word. RALI-2 used auto-

distance

matic Dirichlet smoothing and 1.7 for the target word weight. Results are listed in Table 2.

As we can see, the results are not significantly different from the best uniform model. This may be due to differences in the segmentation parame­ters of our external corpus. Another reason could be that the systems use almost the same weights: the best fixed window had size 1, and the lapanese curve is steeper than the English one.

This steeper curve can be explained by the grammatical structure of the lapanese language. While English can be considered a Subject-Verb-Complement language, lapanese is consid­ered Subject-Complement-Verb. Verbs are mostly found at the end of the sentence, far from their sub­ject, and vice versa. The window distance is there­fore less useful in lapanese than in English since it has more non-local dependencies. These results show that the curves work as expected even in dif­ferent languages.

This paper proposed an unsupervised method for finding weights for counts in text windows ac­cording to their distance to the target word. Results from the Semeval-2007 English lexical sam­ple showed a substantial improvement in preci­sion. Yet, as we have seen with the lapanese task, window distance is not always a good indicator of word relatedness. Fortunately, we can easily imag­ine extensions to the current scheme that bins word counts by factors other than word distance. For in­stance, we could also bin counts by parsing tree distance, sentence distance or POS-tags.

The authors would like to thank Florian Boudin and Satoko Fujisawa for helpful comments on this work. This work is partially supported by lapanese MEXT Grant-in-Aid for Scientific Research on Info-plosion (#21013046) and the lapanese MEXT Research Student Scholarship program.