Supervised corpus-based approach has been the most successful in WSD to date. However, this ap­proach faces severe data scarcity problem, resulting features being sparsely represented in the training data. This problem is especially prominent for the bag-of-words feature. A direct consequence is that the global context information, which the bag-of-words feature is supposed to capture, may be poorly represented.

Our system tries to address this problem by clustering features to relieve the scarcity problem, specifically on the bag-of-words feature. In the pro­cess, we construct topic features, trained using the latent dirichlet allocation (LDA) algorithm. We train the topic model (Blei et al., 2003) on unlabeled data, clustering the words occurring in the corpus to a pre­defined number of topics. We then use the resulting topic model to tag the bag-of-words in the labeled corpus with topic distributions.

We incorporate the distributions, called the topic features, using a simple Bayesian network, modified from naïve Bayes model, alongside other features and train the model on the labeled corpus.

For both the lexical sample and all-words tasks, we use the following standard baseline features.

POS Tags For each word instance w, we include POS tags for P words prior to as well as after w within the same sentence boundary. We also include the POS tag of w. If there are fewer than P words prior or after w in the same sentence, we denote the corresponding feature as NIL.

Local Collocations We adopt the same 11 col­location features as (Lee and Ng, 2002), namely

C-3,-1, C-2,1, C-!^ and C1,3.

We construct a variant of the naïve Bayes network as shown in Figure 2. Here, w refers to the word. s refers to the sense of the word. In training, s is observed while in testing, it is not. The features f1to fn are baseline features mentioned in Section 2.1 (including bag-of-words) while z refers to the la­tent topic that we set for clustering unlabeled corpus. The bag-of-words b are extracted from the neigh­bours of w and there are L of them. Note that L can be different from G, which is the number of bag-of-words in baseline features. Both will be determined by the validation result.

The log-likelihood of an instance, £(w, s, F, b) where F denotes the set of baseline features, can be written as

+ Y1 log Yl p(zk |s)p(bi |zfc ) .

The log p(w) term is constant and thus can be ignored. The first portion is normal naïve Bayes. And second portion represents the additional LDA plate. We decouple the training process into separate stages. We first extract baseline features from the task training data, and estimate, using normal naïve Bayes, p(s|w) and p(f |s) for all w, s and f.

Next, the parameters associated with p(b|z) are estimated using LDA from unlabeled data, which is ß. To estimate p(z|s), we perform LDA inference on the training corpus in order to obtain Yd for each document d. We then use the Yd and ß to obtain p(z|b) for each word using where equation (1) is used for estimation of p(zi |Yd).

This effectively transforms b to a topical distri­bution which we call a soft tag where each soft tag is probability distribution t1,..., tK on topics. We then use this topical distribution for estimating p(z|s). Let si be the observed sense of instance i and t1j,..., t%K be the soft tag of the j-th bag-of-word feature of instance i. We estimate p(z|s) as

This approach requires us to do LDA inference on the corpus formed by the labeled training data, but not the testing data. This is because we need Y to get transformed topical distribution in order to learn p(z|s) in the training. In the testing, we only apply the learnt parameters to the model.

We describe here the experimental setup on the En­glish lexical sample task and all-words task. Note that we do not distinguish the two all-words tasks as the same parameters will be applied.

For lexical sample task, we use 5-fold cross val­idation on the training data provided to determine our parameters. For all-words task, we use SemCor as our training data and validate on Senseval-2 and Senseval-3 all-words test data.

We use MXPOST tagger (Adwait, 1996) for POS tagging, Charniak parser (Charniak, 2000) for ex­tracting syntactic relations, and David Blei's version of LDA for LDA training and inference. All default parameters are used unless mentioned otherwise.

For the all-word tasks, we use sense 1 as back-off for words that have not appeared in SemCor. We use the same fine-grained system for both the coarse and fine-grained all-words tasks. We make predictions for all words for all the systems - precision, recall and accuracy scores are all the same.

Baseline features For lexical sample task, we choose P = 3 and G = 3. For all-words task, we choose P = 3 and G = 1. (G = 1 means only the nearest word prior and after the test word.) Smoothing For all standard baseline features, we use Laplace smoothing but for the soft tag (equation (2)), we use a smoothing parameter value of 2 for all-words task and 0.1 for lexical sample task.

Unlabeled Corpus Selection The unlabeled cor­pus we select from for LDA training include 20 Newsgroups, Reuters, SemCor, Senseval-2 lexical sample data, Senseval-3 lexical sample data and SemEval-1 lexical sample data. Although the last four are labeled corpora, we only need the words from these corpora and thus they can be regarded as unlabeled too. For lexical sample data, we define the whole passage for each training and testing instance as one document.

For lexical sample task, we use all the unlabeled corpus mentioned with K = 60 and L = 18. For all-words task, we use a corpora consisting only 20 Newsgroups and SemCor with K = 40 and L = 14.

Validation Result Table 2 shows the results we get on the validation sets. We give both the system accuracy (named as Soft Tag) and the naïve Bayes result with only standard features as baseline.

We now present the official results on all three tasks we participated in, summarized in Table 3.

The system ranked first, fourth and second in the lexical sample task, fine-grained all-words task and coarse-grained all-words task respectively. For coarse-grained all-words task, we obtained 86.1, 88.3, 81.4, 76.7 and 79.1 for each document, from d001 to d005.

Task Precision/Recall Lexical sample(Task 17) 88.7 Fine-grained all-words(Task 17) 57.6 Course-grained all-words(Task 7) 81.6

For the lexical sample task, we compare the re­sults to that of our naïve Bayes baseline and Sup­port Vector Machine (SVM) (Vapnik, 1995) base­line. Our SVM classifier (using SVMlight) follows that of (Lee and Ng, 2002), which ranked the third in Senseval-3 English lexical sample task. We also analyse the result according to the test instance's part-of-speech and find that the improvements are consistent for both noun and verb.

Our coarse-grained all-words task result outper­formed the first sense baseline score of 0.7889 by about 2.7%.