Accurate word sense disambiguation (WSD) can support many natural language processing and knowledge management tasks. The main goal of the PNNL WSD system is to support Semantic Web applications, such as semantic-driven search and navigation, through a reliable mapping of words in naturally occurring text to ontological classes. As described in Sanfilippo et al. (2006), this goal is achieved by defining a WordNet-based (Fellbaum, 1998) ontology that offers a manage­able set of concept classes, provides an extensive characterization of concept class in terms of lexical instances, and integrates an automated class recog­nition algorithm. We found that the same features that are useful for predicting word classes are also useful in distinguishing individual word senses.

Our main objective in this paper is to predict in­dividual word senses using a large combination of features including contextual, semantic, and syn­tactic information. In our earlier paper (Sanfilippo et al., 2006), we reported that the PNNL WSD system exceeded the performance of the best perform­ers for verbs in the SENSEVAL-3 English all-words task dataset. SemEval 2007 is our first op­portunity to enter a word sense disambiguation competition.

While many unsupervised word sense disambigua­tion systems have been created, supervised systems have generally produced superior results (Snyder and Palmer, 2004; Mihalcea et al., 2004). Our sys­tem is based on a supervised WSD approach that uses a Maximum Entropy classifier to predict WordNet senses.

We use SemCor, OMWE 1.0 (Chklovski and Mihalcea, 2002), and example sentences in Word­Net as the training corpus. We utilize the OpenNLP MaxEnt implementation of the maxi­mum entropy classification algorithm (Berger et al., 1996) to train classification models for each lemma and part-of-speech combination in the train­ing corpus. These models are used to predict WordNet senses for words found in natural text. For lemma and part-of-speech combinations that are not present in the training corpus, the PNNL WSD system defaults to the most frequent Word­Net sense.

Grammatical Dependen­cies Morphosyntactic Features

Table 1. Connexor sample output for the sentence "The engine throbbed into life".

pre.1.num.SG,subj.instrumentality,subj.object, subj.arti-fact, subj.device, subj.engine, subj.motor, subj.whole, subj.entity, subj.machine, subj.pos.N, hassubj., subj.case.NOM, subj.num.SG,

throbbed haspre.1.,haspre.2.,haspost.1., haspost.2., haspost.3., self.throb, self.pos.V, main.,throbbed, self.tense.PAST post.2.case.NOM, post.2.num.SG, hasgoa., pcomp.life, pcomp.state, pcomp.being, pcomp.pos.N, hasgoa.pcomp., goa.pcomp.case.NOM, goa.pcomp.num.SG

. post.3.. Table 2. Feature vector for throbbed in the sen­tence "The engine throbbed into life".

As the example in Table 2 indicates, the combi­nation of contextual, syntactic, and semantic infor­mation types results in a large number of features. Inspection of the training data reveals that some features may be more important than others in es­tablishing word sense assignment for each choice of word lemma. We use a feature selection proce-

pre.1 .instrumentality, pre.l.object, pre.l.artifact, pre.l.device, pre.I.engine, pre.l.motor, pre.1 .whole, pre.1 .entity, pre.l.machine, pre.l.pos.N, pre.l.case.NOM,

dure to reduce the full set of features to the feature subset that is most relevant to word sense assign­ment for each lemma. This practice improves the efficiency of our word sense disambiguation algo­rithm. The feature selection procedure we adopted consists of scoring each potential feature according to a particular feature selection metric, and then taking the best k features.

We choose Information Gain as our feature se­lection metric. Information Gain measures the de­crease in entropy when the feature is given versus when it is absent. Yang and Pederson (1997) report that Information Gain outperformed other feature selection approaches in their multi-class bench­marks, and Foreman (2003) showed that it per­formed amongst the best for his 2-class problems.

To evaluate our approach and feature set, we ran our model on the Senseval-3 English all-words task test data. Using data provided by the Sense-val website, we were able to compare our results for verbs to the top performers on verbs alone. Upali S. Kohomban and Wee Sun Lee provided us with the results file for the Simil-Prime system (Kohomban and Lee, 2005). As reported in Sanfil­ippo et al. (2006) and shown in table 3, our results for verbs rival those of top performers. We had a significant improvement (p-value<0.05) over the baseline of 52.9%, a marginal improvement over the second best performer (SenseLearner) (Mihal-cea and Faruque, 2004), and we were as good as the top performer (GAMBL) (Decadt et al., 2004).

Table 3. Results for verb sense disambiguation on Senseval-3 data, adapted from Sanfilippo et al.

(2006).

Since then, we have expanded our evaluation to all parts of speech. Table 4 provides the evaluation of our system as compared to the three top per­formers on the Senseval-3 data and the baseline. The baseline of 0.631 F-score was computed us­ing the most frequent WordNet sense. The PNNL WSD system performs significantly better than the baseline (p-value<0.05) and rivals the top perform­ers. The performance of the PNNL WSD system relative to the other three systems and the baseline remains unchanged when the unknown sense an­swers (denoted by a 'U') are excluded from the evaluation.

Table 4. Senseval-3 English all-words.

Table 5. Senseval-3 English all-words, No "U".

This was our first opportunity to test our model in a WSD competition. For this competition, we fo­cused our efforts on the fine-grained English all-words task because our system was set up to per­form fine-grained WordNet sense prediction. We are pleased that our system achieved the highest score for this subtask. Our results for the SemEval dataset as compared to baseline are reported in Ta­ble 6. The PNNL WSD system did not assign the unknown sense, 'U', to any word instances in the SemEval dataset.

Although these results are promising, there is still much work to be done. For example, we need to investigate the contribution of each feature to the overall performance of the system in terms of pre­cision and recall. Such a feature sensitivity analysis will provide us with a better understanding of how the algorithm can be further improved and/or made more efficient by leaving out features whose con­tribution is negligible.

Another important point to make is that, while our system shows the best precision/recall results overall, we can only claim statistical relevance with reference to the baseline and results worse than baseline. The size of the SemEval data set (N=465) is too small to establish whether the dif­ference in precision/recall results with the other top systems is statistically significant.

We would like to thank Upali S. Kohomban and Wee Sun Lee for providing us with their Sense-val-3 English all-words task results file for Simil-Prime. Many thanks also to Patrick Paulson, Bob Baddeley, Ryan Hohimer, and Amanda White for their help in developing the word class disam­biguation system on which the work presented in this paper is based.