WSD is treated in this system as a classification task, where the ith sense (W#i) of a word (W) is classified as the correct sense tag (Mi), given the word W and usually some surrounding context. In the Senseval-2 English all-words task, all ambigu­ous content words (nouns, verbs, adjectives, and ad­verbs) are to be classified with a sense tag from the WordNet 1.7 lexical database (Miller, 1990). For example, the words "great", "devastated'"', and "re­gion" in the sentence "The great hurricane devas­tated the region" are classified with the correct sense tags 2, 2, and 2, respectively. We will refer to this task using the following notation:

where S is the input sentence, and M is the se­mantic tag assigned to each word. While a context larger than the sentence S can be and is used in our model, we will refer to the context as S. In this for­mulation, each word W{ in the sentence is treated as a random variable M{ taking on the values {l..Ni}, where Ni is the number of senses for the word WV Therefore, we wish to find instantiations of M such that P(M\S) is maximized.

To make the computation of Mbest(S) more tractable, it can be decomposed into Mtest(S) ^ arg max(UiP(Mi\S)): where it is assumed that each word can be disambiguated independently. However, this assumption does not always hold, since disambiguating one word often affects the sense assignment of another word within the same sentence. Alternatively, the process can be modeled as a Markov model, e.g., Mbe8t(S) « arg max(IliP(Wi\Mi) x P(Mi\Mi.l)).' While the Markov model requires fewer param­eters, it is unable to capture the long-distance dependencies that occur in natural languages. Although the first decomposition better captures these dependencies, computing P(Mi\S) using the full sentential context is rarely used, since the number of parameters required grows exponen­tially with each added context. Therefore, one can further simplify this model by narrowing the context to 2n number of surrounding words, i.e.. P(Mi\S) * P(Mi\Wi-n,.-W-uWi+u..Wi+n). However, narrowing the context also discards long-distance relationships, making it closer to a Markov model.

Without having to artificially limit the size of the context, another possible simplification is to make independence assumptions between the con­text words. In the simplest case, every context is assumed to be independent from each other, i.e., P(Mi\S) « ïlxP(Mi\Wx), like a Naive Bayes classi­fier. While the parameters can be simply established by a set of bi-grams, the independence assumption is often too strong and thus negatively affects accu­racy. The difficulty is in choosing the context that would maximize the accuracy while allowing for re­liable parameter estimation from training data.

In our model, we aim to strike this balance by choosing the context words based on structural in­formation, rather than positional information. The hypothesis is that an ambiguous word is probabilisti­cally dependent on its structurally related words and is independent of the rest of the sentence. There­fore, long-distance dependencies can still be cap­tured, while the context is kept small. Further-

Figure 1: An example of a Bayesian network and the probability tables at each node that dehne the relationships between a node and its parents. The equation at the bottom shows how the distribution is represented by the network.

more, each word is not classified independently of each other, but is computed as one single query that determines all of the sense assignments that result in the highest overall probability for the whole sen­tence. Therefore, our model is a combination of the decompositions described above, by selectively mak­ing independence assumptions on a per-word basis to best model P(Mi\S), while computing Mbest(S) in one query to allow for interactions between the word senses M;.

The first step in constructing a Bayesian network is to determine its structure G, which defines each node's dependency relationship with the rest of the network. In our model, we are making these inde­pendence assumptions based on the structural re­lationships between words. Specifically, given the sentence S and its parse tree, we automatically con­struct a graph G by first creating a node Mi for each word Wi. This process is best illustrated by the ex­ample shown in Figure 2. For each node Mi, an edge is added to node Mx, where Mx is the head word of a verb phrase (board -> approved), the target of the modifier Mi (today's -» meeting), or the preposition Mx where Mi is the target or a constituent of the prepositional phrase (approved -> at). One can see that if the parse tree is known, the construction of network G is straight-forward. For Senseval-2, the

Figure 2: An example of a Bayesian network repre­senting the inter-dependencies between the words of the sentence "The board approved its acquisition by ABC Co. of New York at today's meeting." parse trees provided in Treebank format were used to build the Bayesian networks' structure.

Once the structure of the Bayesian network is de­termined, the context, i.e., the parents PAi: for each word is established. Using the same example, the context for the word "approved" is "board" and "ac­quisition" , and for "at" it is "approved" and "meet­ing" . Our hypothesis is that these structurally re­lated words, among all of the words within the sen­tence, provide the best contextual information for sense disambiguation. That is, given that the par­ents' word form WpAi and senses MpAi are known, the sense assignment for Mi is independent of all other words in the sentence. This is, of course, a simplification due to the constraint in minimizing the context. However, the use of Bayesian networks allows for easy expansion of context by establish­ing more edges between nodes or adding new nodes, provided that the parameters can be determined re­liably.

Once G is determined, the CPTs at each node need to be quantified. Using the same exam­ple above, for the word "approved", its CPT F(apprœed#i\board#i.acquisition#i) would con­tain 2 (number of senses for "approved") x9x4 = 72 entries. For a word without any parents, such as "today's", its priors are used.

While determining the network structure is rel­atively simple, establishing accurate parameters is quite difficult, even with a small context such as ours. Due to the limited size of SemCor, our only labeled training data, we used additional sources to quantify and smooth these parameters. Primar­ily we deployed the same techniques used in our Bayesian Hierarchical Disambiguator (BHD) model (Chao and Dyer, 2000), which uses Internet search engines to estimate parameters based on permuta­tions of synonym words, a method first introduced by Mihalcea and Moldovan (1999). These param­eters are then smoothed by training data obtained from SemCor. The details of BHD are omitted here due to space constraints.

Although BHD was only used on adjective-noun pairs, the same principles are used to quantify all of the CPTs in this model. While only one hierarchi­cal network is needed to smooth the parameter for adjective-noun pairs, up to three hierarchical net­works are used for each potential parent. Since the smoothing computation is very efficient, being linear in the depth of the network, these additions did not impact the speed of the model. The majority of the time was used to query the Internet search engine.

The BHD model, however, did use additional training data that was collected from the Word-Net glosses and manually annotated. While it re­sulted in good accuracy, this was obviously not an option for Senseval-2. Instead, the example sen­tences from WordNet are extracted and first tagged by Brill's POS tagger (Brill, 1995). Then an ex­perimental parser and our WSD system were used to parse and disambiguate the sentences to extract additional training data. For example, for the 6th sense of adjective "great", the pair "great#6 time" is extracted from the example sentence fragment "had a great time at the party" and automatically dis­ambiguated. The labeled pair is then added to the training set for great#6.

Lastly, the priors in this model are determined directly from SemCor's occurrence statistics and estimated using Maximum Likelihood Estimation (MLE). This is another simplification over the BHD model, where the priors were determined using the hundred most frequent adjective-noun pairs culled from the Internet and then manually classified. It is well known that MLE is inaccurate when the num­ber of events are low, as is in this case when rarer senses often have only single occurrences.

Nevertheless, we are able to address both of the manual steps used in the BHD model with auto­mated processes. However, it is our belief that they are also the weakest part of our model and contribute the most to the errors.

With both the structure G and the parameters established, the query we pose is to compute the instantiations for each random variable that would result in the highest joint probability, i.e., arg maxP(Mi\S). This is computed easily using the Maximum A Posteriori (MAP) query. This was im­plemented using the Joint Tree algorithm (Darwiche, 1995) and can be computed in 0(|c|™) time, where |c| is the size of the variable (number of senses), and w is the tree width. Given that our networks are sparsely connected, w is usually close to 3, the aver­age number of parents 4- 1.

The advantage of using the MAP query is that it computes variable instantiations that will maxi­mize the overall probability across the whole sen­tence, rather than the localized context. Furthermore, the resulting instantiation and probability is guaranteed to be maximum. So given the indepen­dence assumptions made on the context and the es­timated parameters, MAP will always produce the most probable sense tagging for every word in the sentence.

It is well known that word senses are often influ­enced by contexts larger than the sentence, such as surrounding sentences or even the whole passage. We experimented with the One-sense-per-discourse (OSPD) hypothesis (Yarowsky, 1993) by applying the probabilities described in Stetina et al. (1998) to words that have previously appeared in the text and thus have been disambiguated. The only mod­ification needed to our model described thus far is to apply OSPD probabilities, which is dependent on the distance between the sentences, to each sense of a re-occurring word before the MAP query. It is our observation that this incarnation of the OSPD hypothesis, chosen for its ease of implementation, tends to propagate erroneous sense tagging from ini­tial sentences to the remainder of the passage. A better approach would be to determine the one sense that would maximize the consensus across the whole passage, as well as within each individual sentence. How this can be achieved efficiently in a probabilistic framework is currently being investigated.

For Senseval-2, we submitted three models for comparison, which differ by their methods of pa­rameter estimation. Model 2 uses the training data from SemCor and Hierarchical networks to smooth the parameters from Internet search engines. Model 3 incorporates additional training data gathered au­tomatically from the WordNet glosses. Lastly, model 1 combines all training data, as well as the OSPD hypothesis.

One can see that the model that uses all of the available data achieved best accuracy (model 1) but unfortunately also had the lowest recall due to the added complexity. Some highly polysemous words were omitted due to time and memory constraints. Between the 2 training sets, it was unfortunate that the addition of the automatically generated training set reduced the accuracy slightly, mainly due to the noisy data produced by our experimental system.

Nevertheless, we believe that there is a wealth of information contained within WordNet's glosses. Since one of our aims is to use as much automated processing as possible, we are focusing on improving the accuracy of the automatically generated train­ing data. Our goal is that as the WSD accuracy of our system improves, so will the reliably of these automatically generated training data. Having im­proved training data will further improve the sys­tem's WSD accuracy, i.e., a bootstrapping system. We are at the initial stage of this process, but some fundamental problems such as reliable POS tagging and parsing of sentence fragments need to be ad­dressed first. Furthermore, parameter estimation based on Internet statistics might prove to be too noisy, so we are currently focusing on learning al­gorithms such as Expectation Maximization to tune the parameters. Lastly, if our context is found to be too limited, additional features can be added to the Bayesian networks to improve the classification accuracy.