The problem of supervised word sense disam­biguation (wsd) has been approached using many different classification algorithms, includ­ing naive Bayes, decision trees, decision lists, and memory-based learners. While it is un­questionable that certain algorithms are better suited to the wsd problem than others (for a comparison, see Mooney (1996)), it seems to be the case that, given similar features as input, various algorithms do not behave dramatically differently. This was seen in the Senseval-2 re­sults where a large fraction of the systems had scores clustered in a fairly narrow region.

We began building our system with 23 su­pervised wsd systems, each submitted by a student taking the natural language processing course (CS224N) at Stanford University. Stu­dents were free to implement whatever wsd

This paper is based on work supported in part by the National Science Foundation under Grants IIS-0085896 and IIS-9982226, by an NSF Graduate Fellowship, and by the Research Collaboration between NTT Communi­cation Science Laboratories, Nippon Telegraph and Tele­phone Corporation and CSLI, Stanford University.

Cross I Validation J

First-tier Classifiers

method they chose. While most implemented variants of naive Bayes, some implemented a range of other methods, including n-gram mod­els, vector space models, and even memory-based learners. Although none of these systems alone would have produced more than middle-level performance on the Senseval-2 task, we decided to investigate how they would behave in combination.

In section 2, we discuss the first-tier classifiers in greater depth and describe our methods of combination. Section 3 discusses performance, analyzing what benefit was found from combi­nation, and when. We also discuss aspects of the component systems which substantially in­fluenced overall performance.

Figure 1 shows the high-level organization of our system. First, each of the 23 classifiers is run with 5-fold cross-validation on the train­ing data. Classifiers are ranked, for each word, based on their held-out accuracy. In any given run of the system, for some ft, the top k clas­sifiers are kept, while lower-ranking classifiers are discarded. These remaining classifiers are combined by one of three methods.

• Majority voting: The sense output by the most classifiers is chosen. Ties are broken in favor of the highest-ranked classifier.

Table 1 shows the results per word, and table 2 shows results by part-of-speech. A wide range of models are chosen, and the chosen model usu­ally beats the best single classifier for that word, on average by 1.9%. The improvement over the globally best single classifier is even greater.

Notably, if we use the test data as an oracle to chose the best combination method, rather than relying on held-out data, accuracy jumps by an average of 3.6%. This gap is dramati­cally larger than the gap between the top scor­ing systems for this S ense val-2 task. While the knowledge of actual best performance is ob­viously not available, one might suspect that a more sophisticated or better-tuned method of

Figure 2: The accuracy of the various combina­tion methods as the number of component systems changes. The best single classifier is chosen per word from held-out data and averaged. Chosen combina­tion is also selected per word and averaged.

choosing a final combination model might lead to significant improvement.

Figure 2 shows how the three combination methods' average scores varied with the num­ber of component classifiers used. A critical as­pect of our system is that the first-tier classi­fiers are very diverse, not only in implementa­tion but also in performance. Initially, accuracy increases as added classifiers bring value to the ensemble. However, as lower-quality classifiers are added in, the better classifiers are steadily drowned out. The weighted vote and maxi­mum entropy combinations are less affected by low-quality classifiers than the majority vote, being able to suppress them with low weights. Still, majority vote was a good method to have around for words where weights could not be usefully set by the other methods.

When combining heterogeneous classifiers, one would like to know when and how the combination will outperform the individuals. One factor is how complementary the mistakes of the individual classifiers are. We can mea­sure this complementarity by averaging, over all pairs of classifiers, the fraction of errors that pair has in common. This gives average pairwise error independence. Another factor is the difficulty of the word being disambiguated. A high most-frequent sense baseline means that there is little room for improvement by combining classifiers. Figure 3 shows, for the overall top 7 first-tier classifiers, the absolute gain between their average accuracy and the accuracy of their majority. The x-axis is the dif­ference between the pairwise independence and the baseline accuracy. The pattern is loose, but clear. The gain increases with complementarity and decreases with the baseline.

We have demonstrated that the combination of a number of heterogeneous classifiers can lead to a substantial performance increase over the individual classifiers. Our system is robust to both the wide range of accuracy of the first-tier classifiers and to sparsity of training data when building the second-tier classifier. The system's overall accuracy is high, despite the medium level of accuracy of the component systems.

We wish to thank the following people for contributing their classifiers to the Stanford-CS224N system: Zoe Abrams, Jenny Berglund, Dmitri Bobrovnikoff, Chris Callison-Burch, Marcos Chavira, Shipra Dingare, Elizabeth Douglas, Sarah Harris, Ido Milstein, Jyotir-moy Paul, Soumya Raychaudhuri, Paul Ruhlen, Magnus Sandberg, Adil Sherwani, Philip Shi-lane, Joshua Solomin, Patrick Sutphin, Yuliya Tarnikova, Ben Taskar, Kristina Toutanova, Christopher Unkel, and Vincent Vanhoucke.