Part-of-speech tagger availability varied across the languages included in this sense-disambiguation sys­tem evaluation. Transformation-based taggers (Ngai and Florian, 2001) were trained on standard data for English (Penn Treebank), Swedish (SUC-1 cor­pus) and Estonian (MultextEast corpus). For Czech, an available POS tagger (Hajië and Hladkâ, 1998), which includes lemmatization, was used. The re­maining languages - Spanish, Italian and Basque -were tagged using an unsupervised tagger (Cucerzan

"Many mothers do not even try to toilet train their children until the age of 2 years or later ..."

and Yarowsky, 2000). Lemmatization was per­formed using a combination of supervised and un­supervised methods (Yarowsky and Wicentowski, 2000), and using existing trie-based supervised mod­els for English.

Extracted syntactic relationships in the feature space depended on the keyword's part of speech:

• for verb keywords - the head noun of the verb's object, particle/preposition and object-of-preposition were extracted when available.

• for noun keywords - the headword of any verb-object, subject-verb or noun-noun relationships identified for the keyword.

• for adjective keywords - the head noun modified by the adjective (if identifiable).

These syntactic features were extracted using sim­ple heuristic patterns and regular expressions over the parts-of-speech surrounding the keyword.

The lexical choice task can be cast as a classifica­tion task: training data is given in the form of a set of word-document pairs T = [(wi, Dij), S{j]i ■ (S\j being the sense associated with the document D\j of keyword Wi), labeled with the corresponding gold standard class. The goal is to establish the clas­sification of a set of unlabeled word-document pairs T' = {(wi, D[j)not previously seen in the train­ing data. The training data T is used to estimate class probabilities and then the sense classification is made by choosing the class with the maximum a posteriori class probability:

The disambiguation models used in our exper­iments are feature-based models. A feature is a boolean function defined as fw : F x V —» {0,1}, where F is the entire set of features and V is the document space. An overview of the exploited fea­ture space was given in Section 1.

Our Bayesian and cosine-based models use a com­mon vector representation, capturing both tra­ditional bag-of-words features and the extended Ngram and predicate-argument features in a single data structure.

In these models, a vector is created for each doc­ument in the collection:

D{ = (Ai)J=lj|F| where F is the entire utilized feature space where Cij is the the number of times the feature fj appears in document Di, N{ is the number of words in the document Di and Wj is the weight associated with the feature fj.

To avoid confusion between the same word in mul­tiple feature roles, feature values are marked with their positional type (e.g. children_object, toilet__L, and their_R as distinct from children, toilet and their in unmarked bag-of-words context).

The basic sense disambiguation algorithm pro­ceeds as follows:

1. Vectors in the training data are assigned to classes based on their classification;

2. For each vector in the test data, the a posteriori class distribution is computed as where Cs is the centroid corresponding to the sense S and Sim is the similarity measure used by the algorithm (cosine or Bayes). 3. The sample D is labeled with sense S if S = argmaxP(S'|I>).

The decision list model we used in our system is a non-hierarchical variant of the method of inter­polated decision lists described in Yarowsky (2000). For each feature /; a smoothed log of likelihood ratio (log p^^.j) is computed for each sense Sj, with smoothing based on an empirically estimated func­tion of feature type and relative frequency. Can­didate features are ordered by this smoothed ra­tio (putting the best evidence first), and the re­maining probabilities are computed via the interpo­lation of the global and history-conditional proba­bilities. By utilizing the single strongest-matching evidence in context, non-independent feature spaces combine readily without inflated confidence, and can be mapped to accurate and robust probability esti­mates as shown in Figure 2.

The English task differs slightly from the other lexical-choice tasks in that phrasal verbs are ex­plicitly marked in the training and test data. To make reasonable use of this information, when a phrasal verb is marked, only corresponding phrasal senses are considered; conversely when a phrasal

Figure 2: Mapping between raw confidence scores and classification accuracy for English decision lists verb is not marked, no phrasal senses are considered. Likewise, when a training or test sentence matches a compound noun in the observed sense inventory (e.g. art_gallery%l:06:00::) only the matching phrasal sense(s) are considered unless there is at least one non-phrasal sense tagged in the training data for that compound (indicating the potential for both compositional and non-compositional interpre­tations).

Several classifier combination approaches were inves­tigated in the system development phase. They are outlined below, along with their cross-validated per­formance on the English lexical-sample training data (in Table 1). In each case four individual classifiers were combined: the cosine model, two Bayes models (one based on words and one based on lemmas), and the decision-list model.

The first two model combination approches sim­ply averages the output of the participating clas­sifiers over each candidate sense tag, in terms of P(Sj\Di) and rank(Sj\D{) respectively, with each classifier given an equal vote.

The remaining methods assign potentially vari­able weights to the votes of different classi­fiers. Interestingly, Equal Weighting of all four classifiers slightly outperforms classifier weighting proportional to each model's aggregate accuracy (Performance-Weighted voting), similar to the tech­nique used for classifier combination in part-of-speech tagging in van Halteren et al. (1998). Finally, it was observed that on sentences where decision lists have high model confidence their accuracy exceeds other classifiers. Thus the most effective approach, based on training-data cross validation, was found to be a very basic Thresholded Model Voting:

• If the decisionlist_confidence> 0.985 (an em­pirically selected threshold) then return the out­put of the decision list;

• Otherwise, each system votes for the sense that is most likely under it and, another vote is ob­tained from the most probable class yielded by linear interpolation of the 4 classifiers.

This simple top-performing approach was utilized in the evaluation system, and is reasonably close to the performance of an Oracle upper bound for classifier combination (using the output of the single best clas­sifier on each test instance - unknowable in practice).

Accuracy

Because of the importance of morphological analy­sis in a highly inflected language such as Estonian, a lemmatizer based on Yarowsky and Wicentowski (2000) was first applied to all words in the train­ing data (and, at evaluation time, the test data). For each lemma, the P (sense]lemma) distribution was measured on the training data. For all lem­mas exhibiting only one sense in the training data, this sense was returned. Likewise, if there was in­sufficient data for word-specific training (the sum of the minority sense examples for the word in training data was below a threshold) the majority sense in training was returned for all instances of that lemma. In the remaining cases where a lemma had more than one sense in training, with sufficient minority exam­ples to adequately be modeled, the generic JHU lex­ical sample sense classifier was trained and applied.

Czech is another example of a highly inflected lan­guage. A part-of-speech tagger and lemmatizer kindly provided by Jan Hajië of Charles Univer­sity (Hajië and Hladka, 1998) was first applied to the data. Consistent with the spirit of evaluating sense disambiguation rather than morphology, the JHU system focused on those words where more than one sense was possible for a root word (e.g.