It is well known that WSD constitutes one of the hardest problems in Natural Language Pro­cessing, yet is a necessary step in a large range of applications including machine translation, knowledge acquisition, coreference, information retrieval and others. This motivates a continu­ously increasing number of researchers to develop WSD systems and devote time to finding solu­tions for this challenging problem.

The system presented here was initially de­signed for the semantic disambiguation of all words in open text. The Senseval competitions created a good environment for supervised sys­tems and this encouraged us to improve our sys­tem with the capability of incorporating larger training data sets when provided.

There are two important modules in this sys­tem. The first one uses pattern learning relying on large sense tagged corpora to tag all words in open text. The second module is triggered only for the words with large training data, as was the case with the words from the lexical sample tasks. It uses an instance based learning algorithm with active feature selection.

To our knowledge, both pattern learning and active feature selection are novel approaches in the WSD field, and they led to very good results during the Senseval-2 evaluation exercise.

The WSD algorithm used in this system has the capability of tagging words when no specific sense tagged corpora is available, automatically scaling up to larger training data when provided.

Due to space constraints, we will not be able to give a detailed description of the system. How­ever we try to gain space and replace one thou­sand words with a picture: Figure 1 shows an overview of the system architecture. It illustrates the two main components, namely pattern learn­ing from available sense tagged corpora and dic­tionary definitions and instance based learning with active feature selection. The two modules are preceded by a preprocessing phase which in­cludes compound concept identification, and fol­lowed by a default phase that assigns the most frequent sense as a last resort, when no other previous methods could be applied. The shaded areas in Figure 1 are specific for the case when larger training data sets are available.

During the preprocessing stage, SGML tags are eliminated, the text is tokenized, part of speech tags are assigned using Brill tagger (Brill, 1995), and Named Entities (NE) are identified with an in-house implementation of an NE recognizer. To identify collocations, we determine sequences of words that form compound concepts defined in WordNet.

In the second step, patterns are learned from WordNet, SemCor and GenCor, which is a large

FREE UNTAGGED TEXT

PREPROCESSING

eliminate SGMLtags tokenization part of speech tagging Named Entity recognitior

CONCEPT IDENTIFIER

- identify compound concepts - identify proper nouns with a semantic role (NE learn patterns from SemCor and WordNet GenCor

additional sense tagged corpora if additional training data available, validate patterns keep only error free

instance based learning with active feature selection - assign the most frequent sense (e.g. first sense in WordNet

sense tagged corpus automatically built via a set of heuristics. If additional training data is avail­able, patterns are filtered through a validation process. Practically, the patterns are applied on the sense tagged data and we keep only those with no counter-examples found in the training sets.

The third step consists of a learning mecha­nism with active feature selection. This step is initiated only for those words with a sufficiently large number of examples, as was the case with the words in the Senseval lexical sample tasks.

This module is intended for solving the semantic ambiguity of all words in open text. To this end, we build disambiguation patterns using SemCor, WordNet and GenCor. Several processing steps were required to transform the first two resources into a useful corpus for our task. Moreover, these lexical resources coupled with a set of heuristics were used as seeds for generating a new sense tagged corpus called GenCor. SemCor The senseval-2 English tasks have de­cided to use the WordNet 1.7 sense inventory, and therefore we had to deal with the task of map­ping SemCor senses, which were assigned using an earlier version of WordNet, to the correspond­ing senses in WordNet 1.7. When a word sense from WordNet 1.6 is missing we assign a default sense of 0. WordNet The main idea in generating a sense tagged corpus out of WordNet is very simple. It is based on the underlying assumption that each ex­ample pertains to a word belonging to the current synset, thereby allowing us to assign the correct sense to at least one word in each example. For instance, the example given for mother#4 is "ne­cessity is the mother of invention", and the word mother can be tagged with its appropriate sense. GenCor is a generated sense tagged corpus, con­taining at the moment about 160,000 tagged words, which uses as seeds the sense tagged ex­amples from SemCor and WordNet, as well as some of the principles for generating sense tagged corpora presented in (Mihalcea and Moldovan, 1999). Due to space limitations we cannot present here the methodology for creating this corpus. A thorough description is provided in (Mihalcea, 2001).

Learning mechanisms for disambiguating word senses have a long tradition in the WSD field. For our system, we have decided for an instance based algorithm with information gain feature weight­ing. The reasons for this decision are three fold: first, it has been advocated that forgetting ex­ceptions is harmful for language learning applica­tions (Daelemans et al, 1999), and instance based algorithms are known for their property of taking into consideration every single training example when making a classification decision; secondly, instance based learning algorithms have been suc­cessfully used in WSD applications (Veenstra et al, 2000); finally, this type of algorithms are ef­ficient in terms of training and testing time. We have initially used the MLC++ implementation, and later on switched to Timbl (Daelemans et al., 2001).

Even more important than the choice of learn­ing methodology is the selection of features em­ployed during the learning process. There are several features recognized as good indicators of word sense, including the word itself (CW) and its part of speech (CP), surrounding words and their parts of speech (CF), collocations (COL), syntactic roles, keywords in contexts (SK). More recently, other possible features have been inves­tigated: bigrams in context (B), named entities (NE), the semantic relation with the other words in context, etc.

Our intuition was that different sets of features have different effects depending on the ambiguous word considered. Feature weighting was clearly proven to be an advantageous approach for a large range of applications, including WSD. Still, weights are computed independently for each fea­ture and therefore this strategy does not always guarantee to provide the best results.

For our system, we actively select features us­ing a forward search algorithm. In this way, we practically generate meta word experts. Each word will have a different set of features that will eventually lead to the best disambiguation accu­racy.

Using this approach, we combine the advan­tages of instance based learning mechanisms that have the nice property of anot forgetting ex­ceptions", with an optimized feature selection scheme. One could argue that decision trees have the capability of selecting relevant features, but it has been shown (Almuallim and Dietterich, 1991) that irrelevant features significantly affect the performance of decision trees as well.

The algorithm for active feature selection is sketched in Figure 2. It is worth mentioning that in step 2, the training and testing corpora are ex­tracted for each ambiguous word. This means that examples pertaining to the word "dress down" are separated from the examples for the single word "dress".

1. Generate pool of features PF = {Fi}. Initialize the set of selected features with the empty set SF={0}.

2. Extract training and testing corpora for the given target ambiguous word.

3.1. Run a 10-fold cross validation on the training set; each example in the training set contains the features in SF and the feature F{.

3.2. Determine the feature Fi leading to the best accuracy.

3.3. Remove Fi from PF and add it to SF.

4. Repeat step 3 until no improvements are ob­tained.

The pool PF contains a large number of fea­tures, including those previously mentioned CW, CP, CF, COL, SK, B, NE, as well as other fea­tures like the noun before and after (NB, NA), head of the noun phrase, surrounding verbs, and others.

The overall performance of the system in the En­glish all words task was 69% for fine-grained scor­ing, respectively 69.8% for coarse-grained scoring (SMUaw). In the English lexical sample task, we obtained 63.8% for fine-grained scoring, respec­tively 71.2% for coarse-grained scoring (SMUls). These results ranked our system before deadline as the best performing for both tasks. Discussion

There were several interesting cases encoun­tered in the Senseval-2 data, justifying our ap­proach of using active feature selection. The in­fluence of a feature greatly depends on the tar­get word: a feature can increase the precision for a word, while making things worse for another word. For example, a word such as free does not benefit from the surrounding keywords (SK) feature, whereas colourless gains almost 7% in pre­cision when this feature is used. free.a[CW CP CF SK] 57.85% free.a[CW CP CF ] -> 63.57%

colourless.o[CW CP CF ] -> 78.57% colourless.a[CW CP CF SK] -> 85.71% Another interesting example is constituted by the noun chair, which was disambiguated with high precision by simply using the current word (CW) feature. This is explained by the fact that the most frequent senses are Chair meaning person and chair meaning furniture, and there­fore the distinction between lower and upper case spellings makes the distinction among the differ­ent meanings of this word.

We have also tested the system on the Senseval-1 data, and performed the disam­biguation task in respect with Hector definitions, as required by the first disambiguation exercise. The overall result achieved on this data was higher than the one reported by the best per­forming system. Besides proving the validity of our approach, this fact also proved that our sys­tem is not tight in any ways to the sense inventory or data format employed.

Pattern learning and active feature selection are new approaches in the WSD field. They have been implemented in a system that participated in the senseval-2 competition, with an excel­lent performance in both English all words and English lexical sample tasks.