As part of the SemEval-2007 evaluation exercise, we organized an English lexical sample task for word sense disambiguation (WSD), where the sense-annotated examples were semi-automatically gath­ered from word-aligned English-Chinese parallel texts. Two tracks were organized for this task, each gathering data from a different corpus. In this paper, we describe our motivation for organizing the task, our task framework, and the results of participants.

Past research has shown that supervised learning is one of the most successful approaches to WSD. However, this approach involves the collection of a large text corpus in which each ambiguous word has been annotated with the correct sense to serve as training data. Due to the expensive annotation pro­cess, only a handful of manually sense-tagged cor­pora are available.

An effort to alleviate the training data bottle­neck is the Open Mind Word Expert (OMWE) project (Chklovski and Mihalcea, 2002) to collect sense-tagged data from Internet users. Data gath­ered through the OMWE project were used in the SENSEVAL-3 English lexical sample task. In that task, WordNet-1.7.1 was used as the sense inven­tory for nouns and adjectives, while Wordsmythwas used as the sense inventory for verbs.

Another source of potential training data is par­allel texts. Our past research in (Ng et al., 2003; Chan and Ng, 2005) has shown that examples gath­ered from parallel texts are useful for WSD. Briefly, after manually assigning appropriate Chinese trans­lations to each sense of an English word, the English side of a word-aligned parallel text can then serve as the training data, as they are considered to have been disambiguated and "sense-tagged" by the appropri­ate Chinese translations.

Using the above approach, we gathered the train­ing and test examples for our task from parallel texts. Note that our examples are collected without manu­ally annotating each individual ambiguous word oc­currence, allowing us to gather our examples in a much shorter time. This contrasts with the setting of the English lexical sample task in previous SENSE-VAL evaluations. In the English lexical sample task of SENSEVAL-2, the sense tagged data were cre­ated through manual annotation by trained lexicog­raphers. In SENSEVAL-3, the data were gathered through manual sense annotation by Internet users.

In the next section, we describe in more detail the process ofgathering examples from parallel texts and the two different parallel corpora we used. We then give a brief description of each of the participating systems. In Section 4, we present the results obtained by the participants, before concluding in Section 5.

To gather examples from parallel corpora, we fol­lowed the approach in (Ng et al., 2003). Briefly, af­ter ensuring the corpora were sentence-aligned, we tokenized the English texts and performed word seg­mentation on the Chinese texts (Low et al., 2005). We then made use of the GIZA++ software (Och and Ney, 2000) to perform word alignment on the paral­lel corpora. Then, we assigned some possible Chi­nese translations to each sense of an English word w. From the word alignment output of GIZA++, we selected those occurrences of w which were aligned to one of the Chinese translations chosen. The En­glish side of these occurrences served as training data for w, as they were considered to have been dis-ambiguated and "sense-tagged" by the appropriate Chinese translations. The English half of the par­allel texts (each ambiguous English word and its 3-sentence context) were used as the training and test material to set up our English lexical sample task.

Note that in our approach, the sense distinction is decided by the different Chinese translations as­signed to each sense of a word. This is thus similar to the multilingual lexical sample task in SENSEVAL-3 (Chklovski et al., 2004), except that our training and test examples are collected with­out manually annotating each individual ambiguous word occurrence. The average time needed to assign Chinese translations for one noun and one adjective is 20 minutes and 25 minutes respectively. This is a relatively short time, compared to the effort other­wise needed to manually sense annotate individual word occurrences. Also, once the Chinese transla­tions are assigned, more examples can be automat­ically gathered as more parallel texts become avail­able.

We note that frequently occurring words are usu­ally highly polysemous and hard to disambiguate. To maximize the benefits of our work, we gathered training data from parallel texts for a set of most fre­quently occurring noun and adjective types in the Brown Corpus. Also, similar to the SENSEVAL-3

English lexical sample task, we used WordNet-1.7.1 as our sense inventory.

Three teams participated in the Web corpus track of our task, with each team employing one system. There were no participants in the LDC corpus track, possibly due to the licensing issues involved. All participating systems employed supervised learning and only used the training examples provided by us.

As all participating systems gave only one answer for each test example, recall equals precision and we will only report micro-average recall on the Web corpus track in this section.

Table 2 gives the overall results obtained by each of the systems when evaluated on all the test exam­ples of the Web corpus. We note that all the par­ticipants obtained scores which exceed the baseline heuristic of tagging all test examples with the most

Table 2: Overall micro-average scores of the participants and the most frequent sense (MFS) baseline.

Table 3: Micro-average scores of the most frequent Table 4: Micro-average scores of the most frequent sense baseline and the various participants on each sense baseline and the various participants on each noun. adjective.

frequent sense (MFS) in the training data. This sug­gests that the Chinese translations assigned to senses of the ambiguous words are appropriate and provide sense distinctions which are clear enough for effec­tive classifiers to be learned.

In Table 3 and Table 4, we show the scores ob­tained by each system on each of the 20 nouns and 20 adjectives. For comparison purposes, we also show the corresponding MFS score of each word. Paired t-test on the results of the top two systems show no significant difference between them.

We organized an English lexical sample task using examples gathered from parallel texts. Unlike the English lexical task of previous SENSEVAL evalua­tions where each example is manually annotated, we only need to assign appropriate Chinese translations to each sense of a word. Once this is done, we auto­matically gather training and test examples from the parallel texts. All the participating systems of our task obtain results that are significantly better than the most frequent sense baseline.

Yee Seng Chan is supported by a Singapore Millen­nium Foundation Scholarship (ref no. SMF-20041076).