Research in (Ng et al., 2003; Chan and Ng, 2005) has shown that examples gathered from parallel texts are useful for WSD. In this evaluation, we gath­ered training data from 6 English-Chinese parallel corpora (Hong Kong Hansards, Hong Kong News, Hong Kong Laws, Sinorama, Xinhua News, and English translation of Chinese Treebank), available from the Linguistic Data Consortium (LDC). To gather examples from these parallel corpora, we fol­lowed the approach in (Ng et al., 2003). Briefly, af­ter ensuring the corpora were sentence-aligned, we tokenized the English texts and performed word seg­mentation on the Chinese texts (Low et al., 2005). We then made use of the GIZA++ software (Och and Ney, 2000) to perform word alignment on the paral­lel corpora. Then, we assigned some possible Chi­nese translations to each sense of an English word w. From the word alignment output of GIZA++, we selected those occurrences of w which were aligned to one of the Chinese translations chosen. The En­glish side of these occurrences served as training data for w, as they were considered to have been dis-ambiguated and "sense-tagged" by the appropriate Chinese translations.

We note that frequently occurring words are usu­ally highly polysemous and hard to disambiguate. To maximize the benefits of using parallel texts, we gathered training data from parallel texts for the set of most frequently occurring noun, adjective, and verb types in the Brown Corpus (BC). These word types (730 nouns, 326 adjectives, and 190 verbs) represent 60% of the noun, adjective, and verb to­kens in BC.

The SemCor corpus (Miller et al., 1994) is one of the few currently available, manually sense-annotated corpora for WSD. It is widely used by various systems which participated in the English all-words task of SENSEVAL-2 and SENSEVAL-3, including one of the top performing teams (Hoste et al., 2001; Decadt et al., 2004) which had per­formed consistently well in both SENSEVAL all-words tasks. Hence, we also gathered examples from SemCor as part of our training data.

Besides SemCor, the DSO corpus (Ng and Lee, 1996) also contains manually annotated examples for WSD. As part of our training data, we gath­ered training examples for each of the 70 verb types present in the DSO corpus.

Similar to the top performing supervised systems of previous SENSEVAL all-words tasks, we used the annotated examples available from the SemCor corpus as part of our training data. In gathering ex­amples from parallel texts, a maximum of 1,000 ex­amples were gathered for each of the frequently oc­curring noun and adjective types, while a maximum of 500 examples were gathered for each of the fre­quently occurring verb types. In addition, a max­imum of 500 examples were gathered for each of the verb types present in the DSO corpus. For each word, the examples from the parallel corpora and DSO corpus were randomly chosen but adhering to the sense distribution (proportion of each sense) of that word in the SemCor corpus.

The test data of the two SemEval-2007 tasks we par­ticipated in are based on the WordNet-2.1 sense in­ventory. However, the examples we gathered from the parallel texts and the SemCor corpus are based on the WordNet-1.7.1 sense inventory. Hence, there is a need to map these examples from WordNet-1.7.1 to WordNet-2.1 sense inventory. For this, we rely primarily on the WordNet sense mappings automat­ically generated by the work of (Daude et al., 2000). To ensure the accuracy of the mappings, we per­formed some manual corrections of our own, focus­ing on the set of most frequently occurring nouns, adjectives, and verbs. For the verb examples from the DSO corpus which are based on the WordNet-1.5 sense inventory, we manually mapped them to WordNet-2.1 senses.

Our system employed for the coarse-grained En­glish all-words task was trained with the coarse-

Table 1: Scores for the coarse-grained English all-words task and fine-grained English all-words task, using different sets of training data. SC+DSO refers to using examples gathered from SemCor and DSO corpus. Similarly, SC+DSO+PT refers to using examples gathered from SemCor, DSO cor­pus, and parallel texts.

grained WordNet-2.1 sense inventory released by the task organizers. We obtained a score of 0.825 in this task, as shown in Table 1 under the column SC + DSO + PT. It turns out that among the 16 participants of this task, the system which re­turned the best score was developed by one of the task organizers. Since the score of this system is not considered part of the official scores, our score puts our system in the first position among the par­ticipants of this task. For comparison, the WordNet first sense baseline score as calculated by the task organizers is 0.789. To gauge the contribution of parallel text examples, we retrained our system us­ing only examples gathered from the SemCor and DSO corpus. As shown in Table 1 under the col­umn SC + DSO, this gives a score of 0.817 when scored against the answer keys released by the task organizers. Although adding examples from parallel texts gives only a modest improvement in the scores, we note that this improvement is achieved from a relatively small set of word types which are found to be frequently occurring in BC. Future work can explore expanding the set of word types by automat­ing the process of assigning Chinese translations to each sense of an English word, with the use of suitable bilingual lexicons.

As part of the evaluation results, the task organiz­ers also released the scores of our system on each of the 5 test documents. We show in Table 2 the score we obtained for each document, along with the to­tal number of test instances in each document. We note that our system obtained a relatively low score on the fourth document, which is a Wikipedia entry on computer programming. To determine the rea­son for the low score, we looked through the list of test words in that document. We noticed that the noun program has 20 test instances occurring in that fourth document. From the answer keys released by the task organizers, all 20 test instances belong to the sense of "a sequence of instructions that a computer can interpret and execute", which we do not have any training examples for. Similarly, we noticed that another noun programming has 27 test instances oc­curring in the fourth document which belong to the sense of "creating a sequence of instructions to en­able the computer to do something", which we do not have any training examples for. Thus, these two words alone account for 47 of the errors made by our system in this task, representing 2.1% of the 2,269 test instances of this task.

Our system employed for the fine-grained English all-words task was trained on examples tagged with fine-grained WordNet-2.1 senses (mapped from WordNet-1.7.1 senses and 1.5 senses as described earlier). Unlike the coarse-grained English all-words task, the correct POS tag and lemma of each test instance are not given in the fine-grained task. Hence, we used the POS tag from the mrg parse files released as part of the test data and performed lemmatization using WordNet. We obtained a score of 0.587 in this task, as shown in Table 1. This ranks our system in second position among the 14 partic­ipants of this task. If we exclude parallel text ex­amples and train only on examples gathered from the SemCor and DSO corpus, we obtain a score of 0.578.