In written texts, words which have multiple senses can be classified into two categories; homonyms and polysemous words. Generally speaking, while homonymy sense distinction is quite clear, polysemy sense distinction is very subtle and hard. English texts contain many homonyms. On the other hand, Japanese texts in which most content words are written by ideograms rarely contain homonyms. That is, the main target in Japanese WSD is polysemy, which makes Japanese WSD task setup very hard. What sense distinction of polysemous words is reasonable and effective heavily de­pends on how to use it, that is, an application of WSD.

Considering such a situation, in addition to the ordinary dictionary task we organized an­other task for Japanese, a translation task, in which word sense is defined according to trans­lation distinction. Here, we set up the task as­suming the example-based machine translation paradigm (Nagao, 1981). That is, first, a trans­lation memory (TM) is constructed which con­tains, for each Japanese head word, a list of typ­ical Japanese expressions (phrases/sentences) involving the head word and an English trans­lation for each (Figure 1). We call a pair of Japanese and English expressions in the TM as a TM record. Given an evaluation document containing a target word, participants have to submit the TM record best approximating that usage.

Alternatively, submissions can take the form of actual target word translations, or transla­tions of phrases or sentences including each tar­get word. This allows existing rule-based ma­chine translation (MT) systems to participate in the task, and we can compare TM based sys­tems with existing MT systems.

For evaluation, we distributed newspaper ar­ticles. The number of target words was 40, and 30 instances of each target word were provided, making for a total of 1,200 instances.

The translation memory (TM) was constructed in two steps:

1. By referring to the KWIC (Key Word In Context) of a target word, its typical Japanese expressions are picked up by lex­icographers.

2. The Japanese expressions are translated by a translation company.

KWIC was made from the nine years vol­ume of Mainichi Newspaper corpus. They are morphologically analyzed and segmented into phrase sequences, and then the 100 most fre­quent phrase uni-grams, bi-grams (two types; the target word is in the first phrase or the sec­ond phrase) and tri-grams (the target word is in the middle phrase) are provided to lexicogra­phers (Figure 2).

As a gold standard data of the task, 40 target words were chosen out of 320 TM words. Con­sidering the possible comparison of the trans­lation task and the dictionary task, 40 target words were fully overlapped with 100 target words of the dictionary task.

In the Japanese dictionary task, target words are classified into three categories according to the difficulty (difficult, intermediate, easy), based on the entropy of word sense distri­bution in the training data of the dictionary task(Shirai, 2001). 40 target words of the tranj lation task consists of 20 nouns and 20 verbs: difficult nouns and verbs, 10 intermediate nom and verbs, and 5 easy nouns and verbs.

For each target word, 30 instances were ch< sen from Mainichi Newspaper corpus (in tot. 1,200 instances) and they are also overlappe with the dictionary task. Since the dictionai task uses 100 instances for each target wor< the translation task used 1st, 4th, 7th, ... 90t instances of the dictionary task.

As a gold standard data, zero or more a{ propriate TM records were assigned to each ii stance by the same translation company Aj propriate TM records were classified into th following three classes:

© : A TM record which can be used t translate the instance. POS, tense, plura singular, and subtle nuance do not necei sarily match.

O : If the instance is considered alone, th English translation is correct, but usin the TM record in the given context is nc so good, for example, making very round about translation.

In the Japanese translation task, 9 systems from 7 organizations submitted the answers. The characteristics of the systems are summarized as follows:

• AnonymX, Anonym Y Commercial, rule-based MT systems.

• CRL-NYU (Communications Research Laboratory & New York Univ.)

TM records are classified according to the English head word, and each cluster is supplemented by several corpora. The system returns a TM record when the similarity between a TM record and an input sentence is very high. Otherwise, it returns the English head word of the most similar cluster by using several machine learning techniques.

• Ibaraki (Ibaraki Univ.)

A training data was constructed manually from newspaper articles, 170 instances for each target word. Features were collected in 7-word window around the target word, and decision list method was used for learn­ing.

• Stanford-Titechl (Stanford Univ. & Tokyo Institute of Technology)

The system selects the appropriate TM record based on the character-bigram-based Dice's coefficient. It also utilized the context of the other target word instances in the evaluation text.

• AnonymZ

A sentence (TM records for learning, and an input for testing) is morphologically an­alyzed and converted into a semantic tag sequence, and maximum entropy method was used for learning.

The system selects the most similar TM record based on the cosine similarity be­tween context vectors, which were con­structed from semantic features and syn­tactic relations of neighboring words of the target word.

The system selects the most similar TM record by bottom-up, shared-memory based matching algorithm.

• Stanford-Titech2 (Stanford Univ. & Tokyo Institute of Technology)

The system selects the appropriate TM record based on the case-frame-based sim­ilarity, using NTT Goi-Taikei thesaurus.

The results of all systems are shown in Fig­ure 3. The left bar charts indicate the accuracy based on the lenient evaluation (©, O and A in TM selection and © and O in MT are re­garded as correct); the right bar charts indicate the accuracy based on the strict evaluation (© is only regarded as correct both in TM selection and MT). Note that since the TM does not have a hierarchical structure, there is no evaluation options such as fine, coarse, and mixed.

Figure 4 shows scores for nouns and verbs separately, and Figure 5 shows scores for dif­ficult/intermediate/easy words. Both of them were evaluated by the lenient criteria.

In these figures, "Agreement" and "Baseline" were as described in the previous section. When the system judges that there is no appropri­ate TM record for an instance, it can return "UNASSIGNABLE". In that case, if there is no appropriate TM record assigned in the gold standard data, the answer is regarded as cor­rect.

Among TM selection systems, systems using some extra learning data outperformed other systems just using the TM. The comparison be­tween TM selection systems and MT systems is not easy, but the result indicates the effec­tiveness of the accumulated know-how of MT systems. However, the performance of the best TM selection system is not so different from MT systems, which indicates the promising future ol TM based techniques.

This paper described an overview of senseval-2 Japanese translation task. The data used ir this task are available at Senseval-2 web site, We hope this valuable data helps improve WSE and MT systems.

Acknowledgment

I wish to express my gratitude to Mainichi Newspapers for providing articles. I would alsc like to thank Prof. Takenobu Tokunaga (Tokyc Institute of Technology) and Prof. Kiyoaki Shi-rai (JAIST) and Dr. Kiyotaka Uchimoto (CRL) for their valuable advise about task organiza­tion, Yuiko Igura (Kyoto Univ.) and Intel Group Corp. for data construction, and all par­ticipants to the task.