This paper reports an overview of the SemEval2 Japanese Word Sense Disambiguation (WSD) task. It can be considered an extension of the SENSEVAL-2 Japanese monolingual dictionary-based task (Shirai, 2001), so it is a lexical sam­ple task. Word senses are defined according to the Iwanami Kokugo Jiten (Nishio et al., 1994), a Japanese dictionary published by Iwanami Shoten. It was distributed to participants as a sense inven­tory. Our task has the following two new charac­teristics:

1. All previous Japanese sense-tagged corpora were from newspaper articles, while sense-tagged corpora were constructed in English on balanced corpora, such as Brown corpus and BNC corpus. The first balanced corpus of contemporary written Japanese (BCCWJ corpus) is now being constructed as part of a national project in Japan (Maekawa, 2008), and we are now constructing a sense-tagged corpus based on it. Therefore, the task will use the first balanced Japanese sense-tagged corpus.

Because a balanced corpus consists of docu­ments from multiple genres, the corpus can be divided into multiple sub-corpora of a genre. In supervised learning approaches on word sense disambiguation, because word sense distribution might vary across different sub-corpora, we need to take into account the genres of training and test corpora. There­fore, word sense disambiguation on a bal­anced corpus requires tackling a kind of do­main (genre) adaptation problem (Chang and Ng, 2006; Agirre and de Lacalle, 2008).

2. In previous WSD tasks, systems have been required to select a sense from a given set of senses in a dictionary for a word in one con­text (an instance). However, the set of senses in the dictionary is not always complete. New word senses sometimes appear after the dic­tionary has been compiled. Therefore, some instances might have a sense that cannot be found in the dictionary's set. The task will take into account not only the instances that have a sense in the given set but also the in­stances that have a sense that cannot be found in the set. In the latter case, systems should output that the instances have a sense that is not in the set.

Training data, a corpus that consists of three genres (books, newspaper articles, and white pa­pers) and is manually annotated with sense IDs, was also distributed to participants. For the evalu­ation, we distributed a corpus that consists of four genres (books, newspaper articles, white papers, and documents from a Q&A site on the WWW) with marked target words as test data. Participants were requested to assign one or more sense IDs to each target word, optionally with associated prob­abilities. The number of target words was 50, with 22 nouns, 23 verbs, and 5 adjectives. Fifty in­stances of each target word were provided, consisting of a total of 2,500 instances for the evalua­tion.

In what follows, section two describes the de­tails of the data used in the Japanese WSD task. Section three describes the process to construct the sense tagged data, including the analysis of an inter-annotator agreement. Section four briefly in­troduces participating systems and section five de­scribes their results. Finally, section six concludes the paper.

In the Japanese WSD task, three types of data were distributed to all participants: a sense inventory, training data, and test data.

Except for the word sense IDs, the data described in section two was developed by the National In­stitute of Japanese Language. However, the word sense IDs were newly annotated on the data. This section presents the process of annotating the word sense IDs, and the analysis of the inter-annotator agreement.

The evaluation was returned in the following two ways:

1. The outputted sense IDs were evaluated, as­suming the 'new sense' as another sense ID. The outputted sense IDs were compared to the given gold standard word senses, and the usual precision measure for supervised word sense disambiguation systems was computed using the scorer. The Iwanami Kokugo Jiten has three levels for sense IDs, and we used the middle-level sense in the task. Therefore, the scoring in the task was 'middle-grained scoring.'

2. The ability of finding the instances of new senses was evaluated, assuming the task as classifying each instance into a 'known sense' or 'new sense' class. The outputted sense IDs (same as in 1.) were compared to the given gold standard word senses, and the usual accuracy for binary classification was computed, assuming all sense IDs in the dic­tionary were in the 'known sense' class.

In the Japanese WSD task, 10 organizations reg­istered for participation. However, only the nine systems from four organizations submitted the re­sults. In what follows, we outline them with the following description:

5. whether and how the difference in the text genre was taken into account, 6. method to detect new senses of words, if any.

Note that most of the systems used supervised learning techniques.

1. Naive Bayes, 2. Word form/POS of the target word, word form/POS before or after the target word, content words in the con­text, classes in a thesaurus for those words in the context, the text genre, 3. 'Bunrui-Goi-Hyou', a Japanese thesaurus (National Insti­tute of Japanese Language, 1964), 4. Mor­phological analysis, 5. A genre is included in the features. 6. Assuming that the posterior probability has a normal distribution, the sys­tem judges those instances deviating from the distribution at the 0.05 significance level as a new word sense 1. Agglomerative clustering, 2. Bag-of-words in context, etc. 3. None, 4. Mor­phological analysis, 5. The system does not merge example sentences in different genre sub-corpus into a cluster. 6. First, the system makes clusters of example sentences, then measures the similarity between a cluster and a sense in the dictionary, finally regarding the cluster as a collection of new senses when the similarity is small. For WSD, the system chooses the most similar sense for each clus­ter, then it considers all the instances in the cluster to have that sense.

1. SVM, 2. Word form/POS before or after the target word, content words in the context, etc. 3. None, 4. Morphological analysis, 5. The system was trained with the feature set where features are distinguished whether or not they are derived from only one genre sub-corpus. 6. 'New sense' is treated as one of the sense classes.

The system is an ensemble of JAIST-1 and JAIST-2. The judgment of a new sense is per­formed by JAIST-1. The output of JAIST-1 is chosen when the similarity between a cluster and a sense in the dictionary is sufficiently high. Otherwise, the output of JAIST-2 is used.

1. Maximum entropy, 2. Three word forms/lemmas/POSs before or after the target word, bigrams, and skip bigrams in the con­text, bag-of-words in the document, a class of the document categorized by a topic clas­sifier, etc. 3. None, 4. None, 5. For each tar­get word, the system selected the genre and dictionary examples combinations for train­ing data, which got the best results in cross-validation. 6. The system calculated the en­tropy for each target word given by the Maxi­mum Entropy Model (MEM). It assumed that high entropy (when probabilities of classes are uniformly dispersed) was indicative of a new sense. The threshold was tuned by using the words with a new sense tag in the training data. Three official submissions correspond to different thresholds.

1. Naive Bayes, 2. Only the 'writing' of the words (inside of <mor> tag), 3. The Mainichi 2005 corpus of NTCIR, parsed with chasen+unidic, 4. None, 5. Not taken into ac­count, 6. 'New sense' is only used when it is evident in the training data

For more details, please refer to their description papers.

The evaluation results of all the systems are shown in tables 1 and 2. "Baseline" for WSD indicates the results of the baseline system that used SVM with the following features:

• Morphological features

Bag-of-words (BOW), Part-of-speech (POS), and detailed POS classification. We extract these features from the target word itself and the two words to the right and left of it.

• Syntactic features

- If the POS of a target word is a noun, extract the verb in a grammatical depen­dency relation with the noun.

This paper reported an overview of the SemEval-2 Japanese WSD task. The data used in this task will be available when you contact the task organizer and sign a copyright agreement form. We hope this valuable data helps many researchers improve their WSD systems.

We would like to thank all the participants and the annotators for constructing this sense tagged cor­pus.