JAIST: Clustering and Classification based Approaches
for Japanese WSD
Kiyoaki Shirai Makoto Nakamura
Japan Advanced Institute of Science and Technology {kshirai,mnakamur}@ jaist.ac.jp
This paper reports about our systems in SemEval2 Japanese Word Sense Disambiguation (WSD) task (Okumura et al., 2010). This task is a lexical sample task for Japanese WSD and has the following two characteristics. First, a balanced word-sense tagged corpus is used for the task. Since it consists of sub-corpora of several domains or genres, domain adaptation might be required. Second, the task takes into account not only the instances having a sense in the given set but also the instances having a sense not found in the set (called 'new sense'). Participants are required to identify new senses of words in this task.
The second characteristics of the task is mainly considered in our system. A clustering based approach is investigated to identify new senses. Our system hrst constructs a set of clusters of given word instances using unsupervised clustering techniques. This is motivated by the fact that the new sense is not defined in the dictionary, and sense induction without referring to the dictionary would be required. Clusters obtained would be sets of instances having the same sense, and some of them would be new sense instances. Then each cluster is judged whether instances in it have a new sense or not. An ordinary classification-based approach is also considered. That is, WSD classifiers are trained by a supervised learning algorithm.
Furthermore, simple techniques considering genres of sub-corpora are incorporated into both our clustering and classification based systems.
The paper continues as follows, Section 2 describes our three participating systems, JAIST-1, JAIST-2 and JAIST-3. The results of these systems are reported and discussed in Section 3. Finally we conclude the paper in Section 4.
JAIST-1 was developed by a clustering based method. The overview of the system is shown in Figure 1. It consists of two procedures: (A) clusters of word instances are constructed so that the instances of the same sense are merged, (B) then similarity between a cluster and a sense in a dictionary is measured in order to determine senses of instances in each cluster.
Dictionary
[f-t;X] (service) S,SMt5kT4L. mW, help that people who work in a shop give you help that is provided by a business to customers S3*ft.
volunteer work
As previous work applying clustering techniques for sense induction (Schütze, 1998; Agirre and Soroa, 2007), each instance is represented by a feature vector. In JAIST-1, the following 4 vectors are used for clustering.
Collocation Vector This vector reflects collocation including the target instance. Words or POS s appearing just before and after the target instance are used as features, i.e. they correspond to one dimension in the vector. The weight of each feature is 1 if the feature exists for the instance, or 0 if not.
Context Vector The vector reflects words in the context of the target instance. All content words appearing in the context are used as features. The window size of the context is set to 50. Furthermore, related words are also used as features to enrich the information in the vector.
Association Vector Similarly to context vector, this reflects words in the context of the target instance, but data sparseness is alleviated in a different manner. In advance, the co-occurrence matrix A is constructed from a corpus. Each row and column in A corresponds to one of the most frequent 10,000 content words. Each element a^j in the matrix is P(wi\wj), conditional probability representing how likely it is that two words Wi and Wj will occur in the same document. Now j-th column in A can be regarded as the co-occurrence vector of Wj, o(wj). Association vector is a normalized vector of sum of o(wj) for all words in the context.
Topic Vector Unlike other vectors, this vector reflects topics of texts. The topics Zj automatically derived by PLSI (Probabilistic Latent Semantic Indexing) are used as features. The weight for Zj in the vector is P(zj\di) estimated by Folding-in algorithm (Hofmann, 1999), where di is the document containing the instance. Topic vector is motivated by the well-known fact that word senses are highly associated with the topics of documents.
Target instances are clustered by the agglomera-tive clustering algorithm. Similarities between instances are calculated by cosine measure of vectors. Furthermore, pairs of instances in different genre sub-corpora are treated as 'cannot-link', so that they will not be merged into the same cluster. Clustering procedure is stopped when the number of instances in a cluster become more than a threshold Nc. Nc is set to 5 in the participating system.
The clustering is performed 4 times using 4 different feature vectors. Then the best one is chosen from the 4 sets of clusters obtained. A set of cluster C (={d}) is evaluated by E(C) where 'cohesiveness' coh(Ci) for each cluster d is defined by (2).
\Ci\ maxjsimivij,^) Vij is an instance vector in the cluster d, while cji is an average vector of d. rel-sim(vij, cji) means the relative similarity between the instance vector and average vector. Intuitively, coh(Ci) evaluates how likely instances in the cluster are similar each other. C such that E(C) is maximum is chosen as the final set of clusters.
After clustering, similarity between a cluster d and a sense Sj in the dictionary, sim(d, Sj), is calculated for WSD. d and Sj are represented by cluster vector c% and sense vector Sj, respectively. Then cosine measure between these two vectors is calculated as sim(d, Sj).
The cluster vector c% is defined as (3):
In (3), eik stands for an instance in the cluster d, ti words appearing in the context of e^, o(ti) cooccurrence vector of ti (similar one used in association vector), and N the constant for normalization. So Ci is similar to association vector, but the co-occurrence vectors of words in the contexts of all instances in the cluster are summed. The sense vector Sj is defined as in (4).
Dj stands for définition sentences of the sense Sj in the Japanese dictionary Iwanami Kokugo Jiten (the sense inventory in this task), while Ej a set of example sentences of Sj. Here Ej includes both example sentences from the dictionary and ones excerpted from a sense-tagged corpus, the training data of this task. we is the parameter putting more weight on words in example sentences than in définition sentences. We set we = 2.0 through the preliminary investigation.
Based on sim(Ci, Sj), the system judges whether the cluster is a collection of new sense instances.
The threshold Tns for each target word is determined as follows. First the training data is equally subdivided into two halves, the development data Ddev and the training data Dtr • Next, JAIST-1 is run for instances in Ddev, while example sentences in Dtr are used as Ej in (4) when sense vectors are constructed. For words where new sense instances exist in D^v, Tns is optimized for the accuracy of new sense detection. For words where no new sense instances are found in Ddev, Tns is determined by the minimum of MaxSirrii as follows:
Since even the cluster of which MaxSirrii is minimum represents not a new but a defined sense, the minimum of MaxSirrii is decreased by 7. To determine 7, the ratios MaxSirrii of clusters of new senses MaxSirrii of clusters of defined senses are investigated for 5 words
Our second system JAIST-2 is the classification based method. It is a WSD classifier trained by Support Vector Machine (SVM). SVM is widely used for various NLP tasks including Japanese WSD (Shirai and Tamagaki, 2004). In this system, new sense is treated as one of the sense classes. Thus it would never choose "new sense" for any instances when no new sense instance is found in the training data. We used the LIB SVM package
The following conventional features of WSD are used for training the SVM classifiers.
'Among 50 target words in this task, there exist new sense instances of only 'kanou' (possibility) in Ddev So we checked 4 more words, other than target words.
Words and their POSs appearing before or after a target instance. A number in parentheses indicates the position of a word from a target instance. W(0) means a target instance itself.
Pairs of words (or their POSs) near a target instance.
• Base form of content words appearing in the context (bag-of-words).
The data used in this task is a set of documents with 4 different genre codes: OC (Web page), OW (white paper), PB (book) and PN (newspaper). The training data consists of documents of 3 genres OW, PB and PN, while the test data contains all 4 genres. Considering domain adaptation, each feature fi is represented as fi + g when SVM classifiers are trained, g is one of the genre codes {OW, PB, PN} if fi is derived from the documents of only one genre g in the training data, otherwise g is 'multi'. For instances in the test data, only features fi+gt and fi+multi are used, where gt is the genre code of the document of the target instance. If gt is OC (which is not included in the training data), however, all features are used. The above method aims at distinguishing genre intrinsic features and improving the WSD performance by excluding features which might be associated with different genres.
The third system combines clustering based method (JAIST-1) and classification based method (JAIST-2). The basic idea is that JAIST-1 be used only for reliable clusters, otherwise JAIST-2 is used. Here 'reliable cluster' means a cluster such that MaxSirrii is high. The greater the similarity between the cluster and the sense is, the more likely the chosen sense is correct. Furthermore, JAIST-1 is used for new sense detection. The detailed procedure in JAIST-3 is:
1. If JAIST-1 judges a cluster to be a collection of new sense instances, output 'new sense' for instances in that cluster.
2. For instances in the top Ncj clusters of MaxSirrii,output senses chosen by JAIST-1.
3. Otherwise output senses chosen by JAIST-2.
For the optimization of Nrj, D^ev and Dtr, each is a half of the training data described in Subsection 2.1, are used. Dtr is used for training SVM classifiers (JAIST-2). Then Nci is determined so that the precision of WSD on D^v is optimized. In the participating system, Nci is set to 1.
Table 1 shows the results of our participating systems and the baseline system MFS, which always selects the most frequent sense in the training data. The column WSD reveals the precision (P) of word sense disambiguation, while the column NSD shows accuracy (A), precision (P) and recall (R) of new sense detection.
JAIST-1 is the clustering based method. Performance of the clustering is also evaluated: Purity was 0.9636, Inverse-Purity 0.1336 and F-measure 0.2333. Although this system was designed for new sense detection, it seems not to work well. It could correctly find only three new sense instances. The main reason is that there were few instances of the new sense in the test data. Among 2,500 instances (50 instances of each word, for 50 target word), only 39 instances had the new sense. Our system supposes that considerable number of new sense instances exist in the corpus, and tries to gather them into clusters. However, JAIST-1 was able to construct only one cluster containing multiple new sense instances. The proposed method is inadequate for new sense detection when the number of new sense instances is quite small.
For domain adaptation, features which are intrinsic to different genres were excluded for test instances in JAIST-2. When we trained the system using all features, its precision was 0.7516, which is higher than that of JAIST-2. Thus our method does not work at all. This might be caused by removing features that were derived from different genre sub-corpora, but effective for WSD. More sophisticated ways to remove ineffective features would be required.
JAIST-3 is the ensemble of JAIST-1 and JAIST-2. Although a little improvement is found by combining two different systems in our preliminary experiments, however, the performance of JAIST-3 was worse than JAIST-2 because of the low performance of JAIST-1. We compared WSD precision of three systems for 50 individual target words, and found that JAIST-2 is almost always the best. The only exceptional case was the target word 'ookii'(big). For this adjective, the precision of JAIST-1, JAIST-2 and JAIST-3 were 0.74, 0.16 and 0.18, respectively. The precision of SVM classifiers (JAIST-2) is quite bad because of the difference of text genres. All 50 test instances of this word were excerpted from Web sub-corpus, which was not included in the training data. Furthermore, word sense distributions of test and training data were totally different. JAIST-1 works better in such a case. Thus clustering based method might be an alternative method for WSD when sense distribution in the test data is far from the training data.
The paper reports the participating systems in SemEval-2 Japanese WSD task. Clustering based method was designed for new sense detection, however, it was ineffective when there were few new sense instances. In future, we would like to examine the performance of our method when it is applied to a corpus including more new senses.
WSD | NSD | |||||
P | A | P | R | |||
MFS | 0.6896 | 0.9844 | 0 | 0 | ||
JAIST-1 | 0.6864 | 0.9512 | 0.0337 | 0.0769 | ||
JAIST-2 | 0.7476 | 0.9872 | 1 | 0.1795 | ||
JAIST-3 | 0.7208 | 0.9532 | 0.0851 | 0.2051 | ||
Eneko Agirre and Aitor Soroa. 2007. Semeval-2007 task 02: Evaluating word sense induction and discrimination systems. In Proceedings of the 4th International Workshop on Semantic Evaluations, pages 7-12.
David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research, 3:993-1022.
Thomas Hofmann. 1999. Probabilistic latent semantic indexing. In Proceedings of the SIGIR, pages 50-57.
Manabu Okumura, Kiyoaki Shirai, Kanako Komiya, and Hikara Yokono. 2010. Semeval-2010 task: Japanese WSD. In Proceedings of the SemEval-2010: 5th International Workshop on Semantic Evaluations.
Hinrich Schütze. 1998. Automatic word sense discrimination. Computational Linguistics, 24(1):97-123.
Kiyoaki Shirai and Takayuki Tamagaki. 2004. Word sense disambiguation using heterogeneous language resources. In Proceedings of the First 1JCNLP, pages 614—619.