SemEval-2007 launches totally 18 tasks for evalua­tion exercise, covering word sense disambiguation, word sense discrimination, semantic role labeling, and sense disambiguation for information retrieval, and other topics in NLP. We participated three tasks in SemEval-2007, which are task 2 (Evaluating Word Sense Induction and Discrimination Systems), task 5 (Multilingual Chinese-English Lexical Sam­ple Task) and the first subtask at task 17 (English Lexical Sample, English Semantic Role Labeling and English All-Words Tasks).

The goal for SemEval-2007 task 2 (Evaluat­ing Word Sense Induction and Discrimination Sys-tems)(Agirre and Soroa, 2007) is to automatically discriminate the senses of English target words by the use of only untagged data. Here we address this word sense discrimination problem by (1) estimat­ing the number of word senses of a target word in untagged data using a stability criterion, and then (2) grouping the instances of this target word into the estimated number of clusters according to the simi­larity of contexts of the instances. No sense-tagged data is used to help the clustering process.

The goal of task 5 (Chinese Word Sense Disam­biguation) is to create a framework for the evaluation of word sense disambiguation in Chinese-English machine translation systems. Each participates of this task will be provided with sense tagged train­ing data and untagged test data for 40 Chinese pol-ysemous words. The "sense tags" for the ambigu­ous Chinese target words are given in the form of their English translations. Here we used a semi-supervised classification algorithm (label propaga­tion algorithm) (Niu, et al., 2005) to address this Chinese word sense disambiguation problem.

The lexical sample subtask of task 17 (English Word Sense Disambiguation) provides sense-tagged training data and untagged test data for 35 nouns and 65 verbs. This data includes, for each target word: OntoNotes sense tags (these are groupings of Word­Net senses that are more coarse-grained than traditional WN entries), as well as the sense inventory for these lemmas. Here we used only the training data supplied in this subtask for sense disambiguation in test set. The label propagation algorithm (Niu, et al., 2005) was used to perform sense disambiguation by the use of both training data and test data.

This paper will be organized as follows. First, we will provide the feature set used for task 2, task 5 and task 17 in section 2. Secondly, we will present the word sense discrimination method used for task 2 in section 3. Then, we will give the label propa­gation algorithm for task 5 and task 17 in section 4. Section 5 will provide the description of data sets at task 2, task 5 and task 17. Then, we will present the experimental results of our systems at the three tasks in section 6. Finally we will give a conclusion of our work in section 7.

In task 2, task 5 and task 17, we used three types of features to capture contextual information: part-of-speech of neighboring words (no more than three-word distance) with position information, unordered single words in topical context (all the contextual sentences), and local collocations (including 11 col­locations). The feature set used here is as same as the feature set used in (Lee and Ng, 2002) except that we did not use syntactic relations.

Word sense discrimination is to automatically dis­criminate the senses of target words by the use of only untagged data. So we can employ clustering algorithms to address this problem. Another prob­lem is that there is no sense inventories for target words. So the clustering algorithms should have the ability to automatically estimate the sense number of a target word.

Here we used the sequential Information Bottle­neck algorithm (sIB) (Slonim, et al., 2002) to esti­mate cluster structure, which measures the similarity of contexts of instances of target words according to the similarity of their contextual feature conditional distribution. But sIB requires the number of clus­ters as input. So we used a cluster validation method to automatically estimate the sense number of a target word before clustering analysis. Cluster valida­tion (or stability based approach)is a commonly used method to the problem of model order identification (or cluster number estimation) (Lange, et al., 2002; Levine and Domany, 2001). The assumption of this method is that if the model order is identical with the true value, then the cluster structure estimated from the data is stable against resampling, otherwise, it is more likely to be the artifact of sampled data.

crease when increasing the value of k. Therefore for avoiding the bias that smaller value of k is to be se­lected as cluster number, we use the cluster validity of a random predictor to normalize M (C^.k, CF k ).

In the label propagation algorithm (LP) (Zhu and Ghahramani, 2002), label information of any ver­tex in a graph is propagated to nearby vertices through weighted edges until a global stable stage is achieved. Larger edge weights allow labels to travel through easier. Thus the closer the examples, more likely they have similar labels (the global con­sistency assumption).

In label propagation process, the soft label of each initial labeled example is clamped in each iteration to replenish label sources from these labeled data. Thus the labeled data act like sources to push out la­bels through unlabeled data. With this push from la­beled examples, the class boundaries will be pushed through edges with large weights and settle in gaps along edges with small weights. If the data structure fits the classification goal, then LP algorithm can use these unlabeled data to help learning classification plane.

Let Y G Nnxc represent initial soft labels at­tached to vertices, where Yj = 1 if y is Sj and 0 otherwise. Let Y° be the top l rows of Y and Yj be the remaining u rows. Yj° is consistent with the labeling in labeled data, and the initialization of Yj can be arbitrary.

Optimally we expect that the value of Wj across different classes is as small as possible and the value of Wj within same class is as large as possible. This will make label propagation to stay within same class. In later experiments, we set a as the aver­age distance between labeled examples from differ­ent classes.

Define n x n probability transition matrix Tj = P(j ~^ = E"**Wfc ' wnere Tij is the probability to jump from example Xj to example Xj.

Compute the row-normalized matrix T by Ty = Tjj /J2n=i Tik. This normalization is to maintain the class probability interpretation ofY.

Then LP algorithm is defined as follows:

3. Clamp labeled data by replacing the top l row of Y *+1 with Y°. Repeat from step 2 until Y * con­verges; where j = argmaxj Yhj.

This algorithm has been shown to converge to a unique solution, which is Yu = lim*-^ Y^j = {I ~ Tuu)-lTulYl (Zhu and Ghahramani, 2002). We can see that this solution can be obtained with­out iteration and the initialization of Yj is not im­portant, since Yj° does not affect the estimation of Yu. I is u x u identity matrix. Tuu and Tu\ are acquired by splitting matrix T after the l-th row and the l-th column into 4 sub-matrices.

For task 5 and 17, we constructed connected graphs as follows: two instances u,v will be con­nected by an edge if u is among v'sk nearest neigh­bors, or if v is among u's k nearest neighbors as mea­sured by cosine or JS distance measure. k is set 10 in our system implementation.

The test data for task 2 includes totally 27132 un­tagged instances for 100 ambiguous English words. There is no training data for task 2.

There are 40 ambiguous Chinese words in task 5. The training data for this task consists of 2686 instances, while the test data includes 935 instances.

There are 100 ambiguous English words in the first subtask of task 17. The training data for this task consists of 22281 instances, while the test data includes 4851 instances.

Table 3: The best/worst/average F-score of all the systems at task 2 and the F-score of our system at task 2 for all target words, nouns and verbs with un-supervised evaluation._

Table 3 lists the best/worst/average F-score of all the systems at task 2 and the F-score of our system at task 2 for all target words, nouns and verbs with

Table 4: The best/worst/average supervised recall of all the systems at task 2 and the supervised recall of our system at task 2 for all target words, nouns and verbs with supervised evaluation.

Table 5: The best/worst/average micro-average pre­cision and macro-average precision of all the sys­tems at task 5 and the micro-average precision and macro-average precision of our system at task 5.

unsupervised evaluation. Our system obtained the fourth place among six systems with unsupervised evaluation. Table 4 shows the best/worst/average supervised recall of all the systems at task 2 and the supervised recall of our system at task 2 for all tar­get words, nouns and verbs with supervised evalu­ation. Our system is ranked as the first among six systems with supervised evaluation. Table 7 lists the estimated sense numbers by our system for all the words at task 2. The average of all the estimated sense numbers is 3.1, while the average of all the ground-truth sense numbers is 3.6 if we consider the sense inventories provided in task 17 as the answer. It seems that our estimated sense numbers are close to the ground-truth ones.

Table 5 provides the best/worst/average micro­average precision and macro-average precision ofall the systems at task 5 and the micro-average preci­sion and macro-average precision of our system at task 5. Our system obtained the second place among six systems for task 5.

Table 6 shows the best/worst/average coarse­grained score (precision) of all the systems the lexi­cal sample subtask of task 17 and the coarse-grained score (precision) of our system at the lexical sample

Table 6: The best/worst/average coarse-grained score (precision) of all the systems at the lexical sample subtask of task 17 and the coarse-grained score (precision) of our system at the lexical sam-pie subtask of task 17._ subtask of task 17. The attempted rate of all the sys­tems is 100%. So the precision value is equal to the recall value for all the systems. Here we listed only the precision for the 13 systems at this subtask. Our system is ranked as the third one among 13 systems.

In this paper, we described the implementation of our 12R systems that participated in task 2, task 5, and task 17 at SemEval-2007. Our systems achieved 63.9% F-score and 81.6% supervised recall for task 2, 71.2% micro-average precision and 74.7% macro-average precision for task 5, and 86.4% coarse­grained precision and recall for the lexical sample subtask of task 17. The performance of our system is very good under supervised evaluation. It may be explained by that our system has the ability to find some minor senses so that it can outperforms the baseline system that always uses the most fre­quent sense as the answer.