The target word dataset consisted of 100 words, i.e. 50 nouns and 50 verbs. The training dataset for each target noun or verb was created by follow­ing a web-based semi-automatic method, similar to the method for the construction of Topic Signa­tures (Agirre et al., 2001). Specifically, for each WordNet (Fellbaum, 1998) sense of a target word, we created a query of the following form:

<Target Word> AND <Relative Set>

The <Target Word> consisted of the target word stem. The <Relative Set> consisted of a disjunctive set of word lemmas that were related to the target word sense for which the query was created. The relations considered were WordNet's hypernyms, hyponyms, synonyms, meronyms and holonyms. Each query was manually checked by one of the organisers to remove ambiguous words. The following example shows the query created for the first and second WordNet sense of the target noun failure.

The created queries were issued to Yahoo! search API and for each query a maximum of 1000 pages were downloaded. For each page we extracted fragments of text that occurred in <p> </p> html tags and contained the target word stem. In the final stage, each extracted fragment of text was POS-tagged using the Genia tagger (Tsu-ruoka and Tsujii, 2005) and was only retained, if the POS of the target word in the extracted text matched the POS of the target word in our dataset.

The testing dataset consisted of instances of the same target words from the training dataset. This dataset is part of OntoNotes (Hovy et al., 2006). We used the sense-tagged dataset in which sen­tences containing target word instances are tagged with OntoNotes (Hovy et al., 2006) senses. The texts come from various news sources including CNN, ABC and others.

This section presents the measures of unsuper­vised evaluation, i.e V-Measure (Rosenberg and Hirschberg, 2007) and (2) paired F-Score (Artiles et al., 2009).

In this evaluation, the testing dataset is split into a mapping and an evaluation corpus. The first one is used to map the automatically induced clusters to GS senses, while the second is used to evaluate methods in a WSD setting. This evaluation fol­lows the supervised evaluation of SemEval-2007 WSI task (Agirre and Soroa, 2007), with the dif­ference that the reported results are an average of 5 random splits. This repeated random sam­pling was performed to avoid the problems of the SemEval-2007 WSI challenge, in which different splits were providing different system rankings.

Let us consider the example in Table 3 and as­sume that this matrix has been created by using the mapping corpus. Table 3 shows that C\ is more likely to be associated with G3, G2 is more likely to be associated with G2, G3 is more likely to be associated with G3 and G4 is more likely to be as­sociated with Gi. This information can be utilised to map the clusters to GS senses.

Particularly, the matrix shown in Table 3 is nor­malised to produce a matrix M, in which each entry depicts the estimated conditional probabil­ity P(Gi\Cj). Given an instance / of tw from the evaluation corpus, a row cluster vector IC is created, in which each entry k corresponds to the score assigned to G& to be the winning cluster for instance I. The product of IC and M provides a row sense vector, IG, in which the highest scor­ing entry a denotes that Ga is the winning sense. For example, if we produce the row cluster vector [Gi = 0.8, G2 = 0.1, G3 = 0.1, G4 = 0.0], and multiply it with the normalised matrix of Table 3, then we would get a row sense vector in which G3 would be the winning sense with a score equal to 0.43.

Table 4 shows the V-Measure (VM) performance of the 26 systems participating in the task. The last column shows the number of induced clusters of each system in the test set.The MFS baseline has a V-Measure equal to 0, since by definition its com­pleteness is 1 and homogeneity is 0. All systems outperform this baseline, apart from one, whose V-Measure is equal to 0. Regarding the Random baseline, we observe that 17 perform better, which indicates that they have learned useful information better than chance.

Table 6 shows the results of this evaluation for a 80-20 test set split, i.e. 80% for mapping and 20% for evaluation. The last columns shows the aver­age number of GS senses identified by each sys­tem in the five splits of the evaluation datasets. Overall, 14 systems outperform the MFS, while 17 of them perform better than Random. The ranking of systems in nouns and verbs is different. For instance, the highest ranked system in nouns is UoY, while in verbs Duluth-Mix-Narrow-Gap. It seems that depending on the part-of-speech of the target word, different algorithms, features and parame­ters' tuning have different impact.

The supervised evaluation changes the distri­bution of clusters by mapping each cluster to a weighted vector of senses. Hence, it can poten­tially favour systems generating a high number of homogeneous clusters. For that reason, we applied a second testing set split, where 60% of the testing corpus was used for mapping and 40% for eval­uation. Reducing the size of the mapping corpus allows us to observe, whether the above statement is correct, since systems with a high number of clusters would suffer from unreliable mapping.

Table 7 shows the results of the second super­vised evaluation. The ranking of participants did not change significantly, i.e. we observe only dif­ferent rankings among systems belonging to the same participant. Despite that, Table 7 also shows that the reduction of the mapping corpus has a dif­ferent impact on systems generating a larger num­ber of clusters than the GS number of senses.

For instance, UoY that generates 11.54 clusters outperformed the MFS by 3.77% in the 80-20 split and by 3.71% in the 60-40 split. The reduction of the mapping corpus had a minimal impact on its performance. In contrast, KSU KDD that gener­ates 17.5 clusters was below the MFS by 6.49% in the 80-20 split and by 7.83% in the 60-40 split. The reduction of the mapping corpus had a larger impact in this case. This result indicates that the performance in this evaluation also depends on the distribution of instances within the clusters. Sys­tems generating a skewed distribution, in which a small number of homogeneous clusters tag the ma­jority of instances and a larger number of clusters tag only a few instances, are likely to have a bet­ter performance than systems that produce a more uniform distribution.