For a word, each of its contexts are represented by the words with which it co-occurs. We approx­imate this high dimensional co-occurrence space with the Random Indexing (RI) word space model (Kanerva et al, 2000). RI represents the occur­rence of a word with an index vector, rather than a set of dimensions. An index vector is a fixed, sparse vector that is orthogonal to all other words' index vectors with a high probability; the total number of dimensions in the model is fixed at a small value, e.g. 5,000. Orthogonality is obtained by setting a small percentage of the vector's values to ±1 and setting the rest to 0.

A context is represented by summing the index vectors corresponding to the n words occurring to the left and right of the polysemous word. Each occurrence of the polysemous word in the entire corpus is treated as a separate context. Contexts are represented by a compact first-order occur­rence vector; using index vectors to represent the occurrences avoids the computational overhead of other dimensional reduction techniques such as the SVD.

Clustering separates similar context vectors into dissimilar clusters that represent the distinct senses of a word. We use an efficient hybrid of online K-Means and Hierarchical Agglomerative

Clustering (HAC) with a threshold. The thresh­old allows for the final number of clusters to be determined by data similarity instead of having to specify the number of clusters.

The set of context vectors for a word are clus­tered using K-Means, which assigns a context to the most similar cluster centroid. If the near­est centroid has a similarity less than the cluster threshold and there are not K clusters, the context forms a new cluster. We define the similarity be­tween contexts vectors as the cosine similarity.

Once the corpus has been processed, clusters are repeatedly merged using HAC with the aver­age link criteria, following (Pedersen and Bruce. 1997). Average link clustering defines cluster sim­ilarity as the mean cosine similarity of the pair-wise similarity of all data points from each clus­ter. Cluster merging stops when the two most sim­ilar clusters have a similarity less than the clus­ter threshold. Reaching a similarity lower than the cluster threshold signifies that each cluster repre­sents a distinct word sense.

Before training and evaluating our model, all occurrences of the 100 polysemous words were stemmed in the corpora. Stemming was required due to a polysemous word being used in multiple lexical forms, e.g. plural, in the corpora. By stem­ming, we avoid the need to combine contexts for each of the distinct word forms during clustering.

After training our WSI model on the training corpus, we process the test corpus and label the context for each polysemous word with an induced sense. Each test context is labeled with the name of the cluster whose centroid has the highest co­sine similarity to the context vector. We represent the test contexts in the same method used for train­ing; index vectors are re-used from training.

The first measure used is the V-Measure (Rosen­berg and Hirschberg, 2007), which compares the clusters of target contexts to word classes. This measure rates the homogeneity and completeness of a clustering solution. Solutions that have word clusters formed from one word class are homoge­neous; completeness measures the degree to which a word class is composed of target contexts allo­cated to a single cluster.

The second measure, the F-Score, is an ex­tension from information retrieval and provides a contrasting evaluation metric by using a different interpretation of homogeneity and completeness. For the F-Score, the precision and recall of all pos­sible context pairs are measured, where a word class has the expected context pairs and a provided solution contains some word pairs that are correct and others that are unexpected. The F-Score tends to discount smaller clusters and clusters that can­not be assigned to a word class (Manandhar et al.. 2010).

Previous WSI evaluations provided a test corpus, a set of golden sense labels, and a scoring mecha­nism, which allowed models to do parameter tun­ing prior to providing a set of sense labels. The SemEval 2010 task provided a trial corpus that contains contexts for four verbs that are not in the evaluation corpus, which can be used for train­ing and testing. The trial corpus also came with a set of golden sense assignments. No golden stan­dard was provided for the training or test corpora, which limited any parameter tuning.

HERMIT exposes three parameters: cluster threshold, the maximum number of clusters and the window size for a context. An initial anal­ysis from the trial data showed that the window size most affected the scores; small window sizes resulted in higher V-Measure scores, while larger window sizes maximized the F-Score. Because contexts are represented using only first-order fea­tures, a smaller window size should have less over­lap, which potentially results in a higher number of clusters. We opted to maximize the V-Measure score by using a window size of ±1.

Due to the limited number of training instances, our precursory analysis with the trial data did not show significant differences for the remaining two parameters; we arbitrarily selected a clustering threshold of .15 and a maximum of 15 clusters per word without any parameter tuning.

After the release of the testing key, we performed a post-hoc analysis to evaluate the effects of parameter tuning on the scores. We include two alternative parameter configurations that were op­timized for the F-Score (HERMIT-F) and the su­pervised evaluations (HERMIT-S). The HERMIT-F variation used a threshold of 0.85 and a win­dow size of ±10 words. The HERMIT-S variation used a threshold of 0.85 and a window size of ±1 words. We did not vary the maximum number of clusters, which was set at 15.

For each evaluation, we provide the scores of seven systems: the three HERMIT configurations, the highest and lowest scoring submitted systems, the Most Frequent Sense (MFS) baseline, and a Random baseline provided by the evaluation team. We provide the scores for each experiment when evaluating all words, nouns, and verbs. We also include the system's rank relative to all submitted systems and the average number of senses gen­erated for each system; our alternative HERMIT configurations are given no rank.

Table 1 : V-Measure for the unsupervised evalua­tion

The unsupervised evaluation considers a golden sense labeling to be word classes and a set of in­duced word senses as clusters of target contexts (Manandhar et al, 2010). Tables 1 and 2 display the results for the unsupervised evaluation when measured according to the V-Measure and the F-Score, respectively. Our system provides the best V-Measure of all submitted systems for this eval­uation. This is in part due to the average number of senses our system generated (10.78), which favors more homogenous clusters. Conversely, this configuration does poorly when measured by F-Score, which tends to favor systems that generate fewer senses per word.

When configured for the F-Score, HERMIT-F performs well; this configuration would have ranked third for the F-Score if it had been submit­ted. However, its performance is also due to the relatively few senses per word it generates, 1.54. The inverse performance of both optimized con­figurations is reflective of the contrasting nature of the two performance measures.

The supervised evaluation simulates a super­vised Word Sense Disambiguation (WSD) task. The induced sense labels for the test corpus are split such that the first set is used for mapping in­duced senses to golden senses and the remaining sense labels are treated as sense labels provided by a WSD system, which allows for evaluation. Five splits are done at random to avoid any biases created due to the separation of the mapping cor­pus and the evaluation corpus; the resulting score for this task is the average recall over the five di­visions. Two sets of splits were used for evalua­tion: one with 80% of the senses as the mapping portion and 20% as the evaluation portion and one with 60% as the mapping portion corpus and 40% for evaluation.

The results for the 80/20 split and 60/40 split are displayed in tables 3 and 4, respectively. In both supervised evaluations, our submitted system

Figure 1: A comparison for F-Score and V-Measure for different window sizes. Scores are an average using thresholds of 0.15, 0.55 and 0.75.

does moderately well. In both cases it outperforms the Random baseline and does almost as well as the MFS baseline. The submitted system outper­forms the Random baseline and approaches the MFS baseline for the 80/20 split. The HERMIT-S version, which is optimized for this task, provides similar results.