We interpret the task of ranking a set of given paraphrasing verbs as described by Butnariu et al (2010) as a competition between these verbs. Each verb competes with every other verb in the set and receives a positive score if it is more likely to describe the given noun compound (NC) than the other verb and a negative score if it is less likely to describe the NC. In line with this ap­proach we regard the task as a classification prob­lem where for each comparison our classification algorithm picks the paraphrasing verb that is more likely to describe the NC. This brings the clas­sification problem down to three classes: higher, equal or lower. Sometimes the paraphrasing verbs are accompanied by a preposition. In this paper we will simply refer to all verbs and verb-prepositions as verbs.

The distribution of the verbs in the training data provides us already with valuable information. We incorporate basic features describing this distribu­tion to train our classifier. We also need addi­tional semantic features that provide us with in­sight into the relation between the NC and the verb, therefore we use features constructed from WordNet and the Google N-gram Corpus to train our Memory-based paraphrase interpretation scor­ing tool (Mephisto).

The system consists of three components: the fea­ture extraction component, the classification com­ponent and the ranking component. We will de­scribe all three components.

For development the original training set was di­vided in a development training set of 15,966 lines and a development test set of 1,761 lines, which contains 23 NCs. The distribution and ranking fea­tures were calculated using only the development training set. Because we compare for each NC ev­ery verb to every other verb the TiMBL training instance-base contains 1,253,872 lines, and the de­velopment test set 145,620. The results for differ­ent settings are in Table 3. Although the TiMBL F-score (macro-averaged) of using all features is actually lower than using only semantic features at k=3, the final correlations are in favor of using all features. There does not seem to be an improve­ment when extrapolating from 11 neighbouring in­stances in the instance-base over 3. In fact, when using no external features and k=ll, the classifier overgeneralizes and classifies every instance as =eq= and consequently does not provide a ranking at all. Additionally, classifying with k=ll takes considerably longer than with k=3. The settings we use for our final system are k=3 and we use all features. Table 2 displays a confusion matrix of the classification on the development test set. Not surprisingly the classifier is very bad at recogniz­ing the =eq= class. These mistakes are not as bad as miss-classifying a +up+ instance as -dwn- and vice versa, and fortunately these mistakes happen less often.

The official test set contains 32,830 instances, almost twice as many as the training set. This breaks down into 2,837,226 cases to classify. In Table 4 are the final results of the task with all participating systems and their macro-averaged Spearman, Pearson and Cosine correlation. Also shown is the baseline, which involves scoring a given verb paraphrase by its frequency in the train­ing set. The final results are quite a bit lower than the results on the development set. This could be coincidence (the final test set is about twenty times larger than our development test set), but it could also be due to overfitting on the development set. The ten best and worst scoring compounds are shown in Table 5 with their Least Common Sub­sumer as taken from WordNet. The best-scoring NC "jute products" achieves a Spearman p of 0.75 while the worst-scoring compound, "electron mi­croscope" only achieves 0.12.

We have shown that a Memory-based pairwise approach to ranking with features taken from WordNet and the Google N-gram corpus achieves

Table 5: Best and worst scoring noun compounds with their Least Common Subsumer and Spear­man p correlation good results on the task of ranking verbs para­phrasing noun compounds. We outperform the strong baseline and also systems using an unsuper­vised approach. If we analyse our results we see that our system scores particularly well on noun compounds describing materials: in Table 5 we see that all top ten compounds are either "arti­facts", "physical entities" or "entities" according to WordNet and the relation is quite direct: gen­erally a made of relation seems appropriate. If we look at the bottom ten on the other hand, we see re­lations such as "abstraction" and "measure": these are harder to qualify. Also, an "electron micro­scope" will generally not be perceived as a micro­scope made of electrons. We can conclude that for NCs where the relation between the nouns is more obscure the verbs are harder to rank.

If we look at the Information Gain Ratio, of all features the rank difference of the verbs in the training data seems to be the strongest feature, and of the external features the frequency difference of the entire phrase containing the NC and the verb. A lot more investigations could be made into the viability of using large n-gram collections such as the Google N-gram corpus for paraphrase tasks.

It might also be interesting to explore a some­what more challenging variant of this task by not providing the verbs to be ranked a priori. This would probably be more interesting for real world applications because often the task is not only ranking but finding the verbs in the first place. Our system should be able to handle this task with mi­nor modifications: we simply regards all verbs in the training-data candidates to be ranked. Then, a pre-filtering step should take place to weed out irrelevant verbs based on an indicator such as the LCS of the nouns. In addition a threshold could be implemented to only accept a (further) limited set of verbs in the final ranking.