This paper describes a system for participating in SemEval 2007 task 4; "Classification of Seman­tic Relations Between Nominals". This SemEval task required systems to establish whether or not a particular semantic relation held between two nominals in a sentence. There were 7 semantic re­lations, with approximately 70 positive and 70 negative example sentences for each relation. There were approximately 70 examples in the test sets for each relation.

This task is similar to the problem of determin­ing what semantic relation holds between the constituents of a noun-noun compound. Work in this area has used both statistical information about the frequencies of lexical patterns and hand-built knowledge databases such as WordNet and the­saura. In our system we combine these two knowl­edge sources and build a set of features to use as input to a Support Vector Machine learning algo­rithm.

The use of hit counts from web search engines to obtain lexical information was introduced by Turney (2001). The idea of searching a large cor­pus for specific lexico-syntactic phrases to indicate a semantic relation of interest was first described by Hearst (1992). A lexical pattern specific enough to indicate a particular semantic relation is usually not very frequent, and using the web as a corpus alleviates the data sparseness problem. However, it also introduces some problems. The number of results returned is unstable as pages are created and deleted all the time, and the major search engines return only rounded frequency estimates and do not allow a very sophisticated query interface. Na-kov and Hearst (2005) examined the use of web-based n-gram frequencies for an NLP task and concluded that these issues do not greatly impact the interpretation of the results.

Turney and Littman (2005) use web queries to the AltaVista search engine as the basis for their system to assign semantic relations to modifier-noun phrases. They use a set of 64 short preposi­tional and conjunctive phrases (joining terms) to generate exact queries of the form "noun joining term modifier", and "modifier joining term nouri\ Using 64 joining terms and trying the noun and modifier in either order resulted in a vector of 128 hit counts for each noun-modifier pair. These hit counts were used with a supervised (nearest neighbor) algorithm to label the modifier-noun phrases.

Nakov and Hearst (2006) use queries of the form "noun that * modifier" where '*' is a wildcard operator. By retrieving the words that most commonly occurred in the place of the wildcard they were able to identify very specific predicates that are likely to represent the relation between noun and modifier.

There have also been several approaches which used hand built knowledge sources. Rosario and Hearst (2001) used MeSH, a lexical hierarchy of medical terms. They use this hierarchy to assign semantic properties to head and modifier words in the medical domain. They use a neural network trained on these attributes to assign the noun phrases a semantic relation.

Nastase and Szpakowicz (2003) use the position of the noun and modifier words within general se­mantic hierarchies (Roget's Thesaurus and Word­Net) as attributes for their learning algorithms. They experiment with decision trees, a rule induc­tion system, a relational learner and memory based learning. They conclude that the rule induction system is capable of generalizing to characterize the noun phrases.

Moldovan et al (2004) also use WordNet. They experiment with a Bayesian algorithm, decision trees, and their own algorithm; semantic scattering. As far as we are aware ours is the first system to combine features derived from a hand-built lexical database with corpus frequencies of lexical patterns.

Our system uses both features derived from WordNet and features obtained by collecting web frequencies for lexical patterns. We did not use any information from the sentence in which the two nominals appeared, nor did we use the query used to retrieve the examples. We did make use of the WordNet sense for the features we obtained from

WordNet.

There are 48 features derived from WordNet. Most of these are boolean values indicating whether or not each of the nominals in the sentence appear below certain other high-level concepts in the hypernym hierarchy. We chose 22 high level concepts we believed may be good predictors of whether or not a nominal could be an argument of the semantic relations used in this task. These concepts are listed below in table 1.

For each of these WordNet entries we checked whether or not each of the nominals in the example sentence appeared below the entry in the WordNet hypernym tree. This gave us 44 features. We also checked whether the first nominal was a hypernym of the second; and vice-versa; and whether the first nominal was a meronym of the second; and vice versa. This gives us in total 48 boolean features derived from WordNet.

The average f-value obtained by our system using all of the training data was 65.4. There was a sig­nificant difference in performance across different relations. The results for each relation are below.

Relation

The standard deviation of the f-values is 13.9. The average of the f-values is brought down by two of the relations; origin-entity and theme-tool. The poor performance of these relations was noted during early experimentation with the training data; and the list of WordNet concepts and joining terms was amended to try to improve classifica­tion, but no improvement was achieved. If the re­sults for these relations are omitted the average f-score rises to 73.6

This paper describes a system for participating in SemEval 2007 task 4; "Classification of Semantic Relations Between Nominals". Our system com­bines features generated by analyzing the WordNet hypernym tree with features which indicate the frequencies of certain lexical patterns involving the nominals and common prepositions, using the web as a corpus.

The performance of the system was above the average score of other systems which used the WordNet sense of the training examples but not the query used to obtain them. The system was held back particularly by two relations, theme-tool and origin-entity.

There are many potential avenues for future work in this area. We chose 48 features based on Word-Net and 28 lexical patterns to search the web for. These were chosen arbitrarily on the basis that they looked like they would be informative in general, over all seven relations. A more principled ap­proach would be to begin with a much larger num­ber of features and use information gain to select the most informative features for each relation in­dividually. This should improve performance by ensuring that only the most relevant features for a specific relation are used to train the classifier for that relation.

Also, there is room for more investigation into how short prepositional joining phrases map onto un­derlying semantic relations (Girjiu 2006).