This paper introduces a knowledge-based approach to the task of semantic relation classification, as evaluated in SemEval 2007, Task 4: "Classifying Relations Between Nominals". In Task 4, a full sentence is presented to the system, along with the WordNet sense keys for two noun phrases which appear there and the name of a semantic relation (e.g. "cause-effect"). The system should return "true" if a person reading the sentence would con­clude that the relation holds between the two la­beled noun phrases.

Our system represents a test sentence with a se­mantic graph, including the relation being tested and both of its proposed arguments. Semantic dis­tance is calculated between this graph and a set of graphs representing the training examples relevant to the test sentence. A near-match between a test sentence and a positive training example is evi­dence that the same relation which holds in the example also holds in the test. We compute se­mantic distances to negative training examples as well, comparing the total positive and negative scores in order to decide whether a relation is true or false in the test sentence.

Many systems which perform well on related tasks use syntactic features of the input sentence, coupled with classification by machine learning. This approach has been applied to problems like compound noun interpretation (Rosario and Hearst 2001) and semantic role labeling (Gildea and Jurafsky 2002).

In preparing our system for Task 4, we started by applying a similar syntax-based feature analysis to the trial data: 140 labeled examples of the rela­tion "content-container". In 10-fold cross-validation with this data we achieved an average f-score of 70.6, based on features similar to the sub­set trees used for semantic role labeling in (Mo-schitti 2004). For classification we applied the up­dated tree-kernel package (Moschitti 2006), distri­buted with the svm-light tool (Joachims 1999) for learning Support Vector Machines (SVMs).

Training data for Task 4 is small, compared to other tasks where machine learning is commonly applied. We had difficulty finding a combination of features which gave good performance in cross-validation, but which did not result in a separate support vector being stored for every training sen­tence - a possible indicator of overfitting. As an example, the ratio of support vectors to training examples for the experiment described above was .97, nearly 1-to-1.

As a result of this analysis we started work on our knowledge-based system, with the goal of us­ing the two approaches together. We were also motivated by an interest in using relation defini­tions and background knowledge from WordNet to greater advantage. The algorithm we used in our final submission is similar to recent systems which discover textual entailment relationships (Haghig-hi, Ng et al. 2005; Zanzotto and Moschitti 2006). It gives us a way to encode information from the relation definitions directly, in the form of state­ments in a knowledge representation language. The inference rules that are learned by this system from training examples are also easier to interpret than the models generated by an SVM. In small-data applications this can be an advantage.

The example sentence below is taken (in abbre­viated form) from the training data for Task 4, Re­lation 7 "Content-Container" (Girju, Hearst et al.

The kitchen holds a cooker.

We convert this positive example into a semantic graph by creating a new instance of the relation Contains and linking that instance to the WordNet term for each labeled argument ("kitch-en%1:06:00::", "cooker%1:06:00::"). The result is shown in Figure 1. WordNet sense keys (Fellbaum 1998) have been mapped to a term, a part of speech (pos), and a sense number.

Figure 1. Semantic graph for the training example "The kitchen holds a cooker". Arguments are represented by a WordNet term, part of speech, and sense number.

This graph is instantiated as a statement using the Scone Knowledge Representation System, or

Figure 2. Statements in Scone KR syntax, based on generalizing the training example "The kitchen holds a cooker".

"Scone" (Fahlman 2005). Scone gives us a way to store, search, and perform inference on graphs like the one shown above. After instantiating the graph we generalize it using hypernym information from WordNet. This generates additional Scone state­ments which are stored in a knowledge base (KB), shown in Figure 2. The first statement in the fig­ure was generated verbatim from our training sen­tence. The remaining statements contain hyper-nyms of the original arguments.

For each argument seen in training, we also ex­tract hypernyms and siblings from WordNet. For the argument kitchen, we extract 101 ancestors (artifact, whole, object, etc.) and siblings (struc­ture, excavation, facility, etc.). A similar set of WordNet entities is extracted for the argument cooker. These entities, with repetitions removed, are encoded in a second Scone knowledge base, preserving the hierarchical (IS-A) links that come from WordNet. The hierarchy is manually linked at the top level into an existing background Scone KB where entities like animate, inanimate, person, location, and quantity are already defined.

After using the training data to create these two KBs, the system is ready for a test sentence. The following example is also adapted from SemEval Task 4 training data:

Equipment was carried in a box.

First we convert the sentence to a semantic graph, using the same technique as the one de­scribed above. The graph is implemented as a new Scone statement which includes the WordNet pos and sense number for each of the arguments: "box_n_1 contains equipment_n_1".

Next, using inference operations in Scone, the system verifies that the statement conforms to high-level constraints imposed by the relation defi­nition. If it does, we calculate semantic distances between the argument nodes of our test statement and the analogous nodes in relevant training state­ments. A training statement is relevant if both of its arguments are ancestors of the appropriate arguments of the test sentence. In our example, only two of the three KB statements from Figure 2 are relevant to the test statement "box contains equip­ment": "whole contains whole" and "artifact con­tains artifact". The first statement, "kitchen con­tains cooker" fails to apply because kitchen is not an ancestor of box, and also because cooker is not an ancestor of equipment.

Figure 3 illustrates the distance from "box con­tains equipment" to "whole contains whole", calcu­lated as the sum of the distances between box-whole and equipment-whole.

Figure 3. Calculating the distance through the knowledge base between "equipment contains box" and "whole contains whole". Dashed lines indicate IS-A links in the knowledge base.

The total number of these relevant, positive training statements is an indicator of "support" for the test sentence throughout the training data. The distance between one such statement and the test sentence is a measure of the strength of support. To reach a verdict, we sum over the inverse dis­tances to all arguments from positive relevant ex­amples: in Figure 3, the test statement "box con­tains equipment" receives a support score of (V + Vi + 1 + 1), or 3.

Counter-evidence for a test sentence can be cal­culated in the same way, using relevant negative statements. In our example there are no negative training statements, so the total positive support score (3) is greater than the counter-evidence score (0), and the system verdict is "true".

As the detailed example above shows, this system is designed around its knowledge bases. The KBs provide a consistent framework for representing knowledge from a variety of sources as well as for calculating semantic distance.

This system performed reasonably well for relation 3, Product-Producer, outperforming the baseline (baseline guesses "true" for every test sentence). Performance for this relation was also higher than the average F-score for all comparable groups in Task 4 (all groups in class "B4"). Average recall for this system over all relations was mid-range, compared to other participating groups. Average precision and average f-score fell below the base­line and below the average for all comparable groups. These scores are given in Table 1.

Table 1. Recall, Precision, and F-scores, separated by relation type. Baseline score is calculated by guessing "true" for all test setences.

Analysis of the training data reveals that relation 3 is the class where target nouns occur most often together in nominal compounds and base NPs, with little additional syntax to connect them. While other relations included sentences where the targets were covered by a single VP, Product-Producer did not. It seems that background knowledge plays a larger role in identifying the Producer-Produces relationship than it does for other relations. How­ever this conclusion is softened by the fact that we also spent more time in development and cross-evaluation for relations 3 and 7, our two best per­forming relations.

This system demonstrates a knowledge-based framework that performs very well for certain re­lations. Importantly, the system we submitted for evaluation did not make use of syntactic features, which are almost certainly relevant to this task. We are already exploring methods for combining the knowledge-based decision process with one that uses syntactic evidence as well as corpus sta­tistics, described in Section 2.

Acknowledgement

This work was supported by a generous research grant from Cisco Systems, and by the Defense Ad­vanced Research Projects Agency (DARPA) under contract number NBCHD030010.