SemEval-2010 Task 8 evaluated the multi-way classification of semantic relations between nom­inals in a sentence (Hendrickx et al., 2010). Given two nominals embedded in a sentence, the task requires identifying which of the fol­lowing nine semantic relations holds between the nominals: Cause-Effect, Instrument-Agency, Product-Producer, Content-Container, Entity-Origin, Entity-Destination, Component-Whole, Member-Collection, Message-Topic, or Other if no other relation is appropriate. For instance, the following sentence provides an example of the Entity-Destination relation: "A small [piece]ei of rock landed into the [trunk] e2-"

The two nominals given for this sentence are Ei (piece) and e2 (trunk). This is an Entity-Destination relation because the piece of rock originated from outside of the trunk, but ended up there. Finally, the direction of the relation is (Ei,e2) because Ei, the piece, is the Entity and e2, the trunk, is the Destination.

Analysis of the training data revealed three ma­jor classes of knowledge required for recognizing semantic relations: (i) examples that require back­ground knowledge of an existing relation between the nominals (e.g., example 5884 below), (ii) ex­amples using background knowledge regarding the typical role of one of the nominals (e.g., ex­ample 3402), and (iii) examples that require con­textual cues to disambiguate the role between the nominals (e.g., example 5710).

Example 5884 "The Ca content in the [corn]e\ [flour]e2 has also a strong dependence on the pericarp thickness."

Example 3402 "The [rootball]e\ was in a [crate]e2 the size of a refrigerator, and some of the arms were over 12 feet tall."

Example 5710 "The seniors poured [flour]e\ into wax [paper] e2 and threw the items as projectiles on freshmen during a morning pep rally."

In example 5884, the background knowledge that flour is often made or derived from corn can directly lead to the classification of the example as containing an Entity-Origin relation. Likewise, knowing that crates often act as containers is a strong reason for believing that example 3402 is a Content-Container relation. However, in exam­ple 5710, neither the combination of the nominals nor their individual affiliations lead to an obvious semantic relation. After taking the context into account, it becomes clear that this is an Entity-Destination relation because Ei is going into e2.

We cast the task of determining a semantic re­lation and its direction as a classification task. Rather than classifying both pieces of informa­tion (relation and direction) simultaneously, one classifier is used to determine the relation type, and then, for each relation type, a separate clas­sifier determines the direction. We used a total of 45 feature types (henceforth: features), which were shared among all of the direction classi­fiers and the one relation classifier. These fea­ture types can be partitioned into 8 groups: lexical features, hypernyms from WordNet, dependency parse, PropBank parse, FrameNet parse, nominal-ization, predicates from TextRunner, and nomi­nal similarity derived from the Google N-Gram data set. All features were treated as feature-type:VALUE pairs which were then presented to the SVM classifier as a boolean feature (0 or 1).

We further group our features into the three classes described above: Contextual, Nominal af­filiation, and Pre-existing relations. Table 1 illus­trates sample feature values from example 117 of the training set.

The contextual features consist of lexical features and features based on dependency, PropBank, and FrameNet parses. For lexical features, we extract the words and parts of speech for Ei and e2, the words, parts of speech, and prefixes of length 5 for tokens between the nominals, and the words be­fore and single word after Ei and e2 respectively. The words between the nominals can be strong indicators for the type of relation. For example the words into, produced, and caused are likely to occur in Entity-Destination, Product-Producer, and Cause-Effect relations, respectively. Using the prefixes of length 5 for the words between the nominals provides a kind of stemming (produced —> produ, caused —> cause).

Inspired by a feature from (Beamer et al., 2007), we extract a coarse-grained part of speech se­quence for the words between the nominals. This is accomplished by building a string using the first letter of each token's Treebank POS tag. This fea­ture is motivated by the fact that relations such as Member-Collection usually invoke prepositional phrases such as: of, in the, and of various. The corresponding POS sequences we extract are: "I", "LD", and "U". Finally, we also use the num­ber of words between the nominals as a feature because relations such as Product-Producer and Entity-Origin often have no intervening tokens (e.g., organ builder or Coconut oil).

Syntactic and semantic parses capture long dis­tance relationships between phrases in a sentence. Instead of a traditional syntactic parser, we chose the Stanford dependency parser for the simpler syntactic structure it produces. Our dependency features are based on paths in the dependency tree of length 1 and length 2. The paths encode the de­pendencies and words those dependencies attach to. To generalize the paths, some of the features replace verbs in the path with their top-level Levin class, as determined by running a word sense dis­ambiguation system (Mihalcea and Csomai, 2005) followed by a lookup in VerbNet. One of the fea­tures for length 2 paths generalizes further by re­placing all words with their location relative to the nominals, either BEFORE, BETWEEN, or AFTER.

Consider example 117 from Table 1. The length 2 dependency path (feature depPathLen2VerbNet) neatly captures the fact that Ei is the subject of a verb falling into Levin class 27, and e2 is the di­rect object. Levin class 27 is the class of engender verbs, such as cause, spawn, and generate. This path is indicative of a Cause-Effect relation.

Semantic parses such as ASSERT's PropBank parse and LTH's FrameNet parse identify predi­cates in text and their semantic roles. These parses go beyond the dependency parse and identify the specific role each nominal assumes for the pred­icates in the sentence, so the parses should be a more reliable indicator for the relation type be­tween nominals. We have features for the iden­tified predicates and for the roles assigned to each nominal. Several of the features are only triggered if both nominals are arguments for the same pred­icate. The values from Table 1 show that the fea­tures correctly determined that Ei and e2 are gov­erned by a verb of Levin class 27, and that the lex­ical unit is cause, v.

Although context can be critical to identifying the semantic relation present in some examples, in others we must bring some background knowledge to bear regarding the types of nominals involved. Knowing that a writer is a person provides sup­porting evidence for that nominal taking part in a PRODUCER role. Additionally, writer nominal-izes the verb write which is classified by Levin (Levin, 1993) as an "Image creation" or "Creation and Transformation" verb. This provides further support for assigning writer to a PRODUCER role.

For some examples the context and the individ­ual nominal affiliations provide little help in de­termining the semantic relation, such as example 5884 from before (i.e., corn flour). These ex­amples require knowledge of the interaction be­tween the nominals and we cannot rely solely on determining the role of one nominal or the other. We turned to TextRunner (Yates et al., 2007) as a large source of background knowl­edge about pre-existing relations between nom­inals. TextRunner is a queryable database of noun-verb-noun triples extracted from a large corpus of webpages. For example, the phrases re­trieved from TextRunner for "corn _ flour" include: "is ground into", "to make", "to ob­tain", and "makes". Querying in the reverse direc­tion, for "flour_corn" returns phrases such as: "contain", "filled with", "comprises", and "is made from". We use the top ten phrases for the the "<e2>_<Ei>" results, forming two fea­tures. In addition, we include a feature that has all of the hypernyms for the content words in the verb phrases from the queries for the e1-e2 direction.

Our system achieved the best overall score as mea­sured by macro-averaged Fl (for scoring details see (Hendrickx et al., 2010)) among the ten teams that participated in the semantic relation task at SemEval-2010. The results in Table 2 show the performance of the system on the test set for each relation type and the overall score.

The training data consisted of 8,000 annotated instances, including the numbered examples intro­duced earlier, and the test set contained 2,717 ex­amples. To assess the learning curve for this task we trained on sets of size 1000, 2000, 4000, and 8000, obtaining test scores of 73.08, 77.02, 79.93, and 82.19, respectively. These results indicate that more training data does help, but going from 1,000 training instances to 8,000 only boosts the score by about 9 points of F-measure.

Because our approach makes use of many dif­ferent features, we ran ablation tests on the 8 sets of features from Table 1 to determine which types of features contributed the most to classifying se­mantic relations. We evaluated all 256 (2s) combi­nations of the feature sets on the training data us­ing 10-fold cross validation. The results are shown in Table 3. The last lines of Tables 2 and 3 corre­spond to the system submitted for SemEval-2010 Task 8. The score on the training data is lower be­cause the data includes examples from SemEval-2007, which has more of the harder to classify Other relations.

These tests have shown that the NomLex-Plus feature likely did not help. Further, the depen­dency parse feature added little beyond PropBank and FrameNet. Given the high score for the lexical feature set we split it into smaller sets to see their contributions in the top portion of Table 3. This

Feature Sets Fl

Words between only 64.0

Table 3: Scores obtained for various sets of fea­tures on the training set. The bottom portion of the table shows the best combination containing 1 to 8 feature sets reveals the best individual feature is for the words between the two nominals.

By combining various linguistic resources we were able to build a state of the art system for recognizing semantic relations in text. While the large training size available in SemEval-2010 Task 8 enables achieving high scores using only word-based features, richer linguistic and background-knowledge resources still provide additional aid in identifying semantic relations.

The authors would like to thank Kirk Roberts for providing code and insightful comments.