Semantic relation classification is an important but understudied language problem arising in many NLP applications, including question answering, in­formation retrieval, machine translation, word sense disambiguation, information extraction, etc. This year's SemEval (previously SensEval) competition has included a task targeting the important special case of Classification of Semantic Relations between Nominals. In the present paper we describe the UCB system which took part in that competition.

The SemEval dataset contains a total of 7 se­mantic relations (not exhaustive and possibly over­lapping), with 140 training and about 70 testing sentences per relation. Sentence classes are ap­proximately 50% negative and 50% positive ("near misses"). Table 1 lists the 7 relations together with some examples.

Examples

hormone-growth, laugh-wrinkle laser-printer, ax-murderer honey-bee, philosopher-theory grain-alcohol, desert-storm work-force, copyright-law leg-table, door-car apple-basket, plane-cargo

Table 1: SemEval dataset: Relations with examples (context sentences are not shown).

Each example consists of a sentence, two nomi-nals to be judged on whether they are in the target semantic relation, manually annotated WordNet 3.0 sense keys for these nominals, and the Web query used to obtain that example:

"Among the contents of the <e1>vessel</e1> were a set of carpenters <e2>tools</e2>, several large storage jars, ceramic utensils, ropes and remnants of food, as well as a heavy load of ballast stones." WordNet(e1) = "vessel%1:06:00::", WordNet(e2) = "tool%1:06:00::", Content-Container(e2, e1) = "true", Query = "contents of the * were a"

Lauer (1995) proposes that eight prepositions are enough to characterize the relation between nouns in a noun-noun compound: of, for, in, at, on, from, with or about. Lapata and Keller (2005) improve on his results by using Web statistics. Rosario et al. (2002) use a "descent of hierarchy", which charac­terizes the relation based on the semantic category of the two nouns. Girju et al. (2005) apply SVM, deci­sion trees, semantic scattering and iterative semantic specialization, using WordNet, word sense dis­ambiguation, and linguistic features. Barker and Sz-pakowicz (1998) propose a two-level hierarchy with 5 classes at the upper level and 30 at the lower level. Turney (2005) introduces latent relational analysis, which uses the Web, synonyms, patterns like "X for Y", "X such as Y", etc., and singular value decom­position to smooth the frequencies. Turney (2006) induces patterns from the Web, e.g. CAUSE is best characterized by "Y * causes X", and "Y in * early X" is the best pattern for TEMPORAL. Kim and Bald­win (2006) propose to use a predefined set of seed verbs and multiple resources: WordNet, CoreLex, and Moby's thesaurus. Finally, in a previous publi­cation (Nakov and Hearst, 2006), we make the claim that the relation between the nouns in a noun-noun compound can be characterized by the set of inter­vening verbs extracted from the Web.

Given an entity-annotated example sentence, we re­duce the target entities e1 and e2 to single nouns nouui and noun2, by keeping their last nouns only, which we assume to be the heads. We then mine the Web for sentences containing both noun\ and noun2, from which we extract features, con­sisting of word(s), part of speech (verb, preposi­tion, verb+preposition, coordinating conjunction), and whether noun\ precedes noun2. Table 2 shows some example features and their frequencies.

We start with a set of exact phrase queries against Google: "infh THAT * infl2", "infl2THAT * infh", "infh * infl2", and "infl2 * infh", where infh and infl2 are inflectional vari­ants of noun\ and noun2, generated using WordNet (Fellbaum, 1998); THAT can be that, which, or who; and * stands for 0 or more (up to 8) stars separated by spaces, representing the Google * single-word wildcard match operator. For each query, we collect the text snippets from the result set (up to 1000 per query), split them into sentences, assign POS tags using the OpenNLP tagger, and extract features:

Verb: If one of the nouns is the subject, and the other one is a direct or indirect object of that verb, we extract it and we lemmatize it using WordNet (Fellbaum, 1998). We ignore modals and auxil-

Table 2: Most frequent features for committee member. V stands for verb, P for preposition, and C for coordinating conjunction.

iaries, but retain the passive be, verb particles and prepositions (in case of indirect object).

Preposition: If one of the nouns is the head of an NP which contains a PP, inside which there is an NP headed by the other noun (or an inflectional form thereof), we extract the preposition heading that PP.

Coordination: If the two nouns are the heads of two coordinated NPs, we extract the coordinating conjunction.

In addition, we include some non-Web features:

Sentence word: We use as features the words from the context sentence, after stop words removal and stemming with the Porter stemmer.

Entity word: We also use the lemmas of the words that are part of ei (i = 1, 2).

Query word: Finally, we use the individual words that are part of the query string. This feature is used for category C runs only (see below).

Once extracted, the features are used to calculate the similarity between two noun pairs. Each feature triplet is assigned a weight. We wish to downweight very common features, such as "of" used as a prepo­sition in the 2 — 1 direction, so we apply tf.idf weighting to each feature. We then use the following variant of the Dice coefficient to compare the weight vectors A = (a1,..., an) and B = (b1,... ,bn):

This vector representation is similar to that of

Table 3: Task 4 results. UCB systems A1-A4.

Lin (1998), who measures word similarity by using triples extracted from a dependency parser. In par­ticular, given a noun, he finds all verbs that have it as a subject or object, and all adjectives that modify it, together with the corresponding frequencies.

Participants were asked to classify their systems into categories depending on whether they used the WordNet sense (WN) and/or the Google query (GC). Our team submitted runs for categories A (WN=no, QC=no) and C (WN=no, QC=yes) only, since we believe that having the target entities annotated with the correct WordNet senses is an unrealistic assump­tion for a real-world application.

Following Turney and Littman (2005) and Barker and Szpakowicz (1998), we used a 1-nearest-neighbor classifier. Given a test example, we calcu­lated the Dice coefficient between its feature vector

Table 4: Task 4 results. UCB systems C1 -C4.

and the vector of each of the training examples. If there was a single highest-scoring training example, we predicted its class for that test example. Oth­erwise, if there were ties for first, we assumed the class predicted by the majority of the tied examples. If there was no majority, we predicted the class that was most likely on the training data. Regardless of the classifier's prediction, if the head words of the two entities e1 and e2 had the same lemma, we clas­sified that example as negative.

Table 3 and 4 show the results for our A and C runs for different amounts of training data: 45 (A1 , C1), 90 (A2, C2), 105 (A3, C3) and 140 (A4, C4). All results are above the baseline: always propose the majority label ("true"/"false") in the test set. In fact, our category C system is the best-performing (in terms of F and Acc) among the participating systems, and we achieved the third best results for category A. Our category C results are slightly but consistently better than for A for all measures (P, R, F, Acc), which suggests that knowing the query is helpful. Interestingly, systems UCB-A2 and UCBC2 performed worse than UCB-A1 and UCB-C1 , which means that having more training data does not necessarily help with a 1NN classifier.

Table 5 shows additional analysis for A4 and C4. We study the effect of adding extra Google contexts (using up to 10 stars, rather than 8), and using differ­ent subsets of features. We show the results for: (a) leave-one-out cross-validation on the training data, (b) on the test data, and (c) our official ucb runs.

Acknowledgements: This work is supported in

Ken Barker and Stan Szpakowicz. 1998. Semi-automatic recognition of noun modifier relationships. In Proceedings ofCOLING-ACL'98, pages 96-102.

Christiane Fellbaum, editor. 1998. WordNet: An Electronic Lexical Database. MIT Press.

Roxana Girju, Dan Moldovan, Marta Tatu, and Daniel Antohe. 2005. On the semantics of noun compounds. Computer Speech and Language, 19(4):479-496.

Su Nam Kim and Timothy Baldwin. 2006. Interpreting seman­tic relations in noun compounds via verb semantics. In Pro­ceedings ofCOLING/ACL 2006. (poster), pages 491-498.

Mirella Lapata and Frank Keller. 2005. Web-based models for natural language processing. ACM Transactions on Speech and Language Processing, 2:1-31.

Mark Lauer. 1995. Designing Statistical Language Learners: Experiments on Noun Compounds. Ph.D. thesis, Department of Computing Macquarie University NSW 2109 Australia.

Dekang Lin. 1998. An information-theoretic definition of sim­ilarity. In Proceedings ofInternational Conference on Ma­chine Learning, pages 296-304.

Preslav Nakov and Marti Hearst. 2006. Using verbs to charac­terize noun-noun relations. In Proceedings ofAIMSA, pages 233-244.

Barbara Rosario, Marti Hearst, and Charles Fillmore. 2002. The descent of hierarchy, and selection in relational seman­tics. In ACL, pages 247-254.

Peter Turney and Michael Littman. 2005. Corpus-based learn­ing of analogies and semantic relations. Machine Learning Journal, 60(1-3):251-278.

Peter Turney. 2005. Measuring semantic similarity by latent relational analysis. In Proceedings IJCAI, pages 1136-1141.

Peter Turney. 2006. Expressing implicit semantic relations without supervision. In Proceedings of COLING-ACL, pages 313-320.

Table 5: Accuracy for different features and extra Web contexts: on leave-one-out cross-validation, on testing data, and in the official UCB runs.