WordNet 3.0 (Fellbaum, 1998) is the most fre­quently used lexical database of English. As this re­source consists of lexical and semantic relations, its use constitutes an appealing option to learning rela­tions. In particular, we believe that given two men­tions of the same semantic relation, their arguments should also be similar. Or, in analogy learning terms, if R\(X\,Y\) and R2(X2,Y2) are relation mentions of the same type, then X\ Y\ as X2 Y2. Our prelim­inary experiments with WordNet suggested that few arguments of each relation are connected by imme­diate hyperonymy or meronymy relations. As a re­sult, we decided to use similarity measures defined over WordNet (Pedersen et al., 2004). The Word-Net::Similarity package (Pedersen et al., 2004) in­cludes 11 different measures, which mostly use ei­ther the WordNet glosses (lesk or vector measures) or the paths between a pair of concepts (lch; wup) to determine their relatedness.

To be able to use WordNet::Similarity, we mapped all WordNet sense keys from the training and test sets to the earlier WordNet version (2.1). Given a relation R(X, ), we computed the related-ness scores for each pair of arguments X and Y. The scores together with the sense keys of arguments were further used as features for the machine learn­ing method. As there is no a priori knowledge on what measures are the most important for each relation, all of them were used and no feature selection step has been taken.

We experimented with a number of machine learning methods such as k-nearest neighbour al­gorithm, logistic regression, bayesian networks and others. For each relation a method performing best on the training set was selected (using 5-fold cross-validation).

This classifier models the intuition that when a pair of nominals is used in similar phrases as another pair they share at least one relation, and when no such phrases can be found they do not share any relation. Applied to the semantic relation classification prob­lem this means that when a pair in the test set can be found in the same patterns as pairs from the training set, the classification for the pair will be true.

To find the patterns we followed step 1 to 6 de­scribed in (Turney, 2006), with the exception that we used both Google and the WMTS to compute pattern frequency.

First we extracted the pairs of nominals (X, Y) from the training sentences and created one Google query and a set of WMTS queries for each pair. The Google queries were of the form "X * Y " OR "Y * X". Currently, Google performs mor­phological normalization on every query, so we did not make separate queries for various endings of the nominals. For the WMTS we did make separate queries for various morphological varia­tions. We used the following set of suffixes: '-tion(slal)', '-ly', '-ist', '-ical', '-y', '-ing', '-ed', '-ies , and '-s . For this we used Peter Turney s pairs Perl package. The WMTS queries looked like [ n ]>([5].."X "..[ i ].." Y"..[5]) and [n]>([5].."Y "..[ i ].."X "..[5]) for i = 1, 2,3 and n = i +12, and for each variation of X and Y . Then we extracted sentences from the Google snippets and cut out a context of size 5, so that we were left with similar text segments as those returned by the WMTS queries. We merged the lists of text segments and counted all n-grams that contained both nominals for n = 1 to 6. We sub­stituted the nominals by variables in the n-grams with a count greater than 10 and used these as pat­terns for the classifier. An example of such a pat­tern for the Cause-Effect relation is "generation

of Y by X". After this we followed step 3 to 6 of (Turney, 2006), which left us with a matrix for each of the seven semantic relations, where each row represented a pair of nominals and each column represented the frequency of a pattern, and where each pair was classified as either true or false. The straightforward way to find pattern frequencies for the pairs in the test set would be to fill in these pat­terns with the pairs of nominals from the test set. This was not feasible given the time limitation on the task. So instead, for each pair of nominals in the test set we gathered the top-1000 snippets and computed pattern frequencies by counting how of­ten the nominals occur in every pattern on this set text segments. We constructed a matrix from these frequencies in the same way as for the training set, but without classifications for the pairs. We experi­mented with various machine learning algorithms to predict the classes of the pairs. We chose to use k-nearest neighbors, because it was the only algorithm that gave more subtle predictions than true for every pair or false for every pair. For each semantic rela­tion we used the value of k that produced the highest F\ score on 5-fold cross validation on the training data.

It has been a long tradition to use syntactic structures for relation extraction task. Some of the methods as in (Katrenko and Adriaans, 2004) have used in­formation extracted from the dependency trees. We followed similar approach by considering the paths between each pair of arguments X and Y . Ideally, if each syntactic structure is a tree, there is only one path from one node to the other. After we have ex­tracted paths, we used them as input for the string kernel methods (Hal Daumé III, 2004). The advan­tage of using string kernels is that they can handle sequences of different lengths and already proved to be efficient for a number of tasks.

All sentences in the training data were parsed using MINIPAR (Lin, 1998). From each depen­dency tree we extracted a dependency path (if any) between the arguments by collecting all lemmas (nodes) and syntactic functions (edges). The se­quences we obtained were fed into string kernel.

Alternatively to syntactic information, we also ex­tracted the snippets of the fixed length from each sentence. For each relation mention of R(X,Y), all tokens between the relation arguments X and Y were collected along with at most three tokens to the left and to the right. Unfortunately, the results we ob­tained on the training set were comparable to those obtained by string kernels on dependency paths and less accurate than the results provided by WordNet similarity measures or patterns extracted from the Web and WMTS. As a consequence, string kernel methods were not used for the final submission.

The results of the method described in Sec. 2.2 are quite far below what we expected given earlier re­sults in the literature (Turney, 2006; van Hage, Ka-trenko, and Schreiber, 2005; van Hage, Kolb, and zioni et al., 2004). We think this is caused by the fact that many pairs in the training set are non-stereotypical examples. So often the most com­monly described relation of such a pair is not the re­lation we try to classify with the pair. For example, common associations with the pair (body,parents) are that it is the parents' body, or that the parents are member of some organizing body, while it is a positive example for the Product-Producer relation. We wanted to see if this could be the case by testing whether more intuitive patterns give better results on the test set. The patterns we manually created for each relation are shown in Table 1. If a pair gives any results for these patterns on Google or WMTS, we classify the pair as true, otherwise we classify it as false. The results are shown in Table 2. We did not use these results for the submitted run, be­cause only automatic runs were permitted. The man­ual patterns did not yield many useful results at all. Apparently intuitive patterns do not capture what is required to classify the relations in the test set. The patterns we used for the Part-Whole (6) relation had an average Precision of .50, which is much lower than the average Precision found in (van Hage, Kolb, and Schreiber, 2006), which was around 0.88. We conclude that both the sets of training and test ex­amples capture different semantics of the relations than the intuitive ones, which causes common sense background knowledge, such as Google to produce bad results.

rel. patterns

Table 3 shows the results of the WordNet-based sim­ilarity measure method. In the 'methods' column, the abbreviation LR stands for logistic regression, K-NN stands for ^-nearest neighbour, and DT stands for decision trees.

Table 4 shows the results of the learnt lexical pat­terns method. For all relations we used the ^-nearest neighbour method.

relation method Prec. Recall F\ Acc.

range restrictions: Product-Producer #142 "And, of course, everyone wants to prove the truth of their be­liefs through experience, but the <e1>belief</e1> begets the <e2>experience</e2>." The common-sense domain and range restriction of the Product-Producer relation are respectively something like 'Entity' and 'Agent'. However, 'belief' is generally not considered to be an entity, and 'experience' not an agent. The definition of Product-Producer rela­tion used for the Challenge is more flexible and al­lows therefore many examples which are difficult to find by such common-sense resources as Google or WordNet.

and WMTS.