<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="2"/><title>Evaluation of Generic Relation Identification</title><author surname="Hachey" givenname="Ben"><org  name="Capital Markets CRC Limited" country="Australia" city="Sydney"/></author></firstpageheader><frontmatter><p><b>Generic Relation Identification: Models and Evaluation</b></p><p><b>Ben Hachey</b></p><p>Centre for Language Technology       Capital Markets CRC Limited Macquarie University GPO Box 970</p><p>NSW 2109 Australia Sydney NSW 2001</p><p>bhachey@cmcrc.com</p></frontmatter><abstract>Generic relation identification (GRI) aims to build models of relation-forming entity pairs that can be transferred across domains without modification of model parameters. GRI has high utility in terms of cheap com­ponents for applications like summarisa­tion, automated data exploration and ini­tialisation of bootstrapping of relation ex­traction. A detailed evaluation of GRI is presented for the first time, including ex­plicit tests of portability between newswire and biomedical domains. Experimental re­sults show that a novel approach incorpo­rating dependency parsing is better in terms of recall. And, accuracy is shown to be comparable across domains. <i>[Piace </i>American] saxophonist [person David Murray] recruited [person Amidu Berry] and DJ [person Awadi] from <i>[or9anisation </i>PBS] Figure 1: Example input to GRI task (from ACE 2004). Square brackets indicate the extent of entity mentions with type as italicised superscript. Entity 1 Entity 2 Relation Type American David Murray David Murray Amidu Berry David Murray Awadi Amidu Berry PBS Awadi PBS Citizen/Resident Business Member-Of-Group Member-Of-Group Table 1: Example output from GRi task. Relation types are not part of the relation identification task but are given here for purposes of illustration. </abstract></header><body><section number="1" title="Introduction"><p>Relation extraction (RE) aims to identify men­tions of relations in text. A relation mention is de­fined as a predicate ranging over two arguments, where an argument represents concepts, objects or people in the world and the predicate describes the type of stative association or interaction that holds between the things represented by the ar­guments. Input to the RE task consists of source documents with entity mention markup (e.g., Fig­ure 1). The output is a list of relation-forming entity mention pairs and a label indicating the type of relation that holds between them (e.g., Ta­ble 1). This paper addresses the relation iden­tification task, which identifies pairs of relation-forming entity mentions (e.g., "David Murray" and "Amidu Berry" in the example).</p><p>Relation extraction (RE) can be addressed us­ing supervised (Zelenko et al., 2005; Blitzer et al., 2006), bootstrapping (Brin, 1998; Riloff and Jones, 1999; Agichtein and Gravano, 2000; Has­san et al., 2006) or generic approaches (Conrad and Utt, 1994; Hasegawa et al., 2004). One way to characterise these different approaches is in terms of adaptation cost, i.e. the amount of work necessary to adapt them to a new domain or task. in these terms, supervised approaches (including rule engineering and supervised machine learn­ing) incur the highest cost as systems need to be built largely from scratch for each new domain. Bootstrapping approaches incur less cost as they require only a small amount of seed data. And generic approaches provide domain adaptation for free as parameters do not need to be modified for new domains or tasks. Another way to characterise these approaches is in terms of the ontol­ogy creation problems they address, i.e. whether they address only the instantiation task where in­stances are added to an ontology in a new domain given a <i>relation schema </i>(the taxonomy of rela­tion types to be identified) or whether they also address the task of learning the relation schema for the new domain.<page local="2" global="3"/> in these terms, supervised approaches and bootstrapping approaches address only the ontology instantiation problem while generic approaches also address the problem of learning relation schemas from data. The trade­off is in terms of accuracy, where generic ap­proaches suffer when compared to supervised and bootstrapping approaches. However, generic ap­proaches have high utility in terms of develop­ing cheap components for applications like para­phrase acquisition (Hasegawa et al., 2005), on-demand information extraction (Sekine, 2006) and automatic summarisation (Hachey, 2009a). And, they could be used for initialisation of semi-supervised bootstrapping of relation extraction.</p><p>This paper contains the first detailed evaluation of generic relation identification (GRI), including explicit tests of portability between newswire and biomedical domains. GRi can be split into two sub-tasks, where input consists of source docu­ments with entity mention markup (as in Figure 1). The first sub-task has the goal of identifying relation-forming entity mention pairs and outputs a list of co-occurring entity mention pairs (e.g., Table 1). The second sub-task has the goal of ap­plying a ranking over co-occurring pairs that in­dicates the strength of association. This ranking might be used for filtering low confidence rela­tions or in weighting schemes for extrinsic ap­plications (e.g., automatic summarisation). The experiments here focus primarily on the identifi­cation sub-task, which is evaluated with respect to gold standard data. Experiments are reported that compare window-based models (e.g., setting a threshold on the number of intervening tokens). Results show that a novel approach incorporating intervening words and dependency paths is bet­ter in terms of recall while being statistically in­distinguishable in terms of precision and f-score. Furthermore, performance is shown to be compa­rable when porting from news to biomedical text without modification of model parameters.</p><p>Author       Co-occur Window Constraints</p></section><section number="2" title="Related Work"><p>Table 2 contains an overview of approaches from the GRI literature. The first column (Author) con­tains the first author of the approaches referenced in the following text. The first two rows corre­spond to approaches that address relation identi­fication and characterisation; the third and fourth rows correspond to approaches that focus on the GRI task; and the fifth row corresponds to a re­lated approach to generic <i>event </i>identification and characterisation. The second column (Co-occur Window) describes the co-occurrence window for identifying entity mention pairs (e.g., W/in 5 words means that entity mention pairs need to occur within five tokens of each other). The third column (Constraints) describes any addi­tional constraints placed on entity mention pairs.</p><p>Hasegawa et al. (2004) introduce the task ofre-lation discovery (using unsupervised techniques to annotate pairs of associated objects with a rela­tion type derived from the textual context). Their work includes a simple approach to GRI where all pairs of entity mentions within 5 tokens of each other are considered to be co-occurring. No motivation is given for choosing 5 as the thresh­old. In subsequent work, Zhang et al. (2005) in­corporate syntactic parsing (Collins, 1999) into their approach to GRI. All pairs of entities in the same sentence are considered to be co-occurring provided that there is a spanning parse. Neither Hasegawa et al. nor Zhang et al. explicitly evalu­ate their approaches to relation identification.</p><p>Filatova and Hatzivassiloglou (2003) describe related work that aims to extract entity pair as­sociations that constitute what they term atomic events. They consider any pair of entity men­tions co-occurring within a sentence to be pos­sible participants in event descriptions and they add a constraint requiring that a verbal 'connec­tor' (i.e., a verb or a noun that is a WordNet hyponym of event or <i>activity) </i>be present in the inter­vening token context between the entity mentions.<page local="3" global="4"/> The authors present a limited evaluation of their approach to relation identification which suggests reasonable precision. However, it is based on manual analysis of the system output so is not re-peatable. Furthermore, it does not address recall and it does not compare the system to any lower or upper bounds on accuracy.</p><table caption="Table 2: Approaches from the GRi literature." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Hasegawa</p></td><td class="cell"><p>W/in 5 words</p></td><td class="cell"><p>NA</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Zhang</p></td><td class="cell"><p>Sentence</p></td><td class="cell"><p>Spanning parse</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Conrad</p></td><td class="cell"><p>W/in 25, 100 words</p></td><td class="cell"><p>NA</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Smith</p></td><td class="cell"><p>Sentence</p></td><td class="cell"><p>NA</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Filatova</p></td><td class="cell"><p>Sentence</p></td><td class="cell"><p>Verbal connector</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Conrad and utt (1994) present seminal work on mining pairs of entities from large text collec­tions using statistical measures of association to rank named entity pairs based on co-occurrence. They propose windows of size 25 and 100, which means that any other entity mention within 25 or 100 tokens to the right or left of a given entity mention is considered to co-occur. These win­dow sizes are chosen as they roughly approximate mean sizes of paragraphs and documents in their data. The authors do not specify which window size they use for their evaluation. A manual evalu­ation of system output is reported, which suggests reasonable performance but is not repeatable.</p><p>Smith (2002) considers all pairs of entities in the same sentence to be co-occurring. He performs an evaluation using a corpus of nine­teenth century American historical documents. Extracted entity pairs are compared to a curated resource, which contains expert assessments of the severity of battles in the American civil war. Again, this suggests reasonable performance but is not repeatable. Furthermore, Smith (2002) does not compare to lower or upper bounds.</p><p>In the literature on supervised relation extrac­tion, e.g. Liu et al. (2007), features based on parse trees have been used successfully. However, be­yond requiring a spanning parse tree (Zhang et al., 2005), no previous approaches have inves­tigated the use of syntactic parsing to constrain GRI. The current work investigates the use of domain-neutral co-occurrence windows for GRI that are based on paths connecting entity mention pairs through syntactic parse trees. Furthermore, it presents the first detailed evaluation of GRI on publicly available relation extraction data.</p></section><section number="3" title="Evaluation"><p>To address previous shortcomings, a principled framework is introduced that uses gold standard relation extraction data to optimise and evaluate GRI models. This is derived from news data from the Automatic Content Extraction (ACE) 2004 and 2005 shared tasks<footnote anchor="1"/> and biomedical data de­rived from the BioInfer corpus.<footnote anchor="2"/> The ACE 2004 data is used for development experiments. The ACE 2005 data serves as the held-out news test set and the BioInfer data serves as the biomedi­cal test set. See Hachey (2009b) for details of the data preparation and experimental setup.</p><doubt alpha="52.8" length="36" tooSmall="False" monospace="0.0">GRI T/F   ACE2004   ACE2005 Biolnfer</doubt><p>Accuracy is measured in terms of precision (p) and recall (R):</p><p><i>p        NumCorrect        </i><i>r </i><i>NumCorrect TotalSystemPairs TotalGoldPairs</i></p><p>And, f-score (F) is calculated in the standard way: <i>F </i><i>= 2</i><i>PR/(P </i><i>+ </i>R). Paired Wilcoxon signed ranks tests<footnote anchor="3"/> across entity pair sub-domains are used to check for significant differences between systems. Sub-domains are formed by taking just those relations between two entities of given types (e.g., <i>Person-Organisation, Gene-Protein). </i>Table 3 contains the count of same-sentence entity men­tion pairs that constitute relation mentions (True) and those that are not (False). In the ACE 2004 and 2005 data sets, this results respectively in 949 and 558 true relation mentions spread across seven entity pair sub-domains. In the BioInfer data set, this results in 1591 true relation mentions spread across seven entity pair sub-domains.</p><p>The evaluation here also introduces an upper bound for GRI based on human agreement. This is calculated by first obtaining a mapping from entity mentions marked by annotators to entity mentions in the adjudicated gold standard annota­tion. The mapping used here is derived from the ACE 2005 evaluation script, which computes an optimised one-to-one mapping based on maximal character overlap between system and gold stan­dard entity mention strings.<page local="4" global="5"/> Given this mapping, it is possible to determine for each possible entity mention pair whether the annotators marked a re­lation mention. Interestingly, the annotators have high agreement with the adjudicated data set in terms of precision at 0.906 and lower agreement in terms of recall at 0.675. This suggests that the annotators rarely marked bad relation mentions but each missed a number of relation mentions that the other annotator marked. The mean hu­man f-score agreement is 0.773.</p><footnote label="1">http://www.nist.gov/speech/tests/ace/  2 http://mars.cs.utu.fi/BinInfer  3 The paired Wilcoxon signed ranks test is a non-parametric analogue of the paired t test. The null hypothesis is that the two populations from which the scores are sam­pled are identical. Following convention, the null hypothesis is rejected for values of p less than or equal 0.05.</footnote><table caption="Table 3: Distribution ofrelations." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>True</p></td><td class="cell"><p>949</p></td><td class="cell"><p>558</p></td><td class="cell"><p>1591</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>False</p></td><td class="cell"><p>8304</p></td><td class="cell"><p>5587</p></td><td class="cell"><p>4252</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Total</i></p></td><td class="cell"><p>9253</p></td><td class="cell"><p>6145</p></td><td class="cell"><p>5843</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><section number="4" title="Models"><p>The GRI task can be generalised in terms of the GenericRelationID algorithm in Figure 2. This takes as input an array of entity mentions <i>E </i>and the Boolean function IsPair. The IsPair function returns true if two entity mention indices constitute a co-occurring pair and false otherwise. Figure 2 includes the IsPAIRb(lse1irae function as an example, which simply counts all pairs of en­tity mentions occurring in the same sentence as relation-forming pairs like Smith (2002). The GenericRelationID algorithm starts by ini­tialising the set of entity mention pairs <b>P</b><b> </b>to the empty set. It then loops over all possible pairs of entities from <i>E, </i>which is assumed to be sorted in terms of the order of occurrence. Pairs are added to <b>P</b><b> </b>if the text describes a relation between them. The experiments here will be based on different definitions of the ISPAIR function, based on inter­vening token windows and dependency path win-dows.<footnote anchor="4"/> <b>Atomic Events </b>The first model of entity mention co-occurrence is based on the ap­proach to identifying atomic events from Fila-tova and Hatzivassiloglou (2003). This uses an IsPAlRevent function that accepts all pairs of en­tity mentions that 1) occur in the same sentence and 2) have a verbal 'connector' (i.e., a verb or a noun that is a WordNet hyponym of <i>event </i>or <i>ac­tivity) </i>in the intervening context.</p><footnote label="4">Additional experiments not reported here also explored learnt IsPair functions using decision trees and various combinations of generic features. However, these models did not generalise across domains.</footnote><p>GenericRelationID: E, IsPair 3 <b>while </b>i <b>&lt; </b><i>length(E</i><b>)</b> 5 <b>while </b>j <b>&lt; </b><i>length(E</i><b>)</b> 6 <b>if </b>IsPAlR(i,j <b>)</b></p><doubt alpha="50.0" length="6" tooSmall="False" monospace="0.0">1P —il</doubt><doubt alpha="20.0" length="5" tooSmall="False" monospace="0.0">2i —0</doubt><doubt alpha="28.6" length="7" tooSmall="False" monospace="0.0">4j—i+ 1</doubt><doubt alpha="60.0" length="5" tooSmall="False" monospace="0.0">7P—PU</doubt><doubt alpha="25.0" length="8" tooSmall="False" monospace="0.0">8 j—j+ 1</doubt><doubt alpha="25.0" length="8" tooSmall="False" monospace="0.0">9 i—i+ 1</doubt></section><section number="10" title="return P"><p><b>IsPAIR6aseime </b><b>• </b><i>i,j</i></p><doubt alpha="60.0" length="20" tooSmall="False" monospace="0.0">1ifsent(i) = sent(j)</doubt></section><section number="2" title="return true"><doubt alpha="80.0" length="5" tooSmall="False" monospace="0.0">3else</doubt></section><section number="4" title="return false"><p>Figure 2: Algorithm for generic relation identification with baseline function for identifying co-occurring en­tity mention pairs.</p><p><b>Intervening Token Windows </b>The next model is based on intervening token windows (Toks). It uses an IsPAIRtoks function that counts all pairs of entity mentions that 1) occur in the same sen­tence and 2) have <i>t </i>or fewer intervening tokens. Most previous GRI work has used some variant of this model. Hasegawa et al. (2004), for example, use the IsPAIRtoks function but do not motivate their threshold of t=5.</p><p>Figure 3 contains optimisation results for set­ting the intervening token threshold <i>t </i>on the news development data (ACE 2004). The shaded bars correspond to mean f-scores (actual value printed above the bars) for different settings of <i>t </i>(speci­fied along the bottom of the horizontal axis). The best f-score is shown in bold. Values that are sta­tistically distinguishable from the best <i>(p &lt; </i><b>0.05)</b><b> </b>are underlined. The results suggest that the best performance is achieved with <i>t </i>set to 2, though this is not reliably different from scores for <i>t</i>=1 and <i>t</i>=4 which suggests a range of optimal values from 1 to 4. For the comparisons in the rest of this paper, the Toks model should be assumed to have <i>t </i>set to 2 unless stated otherwise. Recall <i>(R) </i>and precision <i>(P) </i>are plotted as dotted grey and solid black lines respectively and are closest to being balanced at <i>t</i>=1.</p><page local="5" global="6"/><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">1</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">0</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">o o</doubt><doubt alpha="20.0" length="5" tooSmall="True" monospace="0.0">&lt;/&gt; I</doubt><doubt alpha="25.0" length="4" tooSmall="False" monospace="0.0">u_ 2</doubt><doubt alpha="0.0" length="17" tooSmall="False" monospace="0.0">0.8 0.6 0.4 0.2 0</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.427</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.445</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.028</doubt><doubt alpha="0.0" length="18" tooSmall="False" monospace="0.0">0.4140.39903850370</doubt><doubt alpha="0.0" length="20" tooSmall="False" monospace="0.0">0.3850.3700.3540 342</doubt><doubt alpha="0.0" length="11" tooSmall="False" monospace="0.0">0.326 0.317</doubt><doubt alpha="7.7" length="13" tooSmall="True" monospace="0.0">&lt;^ &lt;^ &lt;^*V &lt;^</doubt><figure caption="Figure 3: Window size results for token-based model."></figure><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">subj</doubt><doubt alpha="64.3" length="14" tooSmall="False" monospace="0.0">nnlexm°dx\\J ^</doubt><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">from</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">\</doubt><doubt alpha="61.7" length="107" tooSmall="False" monospace="0.0">American    saxophonist    David    Murray    recruited    Amidu    Berry    and    DJ    Awadi    from PBS</doubt><doubt alpha="46.3" length="82" tooSmall="False" monospace="0.0">Adjective Noun U Noun Verb U        Noun       U       U        N        Prep Noun</doubt><figure caption="Figure 4: Dependency parse for example sentence."></figure><p><b>Dependency Path Windows </b>The experiments here also consider a novel approach to modelling entity mention co-occurrence that is based on syntactic governor-dependency relations (Deps). This uses an IsPAIRdeps function that counts all pairs of entity mentions that 1) occur in the same sentence and 2) have <i>d </i>or fewer intervening token nodes on the shortest dependency path connecting the two entity mentions. Dependency paths are derived using the Minipar software (Lin, 1998), which produces 1) directional links from gover­nors to their dependent lexical items and 2) gram­matical relation types (e.g., <i>subject, object). </i>Fig­ure 4 contains the Minipar parse of the example sentence from Figure 1. The shortest dependency paths between all candidate entity mention pairs are extracted from the parse graph. The path be­tween "American" and "David Murray", for ex­ample, consists of a direct <i>modifier </i>(mod) relation with zero intervening word token nodes. The path between "David Murray" and "Awadi", on the other hand, passes through one word token node ("recruited") after post-processing operations that pass governor-dependency relations along chains of conjoined tokens, resulting in a <i>obj</i>relation be­tween recruited and Awadi.</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">%%%%</doubt><figure caption="Figure 5: Window size results for dependency-based model."></figure><p>Figure 5 contains optimisation results for set­ting the dependency path threshold <i>d </i>on the news development data (ACE 2004). The shaded bars correspond to mean f-score. The best f-score is shown in bold and is achieved at <i>d</i>=0 (which should be assumed from here). Values that are statistically distinguishable are underlined. Re­sults here suggest a range of optimal values from <i>d</i>=0 to <i>d</i>=1. Recall <i>(R)</i><i> </i>and precision <i>(P)</i><i> </i>are plot­ted as dotted grey and solid black lines respec­tively and are closest to being balanced at <i>d</i>=0.</p><page local="6" global="7"/><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">1</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">0</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">o o</doubt><doubt alpha="20.0" length="5" tooSmall="True" monospace="0.0">&lt;/&gt; I</doubt><doubt alpha="25.0" length="4" tooSmall="False" monospace="0.0">u_ 2</doubt><doubt alpha="0.0" length="15" tooSmall="False" monospace="0.0">0.8 0.6 0.4 0.2</doubt><doubt alpha="0.0" length="21" tooSmall="False" monospace="0.0">\^\ ^\ ^^&gt;^\ ^\\^\ ^\</doubt><p>VVVVVVVVVVVVVV%</p><figure caption="Figure 6: Window size results for combined (token and dependency) model."></figure><p><b>Combined Windows </b>Finally, the current work also introduces an entity mention co-occurrence model that combines token and dependency win­dows (Comb). It uses an ISPAIRcomb function that counts all pairs of entity mentions that 1) occur in the same sentence and 2) either have t or fewer intervening tokens or have d or fewer intervening dependency path nodes. Based on tuning experi­ments on the news development data (ACE 2004), the thresholds here are set to t=1 and d=0.</p><p>Figure 6 contains joint optimisation results for the intervening token (t) and dependency path (d) thresholds on the news development data (ACE 2004). The optimal system is chosen in terms of the mean rank of f-scores across entity pair sub­domains. The best mean rank is achieved with t=2 and d=0. Values that are statistically distin­guishable from the best are underlined. The re­sults suggest a range of optimal settings with t ranging from 0 to 2 and d ranging from 0 to 1.</p></section><section number="5" title="Results"><p>Table 4 contains P, <i>R </i>and <i>F </i>results. The best score for each measure is in bold and scores that are statistically distinguishable from the best <i>(p &lt; </i>0.05) are underlined. The baseline system consid­ers all pairs in the same sentence to be relations.</p><p><b>Which window function is best for identify­ing relation mentions? </b>The highest f-score on the news test data is obtained using the depen­dency path model, though this is not statistically distinguishable from the Toks or Comb models. In terms of recall, the Comb model obtains the highest score (0.538), which is significantly better than the Toks and Deps models. The Deps model, however, obtains a precision score that is signifi­cantly better than the Comb model. For the cur­rent work, the combined model is considered to be the best as it achieves the highest recall while the f-score is statistically indistinguishable from the other models. The prioritisation of recall is motivated by the fact that weighting is generally applied to co-occurring entity pairs for applica­tions of GRI. For example, relation mining ap­proaches from the related literature (Conrad and Utt, 1994; Smith, 2002) use statistical measures of association such as pointwise mutual informa­tion, &lt;\)<footnote anchor="2"/> and log likelihood ratio to estimate as­sociation strengths. Thus, a certain amount of noise in GRI should be acceptable if the subse­quent weighting scheme is assumed to give higher weight to true relation-forming entity pairs.</p><p><b>How does system performance compare to hu­man performance? </b>The main difference is in terms of precision, where the Comb model per­forms far worse than the Human upper bound (0.906). However, while Comb recall is signifi­cantly worse than Human recall (0.675), the dif­ference is not large. Furthermore, inter-annotator agreement on ACE is a very strong upper bound for the GRI task as the annotators are given de­tailed guidelines that provide a prescriptive notion of what counts as a relation mention. The GRI task, on the other hand, is not guided by a pre­defined schema and GRI predicts a number of re­lation mentions that are incorrect with respect to the gold standard annotation but could arguably be considered true relation mentions.</p><page local="7" global="8"/><doubt alpha="55.6" length="27" tooSmall="False" monospace="0.0">a) ACE 2005 (News Test Set)</doubt><p>b) BioInfer (Biomedical Test Set)</p><p>Table 4: Comparison of P, R and F on news and biomedical test sets. The best score in each column is in bold and those that are statistically distinguishable from the best are underlined.</p><p><b>Does model performance generalise across do­mains? </b>In the biomedical domain, the Comb model performs best in terms of f-score with a score of 0.453 though it is statistically indistin­guishable from the Toks model. This is a stronger result than in the news domain where there was no significant differences among the f-scores of the Toks, Deps and Comb models. Consistent with the news domain, there are no significant dif­ferences among the precision scores of the Toks, Deps and Comb models and, importantly, the Comb model is significantly better than the Toks and Deps models in terms of recall in both do­mains. Interestingly, the f-score of the Baseline model is statistically indistinguishable from the Comb model on the biomedical data. Since Base­line recall is the same for both domains (1.000), this is due to higher precision (0.268 as opposed to 0.110). This suggests that the biomedical GRI task is easier due to the higher proportion of true relation-forming pairs (27% compared to approx­imately 10% for the ACE data sets). This may be artificially high, however, since the BioInfer cre­ators selectively sampled sentences that include mentions of proteins that are known to interact. The biomedical result is consistent with the news result, however, in that Comb precision is signif­icantly better than Baseline precision on both do­mains.</p></section><section number="6" title="Discussion"><p><b>Recall and precision of the Event model </b>The low recall of the Event model with respect to the other models is not surprising due to the constraint requiring an intervening event word. The low precision, however, indicates that the constraint is not helpful as a method to cap­ture long-distance relation mentions based on intervening token windows. The Event model does particularly poorly on the ACE 2005 <i>GPE-GPE </i>and BioInfer <i>Protein-ProteinFamily </i>entity pair sub-domains due to the fact that true pairs rarely have a verbal connector in the interven­ing token context.   True relation mentions in the ACE 2005 <i>GPE-GPE </i>sub-domain tend to be geographical part-of relations where the two entity mentions are adjacent (e.g., the relation between the <i>GPE </i>entity mention "Peoria" and the <i>GPE </i>entity mention "Illinois" in the frag­ment "<u>Peoria</u>, <u>Illinois</u>"). And, true relation men­tions in the BioInfer <i>Protein-ProteinFamily </i>sub­domain tend to be appositives (e.g., the relation between the <i>Protein </i>entity mention "cofilin" and the <i>ProteinFamily </i>entity mention "actin-binding protein" in the fragment "<u>cofilin</u>, a ubiqui­tous actin-binding protein") or nominal modifiers (e.g., the relation between the <i>ProteinFamily </i>en­tity mention "cyclin-dependent kinase inhibitors" and the <i>Protein </i>entity mention "p57" in the fragment "the cyclin-dependent kinase inhibitors (CKIs) p27 and p57").</p><p><b>Error Analysis </b>For each entity pair sub­domain, ten instances were chosen randomly from the set of erroneously classified instances. These were manually inspected in order to char­acterise the types of errors made by the combined (Comb) GRI system. This suggests that the ma­jority of false positive errors in both the news and biomedical data sets (81% and 54% respec­tively) can be considered implicit relation men­tions (i.e., the relation is not explicitly stated but is more or less implicit given the context of the sentence).<page local="8" global="9"/> For example, our system posits a false positive relation between "Gul" and "Erdogan" in the sentence "Unlike the soft-spoken Gul, Erdo-gan has a reputation as a fighter." These types of false positives are not necessarily problematic in applications of GRE. In fact, these implicit relation mentions are likely to be helpful in ap­plications, e.g. representing the conceptual con­tent of a sentence for extractive summarisation (Hachey, 2009a). One not unexpected difference between domains is that there were considerably more false negative errors in the biomedical data that could be attributed to parsing errors (15% as opposed to 5% in the news data).</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>P</p></td><td class="cell"><p>R</p></td><td class="cell"><p>F</p></td><td class="cell"><p></p></td><td class="cell"><p>P</p></td><td class="cell"><p><i>R</i></p></td><td class="cell"><p><i>F</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Baseline</p></td><td class="cell"><p>0.110</p></td><td class="cell"><p>1.000</p></td><td class="cell"><p>0.195</p></td><td class="cell"><p>Baseline</p></td><td class="cell"><p>0.268</p></td><td class="cell"><p>1.000</p></td><td class="cell"><p>0.415</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Event</p></td><td class="cell"><p>0.050</p></td><td class="cell"><p>0.392</p></td><td class="cell"><p>0.083</p></td><td class="cell"><p>Event</p></td><td class="cell"><p>0.186</p></td><td class="cell"><p>0.418</p></td><td class="cell"><p>0.247</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Toks</p></td><td class="cell"><p>0.291</p></td><td class="cell"><p>0.510</p></td><td class="cell"><p>0.342</p></td><td class="cell"><p>Toks</p></td><td class="cell"><p><b>0.527</b></p></td><td class="cell"><p>0.388</p></td><td class="cell"><p>0.422</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Deps</p></td><td class="cell"><p><b>0.456</b></p></td><td class="cell"><p>0.392</p></td><td class="cell"><p><b>0.360</b></p></td><td class="cell"><p>Deps</p></td><td class="cell"><p>0.450</p></td><td class="cell"><p>0.302</p></td><td class="cell"><p>0.349</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Comb</p></td><td class="cell"><p>0.277</p></td><td class="cell"><p><b>0.538</b></p></td><td class="cell"><p>0.332</p></td><td class="cell"><p>Comb</p></td><td class="cell"><p>0.500</p></td><td class="cell"><p><b>0.454</b></p></td><td class="cell"><p><b>0.453</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Human</p></td><td class="cell"><p>0.906</p></td><td class="cell"><p>0.675</p></td><td class="cell"><p>0.773</p></td><td class="cell"><p>Human</p></td><td class="cell"><p>NA</p></td><td class="cell"><p>NA</p></td><td class="cell"><p>NA</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><doubt alpha="55.6" length="27" tooSmall="False" monospace="0.0">a) ACE 2005 (News Test Set)</doubt><p><b>Comparison of ranking methods </b>Since it is trivial to improve recall simply by increasing to­ken or dependency thresholds, improvements in f-scores require models with higher precision. One possible approach for improving precision would be to incorporate methods from the liter­ature (Conrad and Utt, 1994; Smith, 2002) for ranking entity mention pairs using statistical mea­sures of association, such as pair probability (Pr), log-likelihood (G<footnote anchor="2"/>), <i>4&gt;<footnote anchor="2"/></i>, and pointwise mutual in­formation <i>(PMI). </i>Table 5 contains correlation (point-biserial) scores that compare rank weights obtained from these measures with a binary vari­able indicating whether the instance constitutes a true relation mention according to the annotation. Following Cohen (1988), values over 0.10 (type­set in italicised bold font) are considered to indi­cate a small effect and values over 0.30 (typeset in bold font) are considered to indicate a medium effect. The table suggests that a threshold filtering low values of PMI would be the best filter for the ACE 2005 test set (small to medium correlation of 0.273, 0.356, 0.168 and 0.326 respectively for the Baseline, Toks, Deps and Comb models). On the BioInfer test set, by contrast, no measure has con­sistent correlation across systems and effect sizes are largely negligible. The highest correlation is 0.116 for G<footnote anchor="2"/> on the Comb system. While this ef­fect is small, in conjunction with the ACE 2005 results, it suggests that G<footnote anchor="2"/> would be the better ranking method for domain-neutral relation iden­tification.</p><p>Table 5: Point-biserial correlation analysis comparing a true relation mention indicator feature to various ap­proaches for ranking GRI predictions by pair associa­tion strength.</p></section><section number="7" title="Conclusions"><p>This paper presented a detailed evaluation of the generic relation identification (GRI) task, provid­ing a comparison of various window-based mod­els for the first time. It compared the interven­ing token window approach (Toks) from the liter­ature to a novel GRI approach based on windows defined over dependency paths (Deps). In addi­tion, it introduced a combined approach (Comb) that integrates the intervening token and depen­dency path models. Models were optimised on gold standard data in the news domain and ap­plied directly to data from the news and biomedi­cal domains for testing. The use of the ACE 2005 data for a news test set allowed comparison to a human upper bound for the task.</p><p>Model comparison suggested that the Deps and Comb models are best. In particular, the Comb approach performed reliably better than the other models in terms of recall while maintaining sta­tistically indistinguishable precision and f-score. High recall models were prioritised here based on the fact that applications of generic relation extraction generally incorporate a mechanism for ranking identified relation mentions. Experiments and analysis suggest that GRI accuracy is com­parable when applying the newswire-optimised models directly to the biomedical domain.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Pr</p></td><td class="cell"><p>G<footnote anchor="2"/></p></td><td class="cell"><p><i>4&gt;<footnote anchor="2"/></i></p></td><td class="cell"><p>PMI</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Baseline</p></td><td class="cell"><p>-0.093</p></td><td class="cell"><p><i>0.108</i></p></td><td class="cell"><p><i>0.262</i></p></td><td class="cell"><p><i>0.273</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Toks</p></td><td class="cell"><p>-0.098</p></td><td class="cell"><p><i>0.250</i></p></td><td class="cell"><p><b>0.329</b></p></td><td class="cell"><p><b>0.356</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Deps</p></td><td class="cell"><p>-0.092</p></td><td class="cell"><p>0.067</p></td><td class="cell"><p><i>0.145</i></p></td><td class="cell"><p><i>0.168</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Comb</p></td><td class="cell"><p>-0.091</p></td><td class="cell"><p><i>0.219</i></p></td><td class="cell"><p><i>0.294</i></p></td><td class="cell"><p><b>0.326</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>b) BioInfer (Biomedical Test Set)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Pr</p></td><td class="cell"><p>G<footnote anchor="2"/></p></td><td class="cell"><p></p></td><td class="cell"><p>PMI</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Baseline</p></td><td class="cell"><p>0.030</p></td><td class="cell"><p>0.037</p></td><td class="cell"><p><i>0.105</i></p></td><td class="cell"><p>0.073</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Toks</p></td><td class="cell"><p><i>0.114</i></p></td><td class="cell"><p><i>0.107</i></p></td><td class="cell"><p>-0.009</p></td><td class="cell"><p>-0.004</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Deps</p></td><td class="cell"><p>0.056</p></td><td class="cell"><p>0.070</p></td><td class="cell"><p>-0.023</p></td><td class="cell"><p>-0.008</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Comb</p></td><td class="cell"><p>0.081</p></td><td class="cell"><p><i>0.116</i></p></td><td class="cell"><p>0.003</p></td><td class="cell"><p>0.041</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="9" global="10"/></section><section title="Acknowledgments"><p>This work was supported by Scottish Enterprise Edinburgh-Stanford Link grant R37588 as part of the EASIE project at the University of Edinburgh. It would not have been possible without the guid­ance of Claire Grover and Mirella Lapata. I would also like to thank Robert Gaizauskas and Steve Renals for thoughtful feedback.</p></section><references><p>Eugene Agichtein and Luis Gravano. 2000. Snow­ball: Extracting relations from large plain-text col­lections. In <i>Proceedings of the 5th ACM Interna­tional Conference on Digital Libraries, </i>San Anto­nio, TX, USA. John Blitzer, Mark Dredze, and Fernando Pereira. 2006. Domain adaptation with structural corre­spondence learning. In <i>Proceedings ofthe Confer­ence on Empirical Methods in Natural Language Processing, </i>Sydney, Australia. Sergey Brin. 1998. Extracting patterns and rela­tions from the world wide web. In <i>Proceedings of the EDBT International Workshop on the Web and Databases, </i>Valencia, Spain. Jinxiu Chen, Donghong Ji, Chew Lim Tan, and Zhengyu Niu. 2005. Automatic relation extrac­tion with model order selection and discriminative label identification. In <i>Proceedings of the 2nd In­ternational Joint Conference on Natural Language Processing, </i>Jeju Island, Korea. Jinxiu Chen, Donghong Ji, Chew Lim Tan, and Zhengyu Niu. 2006. Unsupervised relation disam­biguation with order identification capabilities. In <i>Proceedings ofthe Conference on Empirical Meth­ods in Natural Language Processing, </i>Sydney, Aus­tralia.</p><p>Jacob Cohen. 1988. <i>Statistical power analysis for the behavioral sciences. </i>Lawrence Erlbaum Asso­ciates, Inc., San Diego, CA, second edition. Michael Collins. 1999. <i>Head-Driven Statistical Mod­els for Natural Language Parsing. </i>Ph.D. thesis, University of Pennsylvania. Jack G. Conrad and Mary Hunter Utt. 1994. A sys­tem for discovering relationships by feature extrac­tion from text databases. In <i>Proceedings of the 17th SIGIR, </i>Melbourne, Australia. Elena Filatova and Vasileios Hatzivassiloglou. 2003. Marking atomic events in sets of related texts. In Nicolas Nicolov, Kalina Bontcheva, Galia An-gelova, and Ruslan Mitkov, editors, <i>Recent Ad­vances in Natural Language Processing III. </i>John Benjamins, Amsterdam/Philadelphia.</p><p>Ben Hachey. 2009a. Multi-document summarisation using generic relation extraction. In <i>Proceedings ofthe Conference on Empirical Methods in Natural Language Processing, </i>Singapore. Ben Hachey. 2009b. <i>Towards Generic Relation Ex­traction. </i>Ph.D. thesis, University of Edinburgh. Takaaki Hasegawa, Satoshi Sekine, and Ralph Grish-man. 2004. Discovering relations among named entities from large corpora. In <i>Proceedings ofthe 42nd ACL, </i>Barcelona, Spain. Takaaki Hasegawa, Satoshi Sekine, and Ralph Grish-man. 2005. Unsupervised paraphrase acquisition via relation discovery.  Technical report, Proteus Project, Computer Science Department, New York University.</p><p>Hany Hassan, Ahmed Hassan, and Sara Noeman. 2006. Graph based semi-supervised approach for information extraction. In <i>Proceedings of the TextGraphs: The 2nd Workshop on Graph Based Methods for Natural Language Processing, </i>New</p><p>York, NY, USA.</p><p>Tor-Kristian Jenssen, Astrid Lœgreid, Jan Ko-morowski, and Eivind Hovig. 2001. A literature network of human genes for high-throughput analy­sis of gene expression. <i>Nature Genetics, </i>28(1):2128.</p><p>Dekang Lin. 1998. Dependency-based evaluation of MINIPAR. In <i>Proceedings ofthe LREC Workshop Evaluation ofParsing Systems, </i>Granada, Spain. Yudong Liu, Zhongmin Shi, and Anoop Sarkar. 2007. Exploiting rich syntactic information for relation­ship extraction from biomedical articles. In <i>Pro­ceedings ofNAACL-HLT, </i>Rochester, NY, USA.</p><p>Ellen Riloff and Rosie Jones. 1999. Learning dic­tionaries for information extraction by multi-level bootstrapping. In <i>Proceedings of the 16th National Conference on Artificial Intelligence, </i>Orlando, FL,</p><p>USA.</p><p>Satoshi Sekine. 2006. On-demand information ex­traction. In <i>Proceedings ofthe COLING/ACL Main Conference Poster Sessions, </i>Sydney, Australia. David A. Smith. 2002. Detecting and browsing events in unstructured text. In <i>Proceedings ofthe 25th SI-GIR, </i>Tampere, Finland. Dmitry Zelenko, Chinatsu Aone, and Jason Tib-betts. 2005. Trainable evidence extraction system (TEES). In <i>Internationl Conference on Intelligence Analysis, </i>McLean, VA, USA. Min Zhang, Jian Su, Danmei Wang, Guodong Zhou, and Chew Lim Tan. 2005. Discovering relations from a large raw corpus using tree similarity-based clustering. In <i>Proceedings ofthe 2nd IJCNLP, </i>Jeju Island, Korea.</p></references></body></article>