<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="333"/><title>Edinburgh-LTG: TempEval-2 System Description</title><pubinfo>Proceedings of the 5th International Workshop on Semantic Evaluation, ACL 2010,pages 333-336, Uppsala, Sweden, 15-16 July 2010. ©2010 Association for Computational Linguistics</pubinfo><author surname="Grover" givenname="Claire"><org  name="University of Edinburgh" country="United Kingdom" city="Edinburgh"/></author><author surname="Tobin" givenname="Richard"><org  name="University of Edinburgh" country="United Kingdom" city="Edinburgh"/></author><author surname="Alex" givenname="Beatrice"><org  name="University of Edinburgh" country="United Kingdom" city="Edinburgh"/></author><author surname="Byrne" givenname="Kate"><org  name="University of Edinburgh" country="United Kingdom" city="Edinburgh"/></author></firstpageheader><frontmatter><p><b>Edinburgh-LTG: TempEval-2 System Description</b></p><p><b>Claire Grover, Richard Tobin, Beatrice Alex and Kate Byrne</b></p><p>University of Edinburgh Edinburgh, United Kingdom</p><p>{grover,  richard,  balex,  kbyrne3}@inf.ed.ac.uk</p></frontmatter><abstract>We describe the Edinburgh information extraction system which we are currently adapting for analysis of newspaper text as part of the SYNC3 project. Our most recent focus is geospatial and temporal grounding of entities and it has been use­ful to participate in TempEval-2 to mea­sure the performance of our system and to guide further development. We took part in Tasks A and B for English. </abstract></header><body><section number="1" title="Background"><p>The Language Technology Group (LTG) at Edin­burgh has been active in the field of information extraction (IE) for a number of years. Up until re­cently our main focus has been in biomedical IE (Alex et al., 2008) but we have also been pursuing projects in other domains, e.g. digitised histori­cal documents (Grover et al., 2010) and we are currently participants in the EU-funded SYNC 3 project where our role is to analyse news arti­cles and establish spatio-temporal and other re­lations between news events. As a step towards this goal, we have been extending and adapting our IE pipeline to ground spatial and temporal en­tities. We have developed the Edinburgh Geop-arser for georeferencing documents and have eval­uated our system against the SpatialML corpus, as reported in Tobin et al. (2010). We are cur­rently in the process of developing a rule-based date and time grounding component and it is this component that we used for Task A, which re­quires systems to identify the extents of tempo­ral named entities and provide their interpreta­tion. The TempEval-2 data also contains event en­tities and we have adapted the output of our in-house chunker (Grover and Tobin, 2006) to iden­tify events for Task B, which requires systems to identify event denoting words and to compute a range of attributes for them. In future work we will adapt our machine-learning-based relation extraction component (Haddow, 2008) to recognise re­lations between spatial and temporal entities and event entities along the lines of the linking tasks.</p></section><section number="2" title="The Edinburgh IE System"><p>Our IE system is a modular pipeline system built around the LT-XML2<footnote anchor="1"/> and LT-TTT2<footnote anchor="2"/> toolsets. Documents are converted into our internal doc­ument format and are then passed through a se­quence of linguistic components which each add XML mark-up. Early stages identify paragraphs, sentences and tokens. Part-of-speech (POS) tag­ging is done using the C&amp;C tagger (Curran and Clark, 2003a) and lemmatisation is done using morpha (Minnen et al., 2000).</p><p>We use both rule-based and machine-learning named entity recognition (NER) components, the former implemented using LT-TTT2 and the lat­ter using the C&amp;C maximum entropy NER tagger (Curran and Clark, 2003b). We are experiment­ing to find the best combination of the two dif­ferent NER views but this is not an issue in the case of date and time entities since we have taken the decision to use the rule-based output for these. The main motivation for this decision arises from the need to ground (provide temporal values for) these entities and the rules for the grounding are most naturally implemented as an elaboration of the rules for recognition.</p><p>Our IE pipeline also uses the LT-TTT2 chun­ker to provide a very shallow syntactic analysis. Figure 1 shows an example of the results of pro­cessing at the point where the rule-based NER and chunker have both applied. As can be seen from Figure 1, a positive feature for TempEval-2 is that the verb group analysis provides in­formation about tense, aspect, voice, modality and polarity which translate relatively straightfor­wardly into the Task B attributes. The noun group analysis provides verbal stem information (e.g.</p><p>'www.ltg.ed.ac.uk/software/ltxml2 www.ltg.ed.ac.uk/software/lt-ttt2<page local="2" global="334"/></p><doubt alpha="45.5" length="11" tooSmall="False" monospace="0.0">&lt;s id="sl"&gt;</doubt><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">&lt;ng&gt;</doubt><doubt alpha="46.2" length="26" tooSmall="False" monospace="0.0">&lt;w p="DT" id="wl3"&gt;The&lt;/w&gt;</doubt><doubt alpha="64.0" length="89" tooSmall="False" monospace="0.0">&lt;w p="NN"  id="wl7"  l="announcement" vstem="announce" headn="yes"&gt;announcement&lt;/w&gt; &lt;/ng&gt;</doubt><doubt alpha="52.8" length="216" tooSmall="False" monospace="0.0">&lt;vg tense="pres" voice="pass" asp="simple" modal="yes" neg="yes"&gt; &lt;w p="MD"  id="w30" pws="yes"  l="must" neg="yes"&gt;must&lt;/w&gt; &lt;w p="RB"  id="w35" pws="yes" neg="yes"&gt;not&lt;/w&gt; &lt;w p="VB"  id="w39" pws="yes" l="be"&gt;be&lt;/w&gt;</doubt><doubt alpha="50.7" length="71" tooSmall="False" monospace="0.0">&lt;w p="VBN" id="w42" pws="yes"  l="make" headv="yes"&gt;made&lt;/w&gt; &lt;/vg&gt; &lt;ng&gt;</doubt><doubt alpha="64.7" length="51" tooSmall="False" monospace="0.0">&lt;timex unit="day" trel="same" type="date" id="rbl"&gt;</doubt><doubt alpha="53.2" length="77" tooSmall="False" monospace="0.0">&lt;w unit="day" trel="same" p="NN"  id="w47" l="today"&gt;today&lt;/w&gt; &lt;/timex&gt; &lt;/ng&gt;</doubt><doubt alpha="35.3" length="34" tooSmall="False" monospace="0.0">&lt;w p="."  id="w52" sb="true"&gt;.&lt;/w&gt;</doubt><doubt alpha="25.0" length="4" tooSmall="False" monospace="0.0">&lt;/s&gt;</doubt><p>Figure 1 : Example of NER tagger and chunker output for the sentence "The announcement must not be made today."</p><p>vstem="announce") about nominalisations.</p><p>Various attributes are computed for &lt;timex&gt; elements and these are used by a temporal resolu­tion component to provide a grounding for them. The final output of the IE pipeline contains entity mark-up in "standoff" format where the entities point at the word elements using ids. The date and event entities for "made" and "today" are as follows:</p><doubt alpha="66.7" length="111" tooSmall="False" monospace="0.0">&lt;ent tense="pres" voice="pass" neg="yes" modal="yes" asp="simple" id="evl" subtype="make" type="event"&gt; &lt;parts&gt;</doubt><doubt alpha="51.0" length="51" tooSmall="False" monospace="0.0">&lt;part ew="w39" sw="w39"&gt;made&lt;/part&gt; &lt;/parts&gt; &lt;/ent&gt;</doubt><doubt alpha="52.9" length="138" tooSmall="False" monospace="0.0">&lt;ent wdaynum="5" day="Friday" date="16" month="4" year="2010" unit="day" day-number="7 3 38 7 7" tre1="same" type="date" id="rbl"&gt; &lt;parts&gt;</doubt><doubt alpha="50.9" length="53" tooSmall="False" monospace="0.0">&lt;part ew="w47"  sw="w47"&gt;today&lt;/part&gt; &lt;/parts&gt; &lt;/ent&gt;</doubt><p>The date entity has been grounded with respect to the date of writing (16th April 2010). To do the grounding we calculate a day-number value for each date where the day number count starts from 1st January 1 AD. Using this unique day number we are able to calculate the date for any given day number as well as the day of the week. We use the day number to perform simple arithmetic to ground date expressions such as "last Monday", "the day after tomorrow" etc. Grounding informa­tion is spread across the attributes for day, date, month and year. A fully grounded date has a value for all of these while an underspecified date, e.g. "2009", "March 13th", "next year", etc., only has values for some of these attributes.</p></section><section number="3" title="Adaptations for TempEval-2"><p>Our system has been developed independently of TimeML or TempEval-2 and there is therefore a gap between what our system outputs and what is contained in the TempEval-2 data. In order to run our system over the data we needed to convert it into our XML input format while preserving the tokenisation decisions from the original. Certain tokenisation mismatches required that we extend various rules to allow for alternative token bound­aries, for example, we tokenise "wasn't" as was + n' t whereas the TempEval-2 data contains was + n + ' t or occasionally wasn + ' t.</p><p>Other adaptations fall broadly into two classes: extension of our system to cover entities in TempEval-2 that we didn't previously recognise, and mapping of our output to fit TempEval-2 re­quirements.</p><subsection number="3.1" title="Extensions"><p>The date and time entities that our system recog­nises are more like the MUC7 TIMEX entities (Chinchor, 1998) than TIMEX3 ones. In partic­ular, we have focused on dates which can either be fully grounded or which, though underspeci­fied, can be grounded to a precise range, e.g. "last month" can be grounded to a particular month and year given a document creation date and it can be precisely specified if we take it to express a range from the first to last days of the month. TIMEX3 entities can be vaguer than this, for example, en­tities of type DURATION such as "twenty years", "some time", etc. can be recognised as denoting a temporal period but cannot easily be grounded. To align our output more closely to TempEval-2, we added NER rules to recognise examples such as "a long time", "recent years", "the past", "years", "some weeks", "10 minutes".<page local="3" global="335"/> In addition we needed to compute appropriate information to allow us to create TempEval-2 values such as P1W (period of 1 week).</p><p>For event recognition, our initial system created an event entity for every head verb and for ev­ery head noun which was a nominalisation. This simple approach goes a long way towards captur­ing the TempEval-2 events but results in too many false positives and false negatives for nouns. In addition our system did not calculate the informa­tion needed to compute the TempEval-2 class at­tribute. To help improve performance we added attributes to potential event entities based on look­up in lexicons compiled from the training data and from WordNet (Fellbaum, 1998). These attributes contribute to the decision as to whether a noun or verb chunk head should be an event entity or not<footnote anchor="3"/>. The lexicons derived from the training data contain the stems of all the nouns which acted more than once as events as well as information about those predicates which occurred more than once as class ASPECTUAL, ESTATE, REPORT­ING or STATE in the training data. Where look­up succeeds for event, if class look-up also suc­ceeds then the class attribute is set accordingly. If class look-up fails, the default, OCCURRENCE, is used. The WordNet derived lexicon contains in­formation about whether the first sense of a noun has event or state as a hypernym. As a result of the lexical look-up stage, the noun "work", for exam­ple, is marked as having occurred in the training data as an event and as having event as a hyper­nym for its first sense. The conjunction of these cause it to be considered to be an event entity. For verbs, the only substantive change in our system was to not consider as events all main verb uses of "be" (be happy), "have" (have a meal) and "do" (do the dishes).</p></subsection><subsection number="3.2" title="Mapping"><p>For both timex and event entities the creation of the extents files was a straightforward map­ping. For the creation of the attributes files, on the other hand, we used stylesheets to con­struct appropriate values for the TempEval-2 at­tributes based on the attributes in our output XML. The construction of event attributes is not overly complex: for example, where an event entity is specified as tense="nonfin" and</p><footnote label="3">Our system does not recognise adjective events. How­ever, passive participles, which are sometimes treated as ad­jectives in TempEval-2, are frequently treated as verbs in our system and are therefore recognised.</footnote><p>voice="pass" the TempEval-2 tense attribute is given the value PASTPART. For modality our attribute only records whether a modal verb is present or not, so it was necessary to set the TempEval-2 modality attribute to the actual modal verb inside the verb group.</p><p>For timex entities, a single value for the value attribute had to be constructed from the values of a set of attributes on our entity. For example, the information in date="16", month="4" year="2010" has to be converted to 2010-04-16. For durations other attributes provide the rel­evant information, for example for "two days" the attributes unit="day", quty="2" are used to create the value P2D (period of 2 days).</p></subsection></section><section number="4" title="Evaluation and Error Analysis"><p>The recognition results for both timex and event extents are shown in Table 1. For Task A (timex) we achieved a close balance between precision and recall, while for Task B (events) we erred towards recall at some cost to precision.</p><p>Table 1 : Extent Results</p><p>For timex entities our false negatives were all entities of the vaguest kind, for example, "10-hour", "currently", "third-quarter", "overnight", "the week": these are ones which the original sys­tem did not recognise and for which we added ex­tra rules, though evidently we were not thorough enough. The false positives were mostly of the kind that would usually be a date entity but which were not considered to be so in the key, for exam­ple, "1969", "Oct 25", "now", "the past", "a few days". In two cases the system mistakenly identi­fied numbers as times ("1.02", "2.41").</p><p>For event entities we had 73 false negatives. Some of these were caused by verbs being mistagged as nouns ("complies", "stretch", "suit") while others were nouns which didn't occur in the WordNet derived lexicon as events. There were 143 event false positives. Some of these are clearly wrong, for example, "destruction" in "weapons of mass destruction" while others are a consequence of the subtle distinctions that the TempEval-2 guidelines make and which our shal­low approach cannot easily mimic.</p><table caption="Table 2 shows the results for attribute detec­tion for both tasks. In the case of timex attributes"></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Task</b></p></td><td class="cell"><p><b>Precision</b></p></td><td class="cell"><p><b>Recall</b></p></td><td class="cell"><p><b>Fl</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Task A TaskB</p></td><td class="cell"><p>0.85 0.75</p></td><td class="cell"><p>0.82 0.85</p></td><td class="cell"><p>0.84 0.80</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="4" global="336"/><p>there was a set of entities which had systematically wrong values for both type and value: these were dates such as "this week" and "last week". These should have had DATE as their type and a value such as 1998-W19 to indicate exactly which week in which year they denote. Our date grounding does not currently cover the numbering of weeks in a year and so it would not have been possible to create appropriate values. Instead we incor­rectly treated these entities as being of type DU­RATION with value P1W. Many of the remaining errors were value errors where the system resolved relative dates as past references when they should have been future or vice versa. For example, the value for "Monday" in "He and Palestinian leader Yasser Arafat meet separately Monday with ..." should have been 1998-05-04 but our system in­terpreted it as the past Monday, 1998-04-27. There were a few cases where the value was correct but insufficient, for example for "a year ago" the sys­tem returned 1988 when it should have produced 1988-Q3.</p><p>Our scores for event attributes were high for all attributes except for class. The high scoring at­tributes were derived from the output of our chun­ker and demonstrate the quality of this component. There does not appear to be a particular pattern behind the small number of errors for these at­tributes except that errors for the pos attribute re­flect POS tagger errors and there were some com­bined tense and modality errors where "will" and "would" should have been interpreted as future tense but were instead treated as modals. The class attribute represents information that our system had not previously been designed to determine. We computed the class attribute in a relatively minimal way. Since the class value is OCCUR­RENCE in nearly 60% of events in the training data, we use this as the default but, as described in Section 3, we override this for events which are in our training data-derived lexicon as REPORTING, ASPECTUAL, ESTATE or STATE. We do not attempt to assign the LACTION class value and nearly half of our class errors result from this. An­other set of errors comes from missing REPORT­ING events such as "alleging", "telegraphed" and "acknowledged".</p></section><section title="Acknowledgements"><p>The current phase of development of the Ed­inburgh IE system is supported by the SYNC3 project (FP7-231854)<footnote anchor="4"/>.</p><table caption="Table 2: Attribute Results" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Task</b></p></td><td class="cell"><p><b>Attribute</b></p></td><td class="cell"><p><b>Score</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Task A</p></td><td class="cell"><p>type</p></td><td class="cell"><p>0.84</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>value</p></td><td class="cell"><p>0.63</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>TaskB</p></td><td class="cell"><p>polarity</p></td><td class="cell"><p>0.99</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>pos</p></td><td class="cell"><p>0.97</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>modality</p></td><td class="cell"><p>0.99</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>tense</p></td><td class="cell"><p>0.92</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>aspect</p></td><td class="cell"><p>0.98</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>class</p></td><td class="cell"><p>0.76</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><references><p>Beatrice Alex, Claire Grover, Barry Haddow, Mijail Kabadjov, Ewan Klein, Michael Matthews, Richard Tobin, and Xinglong Wang. 2008. Automating evi­ration using a natural language processing pipeline. <i>Genome Biology, </i>9(Suppl 2).</p><p>Nancy A. Chinchor. 1998. <i>Proceedings of the Sev­enth Message Understanding Conference (MUC-7). </i>Fairfax, Virginia.</p><p>James R. Curran and Stephen Clark. 2003a. Inves­tigating GIS and smoothing for maximum entropy taggers. In <i>Proceedings of the 11th Meeting of the European Chapter of the Association for Compu­tational Linguistics (EACL-03), </i>pages 91-98. Bu­dapest, Hungary.</p><p>James R. Curran and Stephen Clark. 2003b. Language independent NER using a maximum entropy tagger. In <i>Proceedings of the 7th Conference on Natural Language Learning, </i>Edmonton, Alberta, Canada.</p><p>Christiane Fellbaum, editor. 1998. <i>WordNet: An Elec­tronic Lexical Database. </i>MIT Press, Cambridge, MA.</p><p>Claire Grover and Richard Tobin. 2006. Rule-based chunking and reusability. In <i>Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC 2006).</i></p><p>Claire Grover, Richard Tobin, Kate Byrne, Matthew Woollard, James Reid, Stuart Dunn, and Julian Ball. 2010. Use of the Edinburgh geoparser for georefer-encing digitised historical collections. <i>Phil. Trans. R. Soc. A.</i></p><p>Barry Haddow. 2008. Using automated feature op­timisation to create an adaptable relation extraction system. In <i>Proc. ofBioNLP 2008, </i>Columbus, Ohio.</p><p>Guido Minnen, John Carroll, and Darren Pearce. 2000. Robust, applied morphological generation. In <i>Pro­ceedings of the 1st International Natural Language Generation Conference, </i>Mitzpe Ramon, Israel.</p><p>Richard Tobin, Claire Grover, Kate Byrne, James Reid, and Jo Walsh. 2010. Evaluation of georeferencing. In <i>Proceedings of Workshop on Geographic Infor­mation Retrieval (GIR'10).</i></p><footnote label="4">http://www.sync3.eu/</footnote></references></body></article>