<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="79"/><title>Automatic Event Reference Identification</title><author surname="March" givenname="Olivia"><org  name="University of Melbourne" country="Australia" city="Melbourne"/></author><author surname="Baldwin" givenname="Timothy"><org  name="University of Melbourne" country="Australia" city="Melbourne"/></author></firstpageheader><frontmatter><p><b>Automatic Event Reference Identification</b></p><p><b>Olivia March</b></p><p>CSSE</p><p>University of Melbourne Victoria 3010 Australia</p><p>oliviacm@csse.unimelb.edu.au</p></frontmatter><abstract>Event reference identification is often treated as a sentence level classification task. However, several different event ref­erences can occur within a single sentence. We present a set of experiments involving real world event reference identification at the word level in newspaper and newswire documents, addressing the issue of effec­tive text representation for classification of events using support vector machines. Our final system achieved an F-score of 0.764, significantly exceeding that of our base­line system. Additionally we achieved a marginally higher performance than a more complex comparable system. </abstract></header><body><section number="1" title="Introduction"><p>Multi-document summarization seeks to com­bine the salient information from multiple inde­pendent documents into a single coherent sum­mary. The Document Understanding Conference (DUC) shared tasks are a measured attempt to ap­ply multi-document summarization to newswire document sets, to produce a fluent 250 word sum­mary (Dang, 2006). The resultant summaries are then scored and assessed, depending on the specifics of the task.</p><p>Newswire and newspaper document sets such as those used in DUC often describe the same series of events across multiple sources, for ex­ample, media coverage of an election in Bolivia. Here, a time-line can be a more effective way of structuring a multi-document summary than a traditional paragraph-style summary. More gen­erally, historical and biographical information is frequently presented as a time-line of events, and</p><p><b>Timothy Baldwin</b></p><p>NICTA Victoria Research Laboratories University of Melbourne</p><doubt alpha="61.5" length="13" tooSmall="False" monospace="0.0">Victoria 3010</doubt><p>Australia</p><footnote>tim@csse.unimelb.edu.au</footnote><p>the ability to produce such time-lines from in­dependent sources is valuable for online content providers such as Wikipedia. The following is an illustration of the utility of time-lines in analysing topics (events highlighted in <b>bold </b>to illustrate their importance in time-line generation):</p><p>The <b>Russian Revolution </b>in 1917 was <b>triggered </b>by a combination of <b>eco­nomic breakdown, </b>war weariness, and discontent with the autocratic sys­tem of government, and it first <b>brought </b>a coalition of liberals and moderate so­cialists to <b>power, </b>but their failed poli­cies led to <b>seizure </b>of power by the Communist Bolsheviks on October 25.</p><p>Existing summarization systems, such as the DUC 07 main task system of Stokes et al. (2007), are effective at general summary tasks but have difficulty with event based summaries. For exam­ple, Stokes et al. (2007) performs well on general questions centred on a specified topic such as:</p><p>What attacks have occurred against tourists in Luxor, Egypt?</p><p>but not so well on topics requiring a listing of events, such as:</p><p>What have been the most recent sig­nificant events in the life and career of actress Angelina Jolie?</p><p>In this paper, we identify the most effective rep­resentation of text for real-world event reference identification in newspaper and newswire text us­ing support vector machines (SVMs). The work presented here is the basis of an event-based sum­mary system which will in turn form a component of the Stokes et al. (2007) system, where each question will be classified as an event-based ques­tion or a general question.<page local="2" global="80"/> The corresponding summary will then be generated by the appropri­ate sub-system, i.e. the original general-purpose system or a dedicated event-based system.</p><p>In creating an event-based time-line, three main tasks need to be considered: (1) how to identify and group references to events in text; (2) how to ground the identified events to a timestamp; and finally (3) how to identify salient events for inclu­sion in the final fixed-length time-line summary. In this paper we specifically focus on the identifi­cation of event references in text.</p><p>Often the task of event identification is linked to that of temporal resolution, whereby a system resolves the time or date of the occurrence of an event. Many systems treat events as any sentence containing an event reference and focus efforts on identifying temporal cues in the text. As there can be multiple event references within a sentence we identify events at the word level.</p><p>The following example is an illustration of the type of event-based identification we aim to achieve in this paper, relative to a plain-text input sentence (events highlighted in <b>bold):</b></p><p>Amid <b>reports </b>that thousands of Iraqi soldiers had <b>surrendered, </b>administra­tion aides were also upbeat in private, with one even <b>talking </b>of <b>victory </b>within a week.</p><p>The focus of this paper is an exploration of the most appropriate feature representation to use in a purely statistical approach to word-level event identification in newswire text.</p><p>In the remainder of this paper, we review the notion of "event" in the literature, and present an overview of the different approaches to event identification. We then describe our experimen­tal methodology, including the TimeBank corpus. Finally we describe a comparative evaluation.</p></section><section number="2" title="Related Work 2.1   Event Definition"><p>The definition of an event varies in granularity de­pending on the desired application of event ex­traction.</p><p>The Scenario Template task of the Message Understanding Conference (MUC) (Grishman and Sundheim, 1996) relied on specified domain-dependent templates which define events for the purpose of information extraction. The events that are extracted depend on the templates and documents supplied. MUC consists of domain-specific scenario (topic) based templates that sys­tems are required to fill. An example domain is financial news in the Wall Street Journal, e.g. with the scenario of change in the corporate executive management personnel of an organization (Grish-man and Sundheim, 1996).</p><p>In the context of Topic Detection and Track­ing (TDT) (Fiscus et al., 1998), an event corre­sponds to an instance of a topic identified at the document level, where the aim is to cluster doc­uments on the same topic rather than individual events within the document. For instance, given an earthquake in the Solomon Islands on August 13, in TDT <i>earthquakes </i>would be the topic and this particular earthquake instance would be the event. The task would then be to cluster all docu­ments that refer to this specific earthquake.</p><p>The aim of the Automatic Content Extraction (ACE) (Doddington et al., 2004) task is to identify and classify events of predefined semantic types. An event reference may be a noun, verb or phrase, however events are tagged at the sentence level (LDC, 2005). In the above-mentioned documents referring to the Solomon Islands earthquake, the ACE data would have each reference marked as an event, in addition to announcements made by officials and any other words/phrases referring to events within the documents.</p><p>Setzer (2001) (and later TimeML (Pustejovsky et al., 2005)) define an event as something that happens with a defined beginning and ending within the scope of a document. By comparison, a state is something that holds between entities without a definite beginning or end. The identifi­cation of events is based on verb and noun analy­sis similar in granularity to that in the ACE task. ACE and TimeML differ primarily on the seman­tic class labels for the events, and the annotated attributes and entities associated with each event. Where TimeML has seven semantic class labels (Occurrence, State, Reporting, I-Action, I-State, Aspectual and Perception), ACE has five more specific labels (Destruction/Damage, Movement, Creation/Improvement, Transfer of Possession or<page local="3" global="81"/></p><p>Control and Interaction of agents) (LDC, 2005).</p><p>For our purposes in this research, we define an event to be something that happens or occurs within the context of the document, similar to Set­zer (2001). However, we do not limit events to being denoted by only verbs or nouns. We as­sume that an event reference is contained within a sentence and does not cross sentence boundaries. Section 3.1 contains a detailed description of how an event is defined in our development data.</p><subsection number="2.2" title="Previous Work in Event Classification"><p>Approaches to event extraction vary from ontology- and frame-based systems to combined rule-based and statistical methods. In this sec­tion we describe a number of representative ap­proaches to event extraction from the literature.</p><p>The REES system (Aone and Ramos-Santacruz, 2000) uses hand-crafted event and relation ontologies to extract 100 relation and event types, 61 of which are events. It is pur­ported to be extensible, however only so far as the ontology is extended. Events and relations relate to templates with type, person, time, place, etc, as defined in the ontology. The system consists of three parts: a tagging module (containing NameTagger, NPTagger and EventTagger), a rule-based co-reference resolution module, and a template generation module (described as a non-hard-coded approach that uses declarative rules to generate and merge templates automatically to "achieve portability"). MUC style data was used for training and testing. The system achieved a 0.70 F-score over 26 event types.</p><p>Jones (2003) applied Hidden Markov Models to the automatic generation of scripts to iden­tify events. HMMs were used to capture corre­lations of significant events in text: the nodes cor­respond to events in the text, and the arcs indicate which clauses occur together in text. The context of neighbouring events and textual similarity are used to decide whether or not to group clauses.</p><p>As a component of TARSQI, aimed at inter­preting entities and events in temporally-based questions, Verhagen and Mani (2005) developed the EVITA event recognition tool. EVITA is used for event recognition in newswire text and anal­ysis of grammatical features, such as tense and aspect. Events are identified using lexical analysis, context analysis of verbs, lexical lookup of adjectival events, and machine learning to deter­mine whether an ambiguous noun is used in an event sense. They combine linguistic and sta­tistical methods to obtain a 74.0% precision and 87.3% recall (Sauir et al., 2005). Time references and events are defined by the TimeML annotation scheme.</p><p>Preprocessing of the data was carried out us­ing the Alembic Workbench for part of speech tagging, lematizing and chunking. A shallow parser is used to retrieve event referring expres­sions which are conveyed by verbs, nouns and adjectives. Verb-based events are identified by lexical lookup and contextual parsing of the ver­bal chunks. Events denoted by nouns are iden­tified via WordNet and disambiguation with a Bayesian classifier trained on SemCor. Adjec­tives are tagged as events when they come at the head of a predictive complement, such as:</p><p>A Philippine volcano, <b>dormant </b>for six centuries, ... (TimeML, 2006)</p><p>EVITA also classifies events semantically, which we do not consider in this paper.</p><p>Bethard and Martin (2006) describe a system to identify events and their semantic class for the purposes of question answering. They de­fine events as a description of situations that in­volve some internal structure, as opposed to states which describe a situation that is static or un­changing. Bethard and Martin (2006) view event identification as a classification task. Their sys­tem STEP is able to identify events with a pre­cision of 82% and recall of 71%. They use a suite of syntactic and semantic features as input to YamCha — a general purpose chunker — and TinySVM. Each word in the document is classi­fied as either beginning (B), inside (I) or outside (O) an event.</p></subsection></section><section number="3" title="Experimental Methodology"><subsection number="3.1" title="Data: TimeML and TimeBank"><p>Given that our notion of event closely mirrors that of TimeBank, we were able to use the TimeBank 1.1 corpus for system development and evalua­tion. TimeBank 1.1 consists of 186 news articles marked up with TimeML standard 1.1. The ar­ticles are sourced primarily from the Wall Street<page local="4" global="82"/></p><p>Journal and Associated Press newswire articles. They contain EVENT tags denoting events at the phrase and word level.</p><p>TimeML is a specification language for event and temporal expressions in text (Pustejovsky et al., 2005), and builds on TIDES TIMEX2 and the temporal annotation language presented in Setzer (2001).</p><p>TimeML addresses four basic problems in event-time identification:</p><p><i>• </i>Time stamping of events (anchor event in time).</p><p><i>• </i>Chronological ordering of events with re­spect to one another.</p><p><i>• </i>Reasoning with contextually under-specified temporal expressions (e.g. <i>last week).</i></p><p><i>• </i>Reasoning about the persistence of events (how long does an event or outcome of an event last).</p><p>TimeML has four major event-time structures: EVENT (see Section 3.1.1), TIMEX3 (used to de­note explicit time references), SIGNAL (used pri­marily to annotate function words indicating how temporal objects relate to one another) and LINK (denoting the relationship between an event and other events or temporal expressions).</p><subsubsection number="3.1.1" title="EVENT"><p>Events are a cover term for situations that hap­pen or occur, and can be punctual or last for a time period. Events are generally expressed by tensed or untensed verbs, nominalisations, adjectives, predicative clauses or prepositional phrases. An EVENT in TimeML contains several attributes: a unique identifier, a class, tense, aspect and option­ally another event. It may also contain another event or time to which it is temporally related.</p><p>There are seven event class types (each of which is illustrated with a set of examples):</p><p><i>• Occurrence: </i>die, crash, build, merger, sell</p><p><i>• State: </i>on board, kidnaped, in love</p><p><i>• Reporting: </i>say, report, announce</p><p><i>• I-Action: </i>attempt, try, promise, offer</p><p><i>• I-State: </i>believe, intend, want</p><p><i>• Aspectual: </i>begin, finish, stop, continue</p><p><i>• Perception: </i>see, hear, watch, feel</p><p>We currently disregard the event classes and use a binary classification, identifying each word as an event or non-event.</p></subsubsection></subsection><subsection number="3.2" title="System Description"><p>We preprocessed the TimeBank data to remove extraneous XML tags, decompose it into sen­tences and words, and discard punctuation. Fi­nally we assign each word with a POS tag using the Stanford POS tagger (Toutanova et al., 2003).</p><p>In our experiments, we use three classifica­tion algorithms: decision trees, naive Bayes and support vector machines. The chosen training and evaluation platform was WEKA (Witten and Frank, 2005). Stratified 10-fold cross-validation was used throughout, where the labeled text frag­ments were randomly allocated to different com­binations of training and testing splits. We report on the average performance over the 10 runs.</p><p>Preliminary experiments over the three classifi­cation algorithms indicated that SVMs were more effective in the classification of event references in text. BSVM (Hsu and Lin, 2002) was used for subsequent experiments, again using 10-fold cross-validation.</p><p>In Section 3.3, we present a range of different feature types, designed to explore the impact of different information sources on event reference classification.</p><p>In our classifiers, we look first at sentence-level event classification (i.e. does a given sen­tence contain one or more event references) and then word-level event classification (i.e. is a given word an event reference). The sentence-level classifier is used to filter out the non-event sen­tences, and only event-containing sentences are passed on to the word-level classifier.</p></subsection><subsection number="3.3" title="Features"><p>We use a basic word feature vector to represent the words being classified. Each distinct word in the input text corresponds to a unique feature. The text is transformed into a vector of <i>N </i>val­ues where <i>N </i>is the number of distinct words in the TimeBank corpus.<page local="5" global="83"/> The following are com­mon text representation techniques in text classi­fication, the effects of which we explore in the context of event identification.</p><p><b>• Context Window</b></p><p>For identification of events at the word level we explore the use of a context window of the zero to three preceding words, in addition to the word being classified.</p><p><b>• Feature Representation</b></p><p>We also look at the effect of representing the current and context window words as a bag of word types (binary representation), or list of position-specified words (represent­ing each word by its positional value [cur­rent word = 1, previous word = 2, etc; other words = 0]).</p><p><b>• Stop Words</b></p><p>Stop words are short, frequently occurring words such as <i>and, or, in, of. </i>Stop word removal reduces the feature space with gen­erally little impact on classification per­formance in document categorisation tasks, but has been shown to be detrimental in sentence-level classification tasks (Khoo et al., 2006). In our research, we use the stop word list of van Rijsbergen (1979).</p><doubt alpha="60.0" length="5" tooSmall="False" monospace="0.0">• POS</doubt><p>Parts of speech (POSs) are assigned to each word using the Stanford maxent POS tagger, and optionally used as an alternative to word features for context words (i.e. each context word is represented as its POS).</p><p><b>• Feature Generalisation</b></p><p>We optionally group numbers, times and named entities into single features. Num­bers and dollar amounts were identified us­ing regular expressions, and named entities were tagged using the Alembic Workbench (Aberdeen etal., 1995).</p></subsection><subsection number="3.4" title="Evaluation Method"><p>We use F-score as our primary evaluation metric, as defined below relative to Precision and Recall:</p><p>Precision</p><doubt alpha="85.7" length="7" tooSmall="False" monospace="0.0">F-Score</doubt><p>Table 1: Performance of comparable event identifi­cation systems and our baseline system, as evaluated over the TimeBank data <i># words correctly classified as events</i> <i># words that are events</i> <i>x Precision x Recall Precision + Recall</i> <i># words correctly classified as events # words classified as events</i></p><doubt alpha="46.2" length="13" tooSmall="False" monospace="0.0">Recall =- (2)</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(3)</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(1)</doubt><p>Our baseline takes the form of a unigram tag­ger, where each word is labeled with its most fre­quent label. Table 1 shows the performance of the baseline system, in addition to the the compa­rable STEP (Aone and Ramos-Santacruz, 2000) and EVITA (Sauir et al., 2005) results — as de­scribed in Section 2.2 — both of which use the TimeBank data in the development and testing of their systems.</p></subsection></section><section number="4" title="Results"><subsection number="4.1" title="Sentence Level Classification of Events"><p>In this section we first present the results of our sentence-level event identification system, where each sentence is classified as containing one or more event references, or alternatively containing no event references. Three learners — C4.5 deci­sion tree, SVM and naive Bayes — were run over a random subset of the TimeBank data, the results of which are shown in Table 2.</p><p>All three methods performed exceptionally well, at an F-score of or close to 1. This level of performance is due to 82% of the sentences in the corpus containing an event reference, and most of these containing more than one event reference. Therefore only one of the many event references within a sentence needs to be identified for the sentence to be correctly classified.</p><p>The naive Bayes classifier and decision tree models proved difficult to run over the entire TimeBank corpus due to resource constraints. As a result, we chose to use BSVM (Hsu and Lin, 2002) for all subsequent sentence-level experi­ments.</p><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">2</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Model</p></td><td class="cell"><p>Precision</p></td><td class="cell"><p>Recall</p></td><td class="cell"><p>F-Score</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Baseline</p></td><td class="cell"><p>0.800</p></td><td class="cell"><p>0.550</p></td><td class="cell"><p>0.652</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>STEP</p></td><td class="cell"><p>0.820</p></td><td class="cell"><p>0.706</p></td><td class="cell"><p>0.759</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>EVITA</p></td><td class="cell"><p>0.740</p></td><td class="cell"><p>0.873</p></td><td class="cell"><p>0.801</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="6" global="84"/><p>Table 2: Performance of sentence level classification with stop word removal and feature generalisation</p><p>As there can be multiple references to differ­ent events within a single sentence, sentence level classification is not fine-grained enough for event clustering and extraction in the context of event-based summary generation. The sentence classi­fier is instead used as a pre-filter for word-level event classification, as discussed in the following section.</p><doubt alpha="95.2" length="21" tooSmall="True" monospace="0.0">Feature Effectiveness</doubt><doubt alpha="66.7" length="81" tooSmall="True" monospace="0.0">Binary      Stop Word Removed,      Ordering     Stop Word Removed, POS, Ordered,</doubt><doubt alpha="76.3" length="38" tooSmall="True" monospace="0.0">Binary Ordering      Stop Word Removed</doubt><doubt alpha="95.0" length="20" tooSmall="True" monospace="0.0">Feature Combinations</doubt><p>Figure 1: The F-score of models trained over differ­ent feature combinations. (* = statistically significant difference: two-tailed paired t-test, <i>p &lt; </i>0.05)</p></subsection><subsection number="4.2" title="Word-level Classification of Events"><p>As stated above, in classifying individual words for event reference, we first use the sentence-level classifier to remove sentences that do not contain events. This proved to have no effect on the ac­curacy of the word level system, but did offer a slight reduction in the time required by the word­level classifier.</p><p>Both the naive Bayes classifier and C4.5 deci­sion tree resulted in majority class classifiers (see Table 1), where all words were tagged as non­events. The SVM, on the other hand, produced a variety of results, and thus forms the basis of all results presented in this section.</p><p>Below, we will discuss our findings into the effects of stop word removal, feature representa­tion, context window size, POS tagging and fea­ture generalisation on word-level event classifica­tion.</p><subsubsection number="4.2.1" title="Context Window Size"><p>Increasing the context window, using a binary feature representation, decreased the accuracy of the classifier, as shown in the leftmost segment of Figure 1. The zero word context improves only 1% with an increase from 15 to 18 thousand train­ing instances compared with an improvement of 7% for the three word prior context model. At this point, we have been unable to improve over the baseline system.</p></subsubsection><subsubsection number="4.2.2" title="Feature Representation"><p>Replacing the binary bag of words representa­tion with a position-specified binary word repre­sentation has a dramatic impact on our results, as indicated in the middle set of results in Fig­ure 1. Maintaining the ordering of 2 word prior context has an F-score of 0.68, compared to a bi­nary representation of 2 word prior context with an F-score of 0.41.</p></subsubsection><subsubsection number="4.2.3" title="Stop Words"><p>Khoo et al. (2006) found that stop word re­moval had a detrimental effect on classification at the sentence level. Our results, however, demon­strate that stop word removal offers a slight in­crease in the accuracy of the word-level classi­fiers. Table 3 shows the results obtained using SVM to identify event references at the word level using different combinations of text repre­sentation techniques. As we can see from the ta­ble, stop word removal had a positive effect on the system irrespective of whether a binary or word order representation was used. With a one word POS context window and feature generali­sation, stop word removal increases the F-score by 0.043. Similarly, using a binary representation and grouping, stop word removal increases the F-score by 0.069.</p></subsubsection><subsubsection number="4.2.4" title="Part Of Speech"><p>Including the POS tags of the context and word being classified increases performance of<page local="7" global="85"/></p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Algorithm</p></td><td class="cell"><p>Precision</p></td><td class="cell"><p>Recall</p></td><td class="cell"><p>F-Score</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Naive Bayes</p></td><td class="cell"><p>0.995</p></td><td class="cell"><p>0.998</p></td><td class="cell"><p>0.996</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SVM</p></td><td class="cell"><p>1.000</p></td><td class="cell"><p>1.000</p></td><td class="cell"><p>1.000</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>C4.5</p></td><td class="cell"><p>1.000</p></td><td class="cell"><p>1.000</p></td><td class="cell"><p>1.000</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>One i i Two k x <i>y. </i>.'i _   Three w.-.^ Baseline -------</p></td><td class="cell"><p>r</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>-</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>-</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>-</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Table 3: The results of word-level event identification using an SVM and different combinations of features</p><doubt alpha="92.9" length="14" tooSmall="True" monospace="0.0">Learning Curve</doubt><doubt alpha="5.9" length="17" tooSmall="True" monospace="0.0">0.75 I-1-1-1-1-1-</doubt><doubt alpha="4.3" length="23" tooSmall="True" monospace="0.0">0.3 I-'-'-'-'-'-'-'-'-1</doubt><doubt alpha="0.0" length="71" tooSmall="True" monospace="0.0">0        2       4        6       8       10      12      14      16 18</doubt><doubt alpha="85.0" length="40" tooSmall="True" monospace="0.0">Number of Training Instances (Thousands)</doubt><p>Figure 2: The learning curves for the models of differ­ent context window sizes, with feature generalisation and stop word removal the overall system. However, the greatest increase comes from the inclusion of the preceding context represented as a part of speech, without context words. This facilitates the generalisation of the patterns leading up to the event reference. Note that the word being tagged is retained as a binary feature, as its exclusion would leave a POS pat­tern feature that is too general given the limited combination of less than fifty distinct POS tags.</p></subsubsection><subsubsection number="4.2.5" title="Feature Generalisation"><p>Grouping specific types of words such as num­bers, named entities and time references into a single feature proved an effective way of reduc­ing the feature space. However, closer analysis of the word-level event classification results showed that some of the terms being grouped together oc­curred as events whereas others did not. There are 610 NUMBER references in the TimeBank data. 247 of these are an event or part thereof and 363 are not, leading to most NUMBER references be­ing tagged as non-events. For example, monetary amounts can be a state (type of event) if they are a milestone for a company, or alternatively they can simply be an amount lost by the company.</p><p>The insurer's earnings from commer­cial property lines <b>fell </b>59% in the lat­est quarter, while it <b>lost </b>$7.2 million in its personal property business, com­pared with earnings of <b>$6.1 million<footnote anchor="1"/> </b>a year ago.</p><p>The other groupings used, TIME and NAMED ENTITY, are never event references in the Time­Bank corpus, and thus their grouping reduces the feature space without loss of the ability to disam-biguate. Further investigation into a more fine­grained grouping of numerical terms is required before grouping numbers is included in our final system.</p></subsubsection></subsection><subsection number="4.3" title="Discussion"><p>Sentence-level classification proved a simple task with an almost perfect classification accuracy. The use of a sentence-level classifier for prior fil­tering of the word-level event system had no ef­fect of the overall F-score.</p><p>Our final system combined POS representa­tion of the two word prior context with context positional information and stop word removal to achieve an F-score of 0.764, exceeding our base­line of 0.653 and marginally exceeding the per­formance of the comparable STEP system with an F-score of 0.759. Over 50% of incorrectly classi­fied words resulted from event words which oc­cur only once in the TimeBank corpus. A fur­ther 20% are words that appear as both events and non-events in the training data. 7% of the remain­ing errors result from incorrect tokenisation during preprocessing.<page local="8" global="86"/> There are 1540 words in the TimeBank data that occur only once as events.</p><footnote label="1">States that describe circumstances in which something obtains and holds true are called predictive states. The va­lidity of this is dependent on the document creation time, and includes some quantitative statements such as those that appear in financial journals (TimeML, 2006)[pg14].</footnote><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Feature combination</p></td><td class="cell"><p>Precision</p></td><td class="cell"><p>Recall</p></td><td class="cell"><p>F-score</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Word order and grouping</p></td><td class="cell"><p>0.712</p></td><td class="cell"><p>0.675</p></td><td class="cell"><p>0.693</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Word order, grouping and stop word removal</p></td><td class="cell"><p>0.761</p></td><td class="cell"><p>0.712</p></td><td class="cell"><p>0.736</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Binary and grouping</p></td><td class="cell"><p>0.623</p></td><td class="cell"><p>0.462</p></td><td class="cell"><p>0.531</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Binary, grouping and stop word removal</p></td><td class="cell"><p>0.673</p></td><td class="cell"><p>0.542</p></td><td class="cell"><p>0.600</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>To investigate the impact of the amount of data on our results, we generate learning curves for word order-sensitive feature representation with stop word removal and feature generalisation, over diminishing amounts of the training data used in cross-validation. The curves are presented in Figure 2 over different context word window sizes. As we can see, the models based on larger context windows are still climbing steeply over the final increment of training instances, while the zero-word context model appears to be flattening out. As such, we would expect that an increase in the number of training instances to improve the performance of the models with larger con­text windows, although it remains to be seen how much they would improve relative to the smaller word windows.</p><p>Maintaining word order has the biggest effect on the performance of the classification system. Representing the preceding context as POS rather than words also increases the accuracy of the overall system and, contrary to expectations, the removal of stop words marginally increased the performance. As indicated by the learning curve (Figure 2), increasing the number of training in­stances has the potential to improve the perfor­mance of the models which include context.</p><p>The two-word POS context model mis-classifies a different subset of words to the three-word POS context model. Therefore, combining different models may increase the performance of the overall system.</p><p>The majority of errors came from previously unseen words in the training data. Future work will thus focus on how the system can better han­dle unseen words.</p></subsection></section><section number="5" title="Future Work"><p>We intend to look at the effect verb types obtained from WordNet (Fellbaum, 1998) or VERBOCEAN (Chklovski and Pantel, 2004) have on word-level event identification, to assist in better handling of unseen words. We will also look at how the sys­tem performance scales over a larger data set.</p></section><section number="6" title="Conclusion"><p>We presented a set of experiments involving real-world event reference identification at the word level in newspaper and newswire documents. We addressed the issue of effective text representation for classification of events using support vector machines. Our final system combined two word POS prior context, with positional information and stop word removal. It achieved an F-score of 0.764, significantly exceeding that of our baseline system with an F-score of 0.653.</p></section><section title="Acknowledgments"><p>NICTA is funded by the Australian government as represented by Department of Broadband, Communi­cation and Digital Economy, and the Australian Re­search Council through the ICT Centre of Excellence programme.</p></section><references><p>John Aberdeen, John D. Burger, David S. Day, Lynette Hirschman, Robinson, Patricia, and Marc B. Vilain. 1995. MITRE: Description of the Alembic system used for MUC-6. In <i>Proceedings ofMUC-6.</i></p><p>Chinatsu Aone and Mila Ramos-Santacruz. 2000. REES: A large-scale relation and event extraction system. In <i>Proceedings of the 6th Applied Natural Language Processing Conference, </i>Seattle, USA.</p><p>Steven Bethard and James Martin. 2006. Identifica­tion of event mentions and their semantic class. In <i>Proceedings of the 2006 Conference on Emprical Methods in Natural Language Processing, </i>Sydney, Australia.</p><p>Timothy Chklovski and Patrick Pantel. 2004. VER-</p><p>BOCEAN: Mining the web for fine-grained seman­tic verb relations. In <i>Proceedings ofConference on Empirical Methods in Natural Language Process­ing (EMNLP-04), </i>Barcelona, Spain.</p><doubt alpha="58.7" length="46" tooSmall="False" monospace="0.0">Hoa Trang Dang. 2006. Overview of DUC 2006. In</doubt><p><i>Proceedings ofthe Document Understanding Con­ference Workshop, </i>New York, USA.</p><p>George Doddington, Alexis Mitchell, Mark Przy-bocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. Automatic content ex­traction (ACE) program - task definitions and per­formance measures. In <i>Proceedings ofthe Fourth International Conference on Language Resources and Evaluation. </i>Lisbon, Portugal.</p><p>Christiane Fellbaum. 1998. <i>WordNet An Electronic Lexical Database. </i>The MIT Press, first edition.</p><page local="9" global="87"/><p>Jon Fiscus, George Doddington, John Garofolo, and Alvin Martin. 1998. NIST's 1998 topic detec­tion and tracking evaluation. In <i>Proceedings ofthe DARPA Broadcast News Workshop.</i></p><p>Ralph Grishman and Beth Sundheim. 1996. Message Understanding Conference 6: A brief history. In <i>Proceedings of the 16th Conference on Computa­tional Linguistics, </i>Copenhagen, Denmark.</p><p>Chih-Wei Hsu and Chih-Jen Lin. 2002. A compari­son on methods for multi-class support vector ma­chines. <i>IEEE Transactions on Neural Networks, </i>13:415-425.</p><p>Dominic R. Jones. 2003. Identifying events using similarity and context. In <i>Proceedings ofthe Sev­enth Conference on Natural Language Learning, </i>Edmonton, Canada.</p><p>Anthony Khoo, Yuval Marom, and David Albrecht.</p><p>2006. Experiments with sentence classification. In <i>Proceedings ofthe 2006 Australasian Language Technology Workshop, </i>Sydney, Australia.</p><p>LDC. 2005. ACE (automatic content extraction) En­glish annotation guidelines for events.</p><p>James Pustejovsky, Robert Ingria, Roser Sauri, Jose Castano, Jessica Littman, Rob Gaizauskas, Andrea Setzer, Graham Katz, and Inderjeet Mani. 2005. The specification language TimeML. In <i>The Lan­guage ofTime: A Reader, </i>chapter 27. Oxford Uni­versity Press.</p><p>Roser Sauir, Robert Knippen, Marc Verhagen, and James Pustejovsky. 2005. EVITA: A robust event recognizer for QA systems. In <i>Proceedings ofHu-man Language Technology Conference and Confer­ence on Empirical Methods in Natural Language Processing, </i>pages 700-707, Vancouver, Canada.</p><p>Andrea Setzer. 2001. <i>Temporal Information In Newswire Articles: An Annotation Scheme and Corpus Study. </i>Ph.D. thesis, University of Sheffield.</p><p>Nicola Stokes, Jiawen Rong, and Lawrence Cavedon.</p><p>2007. NICTA's update and question-based sum­marisation systems at DUC 2007. In <i>Proceedings ofthe Document Understanding Conference Work­shop, </i>Rochester, USA.</p><p>TimeML. 2006. TimeML annotation guidelines ver­sion 1.2.1.</p><p>Kristina Toutanova, Dan Klein, Christopher Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In <i>Proceedings of HLT-NAACL 2003, </i>pages 252­259, Edmonton, Canada.</p><p>C.J. Keith van Rijsbergen. 1979. <i>Information Re­trieval. </i>Butterworth-Heinemann, 2nd edition.</p><p>Marc Verhagen and Inderjeet Mani. 2005. Automat­ing temporal annotaion with TARSQI. In <i>Pro­ceedings of the ACL 2005 Interactive Poster and Demonstration Sessions, </i>Ann Arbor, USA.</p><p>Ian H. Witten and Eibe Frank. 2005. <i>Data Mining: Practical Machine Learning Tools and Techniques. </i>Morgan Kaufmann, San Fransisco, second edition.</p></references></body></article>