In recent years, the temporal structure of text has be­come a popular area of natural language processing research. Consider a sentence like:

(1) The top commander of a Cambodian resistance force said Thursday he has sent a team to recover the remains of a British mine removal expert kidnapped and presumed killed by Khmer Rouge guerrillas almost two years ago.

English speakers immediately recognize that kid­napping came first, then sending, and finally saying, even though before and after never appeared in the text. How can machines learn to do the same?

The 2007 TempEval competition tries to address this question by establishing a common corpus on which research systems can compete to find tempo­ral relations (Verhagen et al., 2007). TempEval con­siders the following types of event-time temporal re­lations:

Task A Eventsand times within the same sentence Task B Events and document times Task C Matrix verb events in adjacent sentences

In each of these tasks, systems attempt to annotate pairs with one of the following relations: b efor e,

before-or-overlap, overlap, overlap-ofafter, after or vague. Competing systems are instructed to find all temporal relations of these types in a corpus of newswire documents.

We approach these tasks as pair-wise classifi­cation problems, where each event/time pair is assigned one of the TempEval relation classes (before, after, etc.). Event/time pairs are en­coded using syntactically and semantically moti­vated features, and then used to train support vector machine (SVM) classifiers.

The remainder of this paper is structured as fol­lows. Section 2 describes the features used to char­acterize event/time relations. Section 3 explains how we used these features to train SVM models for each task. Section 4 discusses the performance of our models on the TempEval data, and Section 5 sum­marizes the lessons learned and future directions.

We used a variety of lexical, syntactic and semantic features to characterize the different types oftempo-ral relations. In each task, the events and times were characterized using the features:

word The text of the event or time words

hlMEthe quarter] VBN NP

pos The parts of speechof the words, e.g. this cru­cial moment has the parts of speech dt-jj-nn.

gov-prep Any prepositions governing the event or time, e.g. in during the Iran-Iraq war, the preposition during governs the event war, and in after ten years, the preposition after governs the time ten years.

gov-verb The verb that governs the event or time, e.g. in rejected in peace talks, the verb rejected governs the event talks, and in withdrawing on Friday, the verb withdrawing governs the time Friday. For events that are verbs, this feature is just the event itself.

gov-verb-pos The part of speech of the governing verb, e.g. withdrawing has the part of speech VBG.

aux Any auxiliary verbs and adverbs modifying the governing verb, e.g. in could not come, the words could and not are considered auxiliaries for the event come, and in will begin withdraw­ing on Friday, the words will and begin are con­sidered auxiliaries for the time Friday.

Events were further characterized using the features (the last six use gold-standard TempEval markup):

modal Whether or not the event has one ofthe aux­iliaries, can, will, shall, may, or any of their variants (could, would, etc.).

gold-stem The stem, e.g. the stem of fallen is fall.

gold-pos The part-of-speech, e.g. noun or verb.

gold-class The semantic class, e.g. reporting.

gold-tense The tense, e.g. past or present.

gold-aspect The aspect, e.g. perfective.

gold-polarity The polarity, e.g. pos or neg.

Times were further characterized using the follow­ing gold-standard TempEval features:

gold-type The type, e.g. date or time. gold-value The value, e.g. past_ref or 1990-09. gold-func The temporal function, e.g. true.

These gold-standard event and time features are sim­ilar to those used by Mani and colleagues (2006).

The features above don't capture much of the dif­ferences between the tasks, so we introduced some task-specific features. Task A included the features:

inter-time The count of time expressions between the event and time, e.g. in Figure 1, there is one time expression, Sept 50, between the event posted and the time the quarter.

inter-path The syntactic path between the event and the time, e.g. in Figure 1 the path between posted and the quarter is vbd>vp>s<pp<np<np.

inter-path-parts The path, broken into three parts: the tags from the event to the lowest common ancestor (LCA), the LCA, and the tags from the LCA to the time, e.g. in Figure 1 the parts are vbd>vp, s and pp<np<np.

inter-clause The number of clause nodes along the syntactic path, e.g. in Figure 1 there is one clause node along the path, the top s node.

Our syntactic features were derived from a syntactic tree, though Boguraev and Ando (2005) suggest that some could be derived from finite state grammars. For Task C we included the following feature:

tense-rules The relation predicted by a set of tense rules, where past tense events come before present tense events, present tense events come b efore future tense events, etc. In the text:

(2) Finally today, we [event learned] that the space agency has taken a giant leap forward. Collins will be [event named] commander of Space Shuttle Columbia.

Using the features described in the previous section, each temporal relation — an event paired with a time or another event — was translated into a set of fea­ture values. Pairing those feature values with the TempEval labels (before, after, etc.) we trained a statistical classifier for each task. We chose sup­port vector machines (SVMs) for our classifiers as they have shown good performance on a variety of natural language processing tasks (Kudo and Mat-sumoto, 2001; Pradhan et al., 2005).

Using cross-validations on the training data, we performed a simple feature selection where any fea­ture whose removal improved the cross-validation F-score was discarded. The resulting features for each task are listed in Table 1. After feature selec­tion, we set the SVM free parameters, e.g. the ker­nel degree and cost of misclassification, by perform­ing additional cross-validations on the training data, and selecting the model parameters which yielded the highest F-score for each task.

Table 1: Features used in each task. An X indicates that the feature was used for that task. For Task C, 1 indicates that the feature was used only for the first event and not the second, and 2 indicates the reverse.

Table 2: (P)recision, (R)ecall and (F)-measure of the models on each task. Precision, recall and F-measure are all equivalent to classification accuracy.

We evaluated our classifers on the TempEval test data. Because the Task A and C models derived fea­tures from the Task B temporal relations, we first ran the Task B classifer over all the data, and then ran the Task A and Task C classifiers over their individual data. The resulting temporal relation classifications were evalutated using the standard TempEval scor­ing script. Table 2 summarizes these results.

Our models achieved an accuracy of 61% on second highest scores on all these tasks. The Temp-

Task Feature Removed

Model Accuracy time-gov-prep gold-value polarity task-b-rel event-aux gold-stem gold-class event-gov-verb-2 event-aux-2 gold-class-1 gold-class-2 event-pos-2, task-b-rel

Eval scoring script also reported a relaxed measure where for example, systems could get partial credit for matching a gold standard label like OVERLAP-OR-AFTER with OVERLAP or AFTER. Under this measure, our models achieved an accuracy of 63% on Task A, 76% on Task B and 60% on Task C, again the second highest scores in the competition.

We performed a basic feature analysis where, for each feature in a task, a model was trained with that feature removed and all other features retained. We evaluated the performance of the resulting models using cross-validations on the training data. Fea­tures whose removal resulted in the largest drops in model performance are listed in Table 3.

For Task A, the most important features were the preposition governing the time and the time's nor­malized value. For Task B, the most important fea­tures were the auxiliaries governing the event, and the event's stem. For Task C, the most important features were the verb and auxiliaries governing the second event. For both Tasks A and C, the features based on the Task B relations were one of the top six features. In general however, no single feature dominated any one task — the greatest drop in per­formance from removing a feature was only 2.9%.

TempEval 2007 introduced a common dataset for work on identifying temporal relations. We framed the TempEval tasks as pair-wise classification prob­lems where pairs of events and times were assigned a temporal relation class. We introduced a variety of syntactic and semantic features, including paths be­tween constituents in a syntactic tree, and temporal relations deduced by running our Task B system on the Task A and C data. Our models achieved an ac­curacy of 61% on Task A, 75% on Task B and 54% on Task C. Analysis of these models indicated that no single feature dominated any given task, and sug­gested that future work should focus on new features to better characterize temporal relations.

This research was performed under an appointment of the first author to the DHS Scholarship and Fellowship Program, administered by the ORISE through an interagency agreement between DOE and DHS. ORISE is managed by ORAU under DOE contract number DE-AC05-06OR23100. All opin­ions expressed in this paper are the author's and do not necessarily reflect the policies and views of DHS, DOE, or ORAU/ORISE.