The TempEval-2 (Pustejovsky and Verhagen, 2009) challenge proposes six tasks. Our system tackles three of these: task A - identifying time ex­pressions, assigning TIMEX3 attribute values, and anchoring them; task C - determining the tempo­ral relation between an event and time in the same sentence; and task E - determining the temporal relation between two main events in consecutive sentences. For our participation in the task, we decided to employ both rule- and ML-classifier-based approaches. Temporal expressions are dealt with by sets of rules and regular expressions, and relation labelling performed by NLTK's maxi­mum entropy classifier with rule-based processing applied during feature generation. The features (described in full in Section 2) included attributes from the TempEval-2 training data annotation, augmented by features that can be directly derived from the annotated texts. There are two main aims of this work: (1) to create a rule-based tempo­ral expression annotator that includes knowledge from work published since GUTime (Mani and Wilson, 2000) and measure its performance, and (2) to measure the performance of a classifier that includes features based on temporal signals.

Our entry to the challenge, USFD2, is a succes­sor to USFD (Hepple et al., 2007). In the rest of this paper, we will describe how USFD2 is con­structed (Section 2), and then go on to discuss its overall performance and the impact of some internal parameters on specific TempEval tasks. Regarding classifiers, we found that despite us­ing identical feature sets across relation classifi­cation tasks, performance varied significantly. We also found that USFD2 performance trends with TempEval-2 did not match those seen when clas­sifiers were trained on other data while perform­ing similar tasks. The paper closes with comments about future work.

The TempEval-2 training and test sets are parti­tioned into data for entity recognition and descrip­tion, and data for temporal relation classification. We will first discuss our approach for temporal ex­pression recognition, description and anchoring, and then discuss our approach to two of the re­lation labelling tasks.

USFD2's rule-based element for timex identifica­tion and description performs well, even achieving above-average recall despite a much smaller rule set than comparable and more complex systems. However, the temporal anchoring component per­forms less strongly. The "all-or-nothing" metric employed for evaluating the annotation of timex values gives non-strict matches a zero score (e.g. if the expected answer is 1990-05-14, no reward is given for 1990-05) even if values are close, which many were.

In previous approaches that used a maxi­mum entropy classifier and comparable feature set (Mani et al., 2006; Derczynski and Gaizauskas, 2010), the accuracy of event-event relation classi­fication was higher than that of event-timex clas­sification. Contrary to this, USFD2's event-event classification of relations between main events of successive sentences (Task E) was less accu­rate than the classification of event-timex rela­tions between events and timexes in the same sen­tence (Task C). Accuracy in Task C was good (63%), despite the lack of explicit signal/TLINK associations and the absence of a sophisticated signal recognition and association mechanism. This is higher than USFD2's accuracy in Task E (45%) though the latter is a harder task, as most TempEval-2 systems performed significantly worse at this task than event/timex relation classi­fication.

Signal information was not relied on by many cusses signals to some extent but the system de­scribed only includes a single feature - the sig­nal text), and certainly no processing of this data was performed for that challenge. USFD2 begins to leverage this information, and gives very com­petitive performance at event/timex classification. In this case, the signals provided an increase from 61.5% to 63.1% predictive accuracy in task C. The small size of the improvement might be due to the crude and unevaluated signal identification and as­sociation system that we implemented.

The performance of classifier based approaches to temporal link labelling seems to be levelling off - the 60%-70% relation labelling accuracy of work such as Mani et al. (2006) has not been greatly exceeded. This performance level is still the peak of the current generation of systems. Re­cent improvements, while employing novel ap­proaches to the task that rely on constraints be­tween temporal link types or on complex linguistic information beyond that describable by TimeML attributes, still yield marginal improvements (e.g. Yoshikawa et al. (2009)). It seems that to break through this performance "wall", we need to con­tinue to innovate with and discuss temporal re­lation labelling, using information and knowl­edge from many sources to build practical high­performance systems.

In this paper, we have presented USFD2, a novel system that annotates temporal expressions and temporal links in text. The system relies on new hand-crafted rules, existing rule sets, machine learning and temporal signal information to make its decisions. Although some of the TempEval-2 tasks are difficult, USFD2 manages to create good and useful annotations of temporal information. USFD2 is available via Google Code.

Both authors are grateful for the efforts of the TempEval-2 team and appreciate their hard work. The first author would like to acknowledge the UK Engineering and Physical Science Research Council for support in the form of a doctoral stu­dentship.