The Sheffield team were involved in TempEval as co-proposers/co-organisers of the task. Forourpar-ticipation in the task, we decided to pursue an ML-based approach, the benefits of which have been ex­plored elsewhere (Boguraev and Ando, 2005; Mani et al., 2006). For the TempEval tasks, this is easily done by treating the assignment of temporal relation types as a simple classification task, using readily available information for the instance features. More specifically, the features used were ones provided as attributes in the TempEval data annotation for the events/times being related, plus some additional fea­tures that could be straightforwardly computed from documents, i.e. without the use of more heavily 'en­gineered' NLP components. The aims of this work were three-fold. First, we wanted to see whether a 'lite' approach of this kind could yield reasonable performance, before pursuing possibilities that re­lied on using 'deeper' NLP analysis methods. Sec­ondly, we were interested to see which of the fea­tures considered would contribute positively to sys­tem performance. Thirdly, rather than selecting a single ML approach (e.g. one of those currently in vogue within NLP), we wanted to look across ML algorithms to see if any approach was better suited to the TempEval tasks than any other, and conse­quently we used the Weka workbench (Witten and Frank, 2005) in our ML experiments.

In what follows, we will first describe how our system was constructed, before going on to discuss our main observations around the key aims men­tioned above. For example, in regard to our 'lite' ap­proach, we would observe (c.f. the results reported in the Task Description paper) that although some other systems scored more highly, the score differ­ences were relatively small. Regarding features, we found for example that the system performed better for Task A when, surprisingly, the tense attribute of events was excluded. Regarding ML algo­rithms, we found not only that there was substantial variation between the effectiveness of different algo­rithms for assigning relations (as one might expect), but also that there was considerable differences in the relative effectiveness of algorithms across tasks, i.e. so that an algorithm performing well on one task (compared to the alternatives), might perform rather poorly on another task. The paper closes with some comments about future research directions.

The TempEval training and test data is marked up to identify all event and time expressions occurring within documents, and also to record the TLINK re­lations that are relevant for each task (except that TLINK relation types are absent in the test data). These annotations provide additional information about these entities in the form of XML attributes, e.g. for event annotations we find attributes such as tense, aspect, part-of-speech and so on.

Our system consists of a suite of Perl scripts that create the input files required for Weka, and handle its output. These include firstly an 'ex­traction' script, which extracts information about event, TiMEXs and tlinks from the data files, and secondly a 'feature selection/reformatting' script, which allows the information that is to be supplied to Weka to be selected, and recasts it into the format that Weka requires for its training/test files. A final script takes Weka's output over the test files and con­nects it back to the original test documents to pro­duce the final output files required for scoring.

The information that the first extraction script ex­tracts for each event, timex and tlink largely corresponds to attributes/values associated with the annotations of these items in the initial data files (although not all such attributes are of use for ma­chine learning purposes). In addition, the script de­termines for each event expression whether it is one deemed relevant by the Event Target List (ETL) for Tasks A and B. This script also maps events and TiMEXs into sequential order - intra-sentential order for task A and inter-sentential order for task C. This information can be used to compute various 'order' features, such as:

event-first: do a related event and timex (for Task A) appear with the event before or after the timex?

adjacent: do a related event and timex (again for Task A) appear adjacently in the sequence of temporal entities or not? (Note that this allows an event and timex to be adjacent if there tokens

that intervene, but not any other temporal entities.) event-between: for a related event/timex

pair, do any other events appear between them? timex-between: for a related event/timex pair, do any other timexes appear between them?

Table 1 lists all the features that we tried using for any of the three tasks. Aside from the OR­DER features (as designated in the leftmost col­umn), which were computed as just described, and the EVENT string feature (which is the literal tagged expression from the text), all other features correspond to annotation attributes. Note that the TLiNK reltype is extracted from the training data to provide the target attribute for training (a dummy value is provided for this in test data).

The output of the extraction script is converted to a format suitable for use by Weka by a second script. This script also allows a manual selection to be made as to the features that are included. For each of the three tasks, a rough-and-ready process was followed to find a 'good' set of features for use with that task, which proceeded as follows. Firstly, the maxi­mal set of features considered for the task was tried with a few ML algorithms in Weka (using a 10-fold cross-validation over the training data) to find one that seemed to work quite well for the task. Then using only that algorithm, we checked whether the string feature could be dropped (since this feature's value set was always of quite high cardinality), i.e. if its omission improved performance, which for all three tasks was the case. Next, we tried dropping each of the remaining features in turn, to identify those whose exclusion improved performance, and then for those features so identified, tried dropping them in combination to arrive at a final 'optimal' fea­ture set. Table 1 shows for each of the tasks which of the features were considered for inclusion (those marked N/A were not), and which of these remained in the final optimal feature set (/).

Having determined the set of features for use with each task, we tried out a range of ML algorithms (again with a 10-fold cross-validation over the train­ing data), to arrive at the final feature-set/ML algo­rithm combination that was used for the task in the competitive evaluation. This was trained over the entire training data and applied to the test data to produce the final submitted results.

Looking to Table 1, and the features that were con­sidered for each task and then included in the final set, various observations can be made. First, note that the string feature was omitted for all tasks, which is perhaps not surprising, since its values will be sparsely distributed, so that there will be very few training instances for most of its individual values. However, the stem feature was found to be use­ful for Task A, which can be interpreted as evidence for a 'lexical effect' on local event-timex relations, e.g. perhaps with different verbs displaying different trends in how they relate to timexes. No correspond­ing effects were observed for Tasks B and C.

The use of ORDER features for Task A was found to be useful - specifically the features indicating whether the event or timex appeared linearly first in the sentence and whether the two were adjacent or not. The more elaborate ORDER features, address­ing more specific cases of what might intervene be­tween the related timex and event expression, were not found to be helpful.

Perhaps the most striking observation to be made regarding the table is that it was found beneficial to exclude the feature tense for Task A, whilst the feature aspect was retained. We have no expla­nation to offer for this result. Likewise, the event

class feature, which distinguishes e.g. perception vs. reporting vs. aspectual etc verbs, was excluded for Task A, although it was retained for Task B.

In regard to the use of different ML algorithms for the classification tasks addressed in TempEval, we observed considerable variation between algorithms as to their performance, and this was not unexpected. However, given the seemingly high similarity of the three tasks, we were rather more surprised to see that there was considerable variation between the perfor­mance of algorithms across tasks, i.e. so that an al­gorithm performing well on one task (compared to the alternatives), might perform rather poorly on an­other task. This is illustrated by the results in Table 2 for a selected subset of the algorithms considered, which shows %-accuracy scores that were computed by cross-validation over the training data, using the feature set chosen as 'optimal' for each task. The algorithm names in the left-hand column are the ones used in WEKA (of which functions . SMO is the WEKA implementation of support-vector ma­chines or SVM). The first row of results give a 'base­line' for performance, corresponding to the assign­ment of the most common label for the task. (These were produced using WEKA's rules . ZeroR al­gorithm, which does exactly that.)

The best results observed for each task are shown in bold in the table. These best performing al­gorithms were used for the corresponding tasks in the competition. Observe that the lazy.KStar

Table 3: Competition task scores for Sheffield sys­tem (USFD), plus average/max scores across all competing systems method, which gives the best performance for Task

A, gives a rather 'middling' performance for Task

B. Similarly, the SVM method that gives the best results for Task C falls quite a way below the per­formance of KStar on Task A. A more extreme case is seen with the results for rules.JRip (Weka's implementation of the RIPPER algorithm), whose score for Task B is close to that of the best-performing system, but which scores only slightly above baseline on Task A.

The competition scores for our system are given in Table 3, shown as (harmonic) F-measures under both strict (FS) and relaxed (FR) metrics (see foot­note 2). The table also shows the average score for each task/metric across all systems taking part in the competition, as well as the maximum score returned by any system. See (Verhagen et al., 2007) for a full tabulation of results for all systems.

SIGNALs and SLINKs are possible candidates as additional features - signals obviously so, whereas the benefits of exploiting subordination information are less clear. Our initial exploratory efforts in this direction involved pulling information regard­ing SIGNALs and SLINKs across from TimeBank(Pustejovsky et al., 2003) so as to make this available for use with the TempEval tasks, in the hope that this would allow us to determine if this informa­tion would be useful without first facing the cost of developing SIGNAL and SLINK recognisers. Re­garding SIGNALs, however, we ran into the prob­lem that there are many TLINKs in the TempEval data for which no corresponding TLINK appears in TimeBank, and hence for which SIGNAL infor­mation could not be imported. We were unable to progress this work sufficiently in the time available for there to be any useful results to report here.

We have explored using a ML-based approach to the TempEval tasks, which does not rely on the use of deeper NLP-analysis components. We observe that although some other systems in the competi­tion have produced higher scores for the tasks, the score differences are relatively small. In the course of this work, we have made some interesting ob­servations regarding the performance variability of different ML algorithms when applied to the diffent TempEval tasks, and regarding the features that con­tribute to the system's performance.