The ultimate aim of temporal processing is the au­tomatic identification of all temporal referring ex­pressions, events and temporal relations within a text. However, addressing this aim is beyond the scope of an evaluation challenge and a more mod­est approach is appropriate.

The 2007 SemEval task, TempEval-1 (Verhagen et al., 2007; Verhagen et al., 2009), was an initial evaluation exercise based on three limited tempo­ral ordering and anchoring tasks that were consid­ered realistic both from the perspective of assem­bling resources for development and testing and from the perspective of developing systems capa­ble of addressing the tasks.

TempEval-2 is based on TempEval-1, but is more elaborate in two respects: (i) it is a multilin­gual task, and (ii) it consists of six subtasks rather than three.

In the rest of this paper, we first introduce the data that we are dealing with. Which gets us in a position to present the list of task introduced by TempEval-2, including some motivation as to why we feel that it is a good idea to split up temporal relation classification into sub tasks. We proceed by shortly describing the data resources and their creation, followed by the performance of the sys­tems that participated in the tasks.

!The Semeval-2007 task was actually known simply as TempEval, but here we use Tempeval-1 to avoid confusion.

The TempEval annotation language is a simplified version of TimeML. using three TimeML tags: TIMEX3, event and tlink.

TIMEX3 tags the time expressions in the text and is identical to the TIMEX3 tag in TimeML. Times can be expressed syntactically by adverbial or prepositional phrases, as shown in the following example.

c. Thursday evening e. later this afternoon

The two main attributes of the TIMEX3 tag are type and val, both shown in the example (2).

For TempEval-2, we distinguish four temporal types: time (at 2:45 p.m.), date (January 27, 1920, yesterday), duration (two weeks) and set (every Monday morning). The val attribute as­sumes values according to an extension of the ISO 8601 standard, as enhanced by TIMEX2.

Each document has one special TIMEX3 tag, the Document Creation Time (DCT), which is in­terpreted as an interval that spans a whole day.

The EVENT tag is used to annotate those ele­ments in a text that describe what is conventionally referred to as an eventuality. Syntactically, events are typically expressed as inflected verbs, although event nominals, such as "crash" in killed by the crash, should also be annotated as EVENTS. The most salient event attributes encode tense, aspect, modality and polarity information. Examples of some of these features are shown below:

We can now define the six TempEval tasks:

A. Determine the extent of the time expressions in a text as defined by the TimeML TIMEX3 tag. In addition, determine value of the fea­tures type and val.

B. Determine the extent of the events in a text as defined by the TimeML event tag. In addition, determine the value of the features

class, tense, aspect, polarity, and modality.

C. Determine the temporal relation between an event and a time expression in the same sentence. This task is further restricted by requiring that either the event syntactically dominates the time expression or the event and time expression occur in the same noun phrase.

D. Determine the temporal relation between an event and the document creation time.

E. Determine the temporal relation between two main events in consecutive sentences.

F Determine the temporal relation between two events where one event syntactically domi­nates the other event.

Of these tasks, C, D and E were also defined for TempEval-1. However, the syntactic locality re­striction in task C was not present in TempEval-1.

Task participants could choose to either do all tasks, focus on the time expression task, focus on the event task, or focus on the four temporal rela­tion tasks. In addition, participants could choose one or more of the six languages for which we pro­vided data: Chinese, English, French, Italian, Ko­rean, and Spanish.

We feel that well-defined tasks allow us to struc­ture the workflow, allowing us to create task-specific guidelines and using task-specific anno­tation tools to speed up annotation. More im­portantly, each task can be evaluated in a fairly straightforward way, contrary to for example the problems that pop up when evaluating two com­plex temporal graphs for the same document. In addition, tasks can be ranked, allowing systems to feed the results of one (more precise) task as a fea­ture into another task.

Splitting the task into substask reduces the error rate in the manual annotation, and that merging the different sub-task into a unique layer as a post­processing operation (see figure 1) provides better

Figure 1 : Merging Relations and more reliable results (annotated data) than do­ing a complex task all at once.

The data for the five languages were prepared in­dependently of each other and do not comprise a parallel corpus. However, annotation specifica­tions and guidelines for the five languages were developed in conjunction with one other, in many cases based on version 1.2.1 of the TimeML an­notation guidelines for English. Not all corpora contained data for all six tasks. Table 1 gives the size of the training set and the relation tasks that were included.

Table 1 : Corpus size and relation tasks

All corpora include event and timex annota­tion. The French corpus contained a subcorpus with temporal relations but these relations were not split into the four tasks C through F

Annotation proceeded in two phases: a dual annotation phase where two annotators annotate each document and an adjudication phase where a judge resolves disagreements between the an­notators. Most languages used BAT, the Brandeis Annotation Tool (Verhagen, 2010), a generic web-based annotation tool that is centered around the notion of annotation tasks. With the task decom­position allowed by BAT, it is possible to structure the complex task of temporal annotation by split­ting it up in as many sub tasks as seems useful. As such, BAT was well-suited for TempEval-2 anno­tation.

We now give a few more details on the English and Spanish data, skipping the other languages for reasons that will become obvious at the beginning of section 6.

The English data sets were based on TimeBank (Pustejovsky et al., 2003; Boguraev et al., 2007), a hand-built gold standard of annotated texts us­ing the TimeML markup scheme. However, all event annotation was reviewed to make sure that the annotation complied with the latest guidelines and all temporal relations were added according to the Tempeval-2 relation tasks, using the specified relation types.

The data released for the TempEval-2 Spanish edition is a fragment of the Spanish TimeBank, currently under development. Its documents are originally from the Spanish part of the AnCora corpus (Taulé et al., 2008). Data preparation fol­lowed the annotation guidelines created to deal with the specificities of event and timex expres­sions in Spanish (Sauri et al., 2009a; Saun et al., 2009b).

For the extents of events and time expres­sions (tasks A and B), precision, recall and the fl-measure are used as evaluation metrics, using the following formulas:

Where tp is the number of tokens that are part of an extent in both key and response, fp is the number of tokens that are part of an extent in the response but not in the key, and fn is the number of tokens that are part of an extent in the key but not in the response.

For attributes of events and time expressions (the second part of tasks A and B) and for relation types (tasks C through F) we use an even simpler metric: the number of correct answers divided by the number of answers.

Eight teams participated in TempEval-2, submit­ting a grand total of eighteen systems. Some of these systems only participated in one or two tasks while others participated in all tasks. The distribu­tion over the six languages was very uneven: six­teen systems for English, two for Spanish and one for English and Spanish.

The results for task A, recognition and normal­ization of time expressions, are given in tables 2 and 3.

The results for Spanish are more uniform and generally higher than the results for English. For Spanish, the f-measure for TIMEX3 extents ranges from 0.88 through 0.91 with an average of 0.89; for English the f-measure ranges from 0.26 through 0.86, for an average of 0.78. However, due to the small sample size it is hard to make any generalizations. In both languages, type de­tection clearly was a simpler task than determining the value.

The results for task B, event recognition, are given in tables 4 and 5. Both tables contain results for both Spanish and English, the first part of each table contains the results for Spanish and the next part the results for English.

The column headers in table 5 are abbrevia­tions for polarity (pol), mood (moo), modality (mod), tense (tns), aspect (asp) and class (cl). Note that the English team chose to include modality whereas the Spanish team used mood.

Table 5 : Event attribute results

As with the time expressions results, the sample size for Spanish is small, but note again the higher f-measure for event extents in Spanish.

Table 6 shows the results for all relation tasks, with the Spanish systems in the first two rows and the English systems in the last six rows. Recall that for Spanish the training and test sets only contained data for tasks C and D.

Interestingly, the version of the TIPSem sys­tems that were applied to the Spanish data did much better on task C compared to its English cousins, but much worse on task D, which is rather puzzling.

Such a difference in performance of the systems could be due to differences in annotation accurate -ness, or it could be due to some particularities of how the two languages express certain temporal aspects, or perhaps the one corpus is more ho­mogeneous than the other. Again, there are not enough data points, but the issue deserves further attention.

For each task, the test data provided the event pairs or event-timex pairs with the relation type set to none and participating systems would re­place that value with one of the six allowed rela­tion types. However, participating systems were allowed to not replace none and not be penalized for it. Those cases would not be counted when compiling the scores in table 6. Table 7 lists those systems that did not classify all relation and the percentage of relations for each task that those sys­tems did not classify.

A comparison with the Tempeval-1 results from Semeval-2007 may be of interest. Six systems participated in the TempEval-1 tasks, compared to seven or eight systems for TempEval-2. Table 8 lists the average scores and the standard devi­ations for all the tasks (on the English data) that Tempeval-1 and Tempeval-2 have in common.

The results are very similar except for task D, but if we take a away the one outlier (the NCSU-joint score of 0.21) then the average becomes 0.78 with a standard deviation of 0.05. However, we had expected that for TempEval-2 the systems would score better on task C since we added the restriction that the event and time expression had to be syntactically adjacent. It is not clear why the results on task C have not improved.

In this paper, we described the TempEval-2 task within the SemEval 2010 competition. This task involves identifying the temporal relations be­tween events and temporal expressions in text. Us­ing a subset of TimeML temporal relations, we show how temporal relations and anchorings can be annotated and identified in six different lan­guages. The markup language adopted presents a descriptive framework with which to examine the temporal aspects of natural language informa­tion, demonstrating in particular, how tense and temporal information is encoded in specific sen­tences, and how temporal relations are encoded between events and temporal expressions. This work paves the way towards establishing a broad and open standard metadata markup language for natural language texts, examining events, tempo­ral expressions, and their orderings.

One thing that would need to be addressed in a follow-up task is what the optimal number of tasks is. Tempeval-2 had six tasks, spread out over six languages. This brought about some logisti­cal challenges that delayed data delivery and may have given rise to a situation where there was sim­ply not enough time for many systems to properly prepare. And clearly, the shared task was not suc­cessful in attracting systems to four of the six lan­guages.

Many people were involved in TempEval-2. We want to express our gratitude to the following key contributors: Nianwen Xue, Estela Saquete, Lo­tus Goldberg, Seohyun Im, André Bittar, Nicoletta Calzolari, Jessica Moszkowicz and Hyopil Shin.

Additional thanks to Joan Banach, Judith Domingo, Pau Giménez, Jimena del Solar, Teresa Sunol, Allyson Ettinger, Sharon Spivak, Nahed Abul-Hassan, Ari Abelman, John Poison, Alexan­dra Nunez, Virginia Partridge, , Amber Stubbs, Alex Plotnick, Yuping Zhou, Philippe Muller and

Irina Prodanof.

The work on the Spanish corpus was supported by a EU Marie Curie International Reintegration Grant (PIRG04-GA-2008-239414). Work on the English corpus was supported under the NSF-CRI grant 0551615, "Towards a Comprehensive Lin­guistic Annotation of Language" and the NSF-INT-0753069 project "Sustainable Interoperabil­ity for Language Technology (SILT)", funded by the National Science Foundation.

Finally, thanks to all the participants, for stick­ing with a task that was not always as flawless and timely as it could have been in a perfect world.

Marc Verhagen. 2010. The Brandeis Annotation Tool.

In Language Resources and Evaluation Conference, LREC2010, Malta.