The task of coreference resolution, defined as the identification of the expressions in a text that re­fer to the same discourse entity (1), has attracted considerable attention within the NLP community.

(1) Major League Baseball sent its head of se­curity to Chicago to review the second in­cident of an on-field fan attack in the last seven months. The league is reviewing se­curity at all ballparks to crack down on spectator violence.

Using coreference information has been shown to be beneficial in a number of NLP applications including Information Extraction (McCarthy and Lehnert, 1995), Text Summarization (Steinberger et al., 2007), Question Answering (Morton, 1999), and Machine Translation. There have been a few evaluation campaigns on coreference resolution in the past, namely MUC (Hirschman and Chinchor, 1997), ACE (Doddington et al., 2004), and ARE (Orasan et al., 2008), yet many questions remain open:

• To what extent is it possible to imple­ment a general coreference resolution system portable to different languages? How much language-specific tuning is necessary?

• How helpful are morphology, syntax and se­mantics for solving coreference relations? How much preprocessing is needed? Does its quality (perfect linguistic input versus noisy automatic input) really matter?

• How (dis)similar are different coreference evaluation metrics—MUC, B-CUBED, CEAF and BLANC? Do they all provide the same ranking? Are they correlated?

Our goal was to address these questions in a shared task. Given six datasets in Catalan, Dutch, English, German, Italian, and Spanish, the task we present involved automatically detecting full coreference chains—composed of named entities (NEs), pronouns, and full noun phrases—in four different scenarios. For more information, the reader is referred to the task website.

The rest of the paper is organized as follows. Section 2 presents the corpora from which the task datasets were extracted, and the automatic tools used to preprocess them. In Section 3, we describe the task by providing information about the data format, evaluation settings, and evaluation met­rics. Participating systems are described in Sec­tion 4, and their results are analyzed and compared in Section 5. Finally, Section 6 concludes.

In this section, we first present the sources of the data used in the task. We then describe the auto­matic tools that predicted input annotations for the coreference resolution systems.

Participants were asked to develop an automatic system capable of assigning a discourse entity to every mention, thus identifying all the NP men­tions of every discourse entity. As there is no standard annotation scheme for coreference and the source corpora differed in certain aspects, the coreference information of the task datasets was produced according to three criteria:

• Only NP constituents and possessive deter­miners can be mentions.

• Mentions must be referential expressions, thus ruling out nominal predicates, appos-itives, expletive NPs, attributive NPs, NPs within idioms, etc.

• Singletons are also considered as entities (i.e., entities with a single mention).

To help participants build their systems, the task datasets also contained both gold-standard and automatically predicted linguistic annotations at the morphological, syntactic and semantic lev­els. Considerable effort was devoted to provide participants with a common and relatively simple data representation for the six languages.

A total of twenty-two participants registered for the task and downloaded the training materials. From these, sixteen downloaded the test set but only six (out of which two task organizers) sub­mitted valid results (corresponding to nine system runs or variants). These numbers show that the task raised considerable interest but that the final participation rate was comparatively low (slightly below 30%).

The participating systems differed in terms of architecture, machine learning method, etc. Ta­ble 3 summarizes their main properties. Systems like BART and Corry support several machine learners, but Table 3 indicates the one used for the SemEval run. The last column indicates the exter­nal resources that were employed in the open set­ting, thus it is empty for systems that participated only in the closed setting. For more specific details we address the reader to the system description pa­pers in Erk and Strapparava (2010).

Table 4 shows the results obtained by two naive baseline systems: (i) Singletons considers each mention as a separate entity, and (ii) All-in-One groups all the mentions in a document into a sin­gle entity. These simple baselines reveal limita­tions of the evaluation metrics, like the high scores of CEAF and B for singletons. Interestingly enough, the naive baseline scores turn out to be hard to beat by the participating systems, as Ta­ble 5 shows. Similarly, ALL-in-one obtains high scores in terms of MUC. Table 4 also reveals dif­ferences between the distribution of entities in the datasets. Dutch is clearly the most divergent cor­pus mainly due to the fact that it only contains sin­gletons for NEs.

Table 5 displays the results of all systems for all languages and settings in the four evaluation met­rics (the best scores in each setting are highlighted in bold). Results are presented sequentially by lan­guage and setting, and participating systems are ordered alphabetically. The participation of sys­tems across languages and settings is rather irreg­ular, thus making it difficult to draw firm conclu-

nOnly 45 entries in Table 5 from 192 potential cases.

This paper has introduced the main features of the SemEval-2010 task on coreference resolution.

We would like to thank the following peo­ple who contributed to the preparation of the task datasets: Manuel Bertran (UB), Oriol Borrega (UB), Orphée De Clercq (U. Ghent), Francesca Delogu (U. Trento), Jesus Giménez (UPC), Eduard Hovy (ISI-USC), Richard Johans­son (U. Trento), Xavier Lluls (UPC), Montse Nofre (UB), Lluis Padrö (UPC), Kepa Joseba Rodriguez (U. Trento), Mihai Surdeanu (Stan­ford), Olga Uryupina (U. Trento), Lente Van Leu-ven (UB), and Rita Zaragoza (UB). We would also like to thank LDC and die tageszeitung for dis­tributing freely the English and German datasets.

This work was funded in part by the Span­ish Ministry of Science and Innovation through the projects TEXT-MESS 2.0 (TIN2009-13391-C04-04), OpenMT-2 (TIN2009-14675-C03), and KNOW2 (TIN2009-14715-C04-04), and an FPU doctoral scholarship (AP2006-00994) held by M. Recasens. It also received financial sup­port from the Seventh Framework Programme of the EU (FP7/2007-2013) under GA 247762 (FAUST), from the STEVIN program of the Ned-erlandse Taalunie through the COREA and SoNaR projects, and from the Provincia Autonoma di Trento through the LiveMemories project.