Coreference resolution can be described as the problem of clustering noun phrases (NP), also called mentions, into sets referring to the same discourse entity.

The "Coreference Resolution in Multiple Lan­guages task" at SemEval-2010 is meant to assess different machine learning techniques in a multi­lingual context, and by means of different evaluation metrics. Two different scenarios are considered: a gold standard scenario (only avail­able for Catalan and Spanish), where correct mention boundaries are provided to the partici­pants, and a regular scenario, where mention boundaries are to be inferred from other linguis­tic annotations provided in the input data. In par­ticular the linguistic annotations provided for each token in a sentence are: position in sentence (ID), word (TOKEN), lemma and predicted lemma (LEMMA and PLEMMA), morpho-syntactic information, both gold and/or predicted (POS and PPOS, FEAT and PFEAT), depend­ency parsing annotations (HEAD and PHEAD, DEPREL and PDEPREL), named entities (NE and PNE), and semantic roles (PRED, PPRED, and corresponding roles in the following columns). In the gold scenario, mention boundaries annotations (in column COREF) can also be used as input.

Our approach to the task was to split corefer-ence resolution into two sub-problems: mention identification and creation of entities. Mention recognition was based on the analysis of parse trees produced from input data, which were pro­duced by manual annotation or state-of-the-art dependency parsers. Once the mentions are iden­tified, coreference resolution involves partition­ing them into subsets corresponding to the same entity. This problem is cast into the binary classi­fication problem of deciding whether two given mentions are coreferent. A Maximum Entropy classifier is trained to predict how likely two mentions refer to the same entity. This is fol­lowed by a greedy procedure whose purpose is to cluster mentions into entities.

According to Ng (2005), most learning based coreference systems can be defined by four ele­ments: the learning algorithm used to train the coreference classifier, the method of creating training instances for the learner, the feature set used to represent a training or test instance, and the clustering algorithm used to coordinate the coreference classification decisions. In the fol­lowing we will detail our approach by making explicit the strategies used in each of above men­tioned components.

The data model used by our system is based on the concepts of entity and mention. The col­lection of mentions referring to the same object in a document forms an entity. A mention is an instance referring to an object: it is represented by the start and end positions in a sentence, a type and a sequence number. For convenience it also contains a frequency count and a reference to the containing sentence.

The first stage of the coreference resolution process tries to identify the occurrence of men­tions in documents.

In the training phase mentions are obtained from the NE (or PNE) column of the corpus and are partitioned into entities using the information provided in the COREF column.

In the regular setting, we used an algorithm for predicting boundaries that relies on the parse tree of the sentence produced from the gold annota­tions in columns HEAD and DEP, if available, or else from columns PHEAD and PDEP, the out­put of a dependency parser provided as input da­ta.

This analysis relied on minimal language knowledge, in order to determine possible heads of sub-trees counting as mentions, i.e. noun phrases or adverbial phrases referring to quanti­ties, times and locations. POS tags and morpho­logical features, when available, were mostly taken into account in determining mention heads. The leaves of the sub-trees of each detected head were collected as possible mentions.

The mentions identified by the NE column were then added to this set, discarding duplicates or partial overlaps. Partial overlaps in principle should not occur, but were present occasionally in the data. When this occurred, we applied a strategy to split them into a pair of mentions.

The same mention detection strategy was used also in the gold task, where we could have just returned the boundaries present in the data, scor­ing 100% in accuracy. This explains the small loss in accuracy we achieved in mention identifi­cation in the gold setting.

Relying on parse trees turned out to be quite effective, especially for languages where gold parses where available. For some other languag­es, the strategy was less effective. This was due to different annotation policies across different languages, and, in part, to inconsistencies in the data. For example in the Italian data set, named entities may include prepositions, which are typ­ically the head of the noun phrase, while our strategy of looking for noun heads leaves the preposition out of the mention boundaries. Moreover this strategy obviously fails when mentions span across sentences as was the case, again, for Italian.

For determining which mentions belong to the same entity, we applied a machine learning technique. We trained a Maximum Entropy classifier written in Python (Le, 2004) to determine whether two mentions refer to the same entity.

We did do not make any effort to optimize the number of training instances for the pair-wise learner: a positive instance is created for each anaphoric NP, paired with each of its antecedents with the same number, and a negative instance is created by pairing each NP with each of its pre­ceding non-coreferent noun phrases.

The classifier is trained using the following features, extracted for each pair of mentions.

Lexical features• Same: whether two mentions are equal;

• Prefix: whether one mention is a prefix of the other;

Suffix: whether one mention is a suffix of the other; Acronym: whether one mention is the acronym of the other.

Edit distance: quantized editing distance between two mentions.

Distance features Sentence distance: quantized distance be­tween the sentences containing the two mentions; Token distance: quantized distance be­tween the start tokens of the two mentions; Mention distance: quantized number of other mentions between two mentions.

Syntax features Head: whether the heads of two mentions have the same POS; Head POS: pairs of POS of the two men­tions heads; Count features Count: pairs of quantized numbers, each counting how many times a mention oc­curs.

Type features Type: whether two mentions have the same associated NE (Named Entity) type.

Pronoun features

When the most recent mention is a pronominal anaphora, the following features are extracted:

• Gender: pair of attributes {female, male or undetermined};

• Number: pair of attributes {singular, plur­al, undetermined};

• Pronoun type: this feature is language de­pendent and represents the type of prono­minal mention, i.e. whether the pronoun is

reflexive,possessive, relative, ...

In the submitted run we used the GIS (Genera­lized Iterative Scaling) algorithm for parameter estimation, with 600 iterations, which appeared to provide better results than using L-BFGS (a limited-memory algorithm for unconstrained op­timization). Training times ranged from one minute for German to 8 minutes for Italian, hence the slower speed of GIS was not an issue.

We used the data as supplied by the task organ­izers for all languages except Italian. A modified version of the Hunpos tagger (Halâcsy, Kornai & Oravecz, 2007; Attardi et al., 2009) was used to add to the Italian training and development cor­pora more accurate POS tags than those supplied, as well as missing information about morphol­ogy. The POS tagger we used, in fact is capable of tagging sentences with detailed POS tags, which include morphological information; this was added to column PFEATS in the data. Just for this reason our submission for Italian is to be considered an open task submission.

The Italian training corpus appears to contain several errors related to mention boundaries. In particular there are cases of entities starting in a sentence and ending in the following one. This appears to be due to sentence splitting (for in­stance at semicolons) performed after named ent­ities had been tagged. As explained in section 2, our system was not prepared to deal with these situations.

Other errors in the annotations of entities oc­curred in the Italian test data, in particular incor­rect balancing of openings and closings named entities, which caused problems to our submis­sion. We could only complete the run after the deadline, so we could only report unofficial re­sults for Italian.

We submitted results to the gold and regular challenges for the following languages: Catalan, English, German and Spanish.

Table 1 summarizes the performance of our system, according to the different accuracy scores for the gold task, Table 2 for the regular task. We have outlined in bold the cases where we achieved the best scores among the partici­pating systems.

We performed some preliminary error analysis. The goal was to identify systematic errors and possible corrections for improving the perfor­mance of our system.

We limited our analysis to the mention boun­daries detection for the regular tasks. A similar analysis for coreference detection, would require the availability of gold test data.

As described above, the strategy used for the ex­traction of mentions boundaries is based on de­pendency parse trees and named entities. This proved to be a good strategy in some languages such as Catalan (F1 score: 82.7) and Spanish (F1 score: 83.1) in which the dependency data avail­able in the corpora were very accurate and con­sistent with the annotation of named entities. In­stead, there have been unexpected problems in other languages like English or German, where the dependencies information were annotated using a different approach.

For German, while we achieved the best Baccuracy on coreference analysis in the gold set­tings, we had a quite low accuracy in mention detection (F1: 59.2), which was responsible of a significant drop in coreference accuracy for the regular task. This degradation in performance was mainly due to punctuations, which in Ger­man are linked to the sub-tree containing the noun phrase rather than to the root of the sen­tence or tokens outside the noun phrase, as it happens in Catalan and Spanish. This misled our mention detection algorithm to create many men­tions with wrong boundaries, just because punc­tuation marks were included.

In the English corpus different conventions were apparently used for dependency parsing and named entity annotations (Table 3), which pro­duced discrepancies between the boundaries of the named entities present in the data and those predicted by our algorithm. This in turn affected negatively the coreference detection algorithm that uses both types of information.

Error analysis also has shown that further im­provements could be obtained, for all languages, by using more accurate language specific extrac­tion rules. For example, we missed to consider a number of specific POS tags as possible identifi­ers for the head of noun phrases. By some simple tuning of the algorithm we obtained some im­provements.

We reported our experiments on coreference res­olution in multiple languages. We applied an ap­proach based on analyzing the parse trees in or­der to detect mention boundaries and a Maxi­mum Entropy classifier to cluster mentions into entities.

Despite a very simplistic approach, the results were satisfactory and further improvements are possible by tuning the parameters of the algo­rithms.