The language dependent modules contain finite state expressions that detect the heads based on the linguistic annotations. Such a language module re­quires a development time of approximately 1 per­son hour in order to adapt the regular expressions

# Feature Description

^Trainingjàet^^ Test Set

MB Classifier

/Resulting Feature^ V_Vectors_J

(^Postprocessor^)' Final Results ^)

Ç Test Set ^

Markables "N y Extractor y

Figure 1 : Overview of the system.

rrij - the antecedent rrik - the mention to be resolved

Y if rrij is pron.; else N

Y if rrij is subject; else N

Y if rrij is a nested NP; else N number - Sg. or PI.

Y if rrik is a pronoun; else N

Y if rrik is a nested NP; else N semantic class - extracted from the NEs in the data the nominative case of rrik if pron.; else NA

C if the mentions are the same string; else I

C if one mention is a substring of the other; else I

C if both mentions are pron. and same string; else I

C if both mentions are both non-pron. and same string; else I values for f. 6 for rrij and rrik values for f. 7 for rrij and rrik values for f. 3 for rrij and rrik values for f. 5 for rrij and rrik values for f. 10 for rrij and rrik values for f. 11 for rrij and rrik

to the given language data (different POS tagsets, differences in the provided annotations). This is the only language dependent part of the system.

We decided to separate the task of finding heads of markables, which then serve as the basis for the generation of the feature vectors, from the identi­fication of the scope of a markable. For the En­glish sentence "Any details or speculation on who specifically, we don't know that at this point.", we first detect the heads of possible antecedents, for example "details". However, the decision on the scope of the markable, i.e. the decision between "details" or "Any details or speculation on who specifically" is made in the postprocessing phase.

One major task of the language modules is the check for cyclic dependencies. Our system re­lies on the assumption that cyclic dependencies do not occur, which is a standard assumption in de­pendency parsing (Kiibler et al., 2009). However, since some of the data sets in the multilingual task contained cycles, we integrated a module in the preprocessing step that takes care of such cycles.

After the identification of the heads of mark­ables, the actual feature extraction is performed. The features that were used for training a classifier (see Table 1) were selected from the feature pool presented by Rahman and Ng (2009). Note that not all features could be used for all languages. We extracted all the features in Table 1 if the cor­responding type of annotation was available; oth­erwise, a null value was assigned.

A good example for the latter concerns the gen­der information represented by feature 7 (for pos­sible feature values cf. Table 1). Let us consider the following two entries - the first from the Ger­man data set and the second from English:

1. Regierung Regierung Regierung NN NN cas=d|num=sg|gend=fem cas=d|num=sg|gend=fem 31 31 PN PN . . .

2. law _ law NN NN NN NN 2 2 PMOD PMOD . . .

Extracting the value from entry 1, where gend=fem, is straightforward; the value being F. However, there is no gender information provided in the English data (entry 2). As a result, the value for feature 7 is U for the closed task.

Based on the features extracted with the feature extractors described above, we trained TiMBL. Then we performed a non-exhaustive parameter

C if both m. are pron. and either same pron. or diff. w.r.t. case; NA if at least one is not pron.; else I C if the mentions agree in number; I if not; NA if the number for one or both is unknown C if both m. are pron. I if neither C if both m. are proper nouns; I if neither; else NA C if the m. have same sem. class; I if not; NA if the sem. class for one or both m. is unknown sentence distance between the mentions concat. concat. concat. concat. concat. concat.

The markable extractor makes use of the depen­dency relation labels. Each syntactic head together with all its dependents is identified as a separate markable. This approach is very sensitive to incor­rect annotations and to dependency cycles in the data set. It is also sensitive to differences between the syntactic annotation and markables. In the Dutch data, for example, markables for named en­tities (NE) often exclude the determiner, a nominal dependent in the dependency annotation. Thus, the markable extractor suggests the whole phrase as a markable, rather than just the NE.

During the development phase, we determined experimentally that the recognition of markables is one of the most important steps in order to achieve high accuracy in coreference resolution: We conducted an ablation study on the training data set. We used the train data as training set and the devel data as testing set and investigated three different settings:

1. Gold standard setting: Uses gold markable annotations as well as gold linguistic anno­tations (upper bound).

2. Gold linguistic setting: Uses automatically determined markables and gold linguistic an­notations.

3. Regular setting: Uses automatically deter­mined markables and automatic linguistic in­formation.

Note that we did not include all six languages: we excluded Italian and Dutch because there is no gold-standard linguistic annotation provided. The results of the experiment are shown in Table 2. From those results, we can conclude that the figures in Setting 2 and 3 are very similar. This means that the deterioration from gold to automat­ically annotated linguistic information is barely visible in the coreference results. This is a great advantage, since gold-standard data has always proved to be very expensive and difficult or im­possible to obtain. The information that proved to be extremely important for the performance of the system is the one providing the boundaries of the markables. As shown in Table 2, the latter leads to an improvement of about 20%, which is observ­able in the difference in the figures of Setting 1 and 2. The results for the different languages show that it is more important to improve markable de­tection than the linguistic information.

In Section 2.1, we described that we decided to separate the task of finding heads of markables from the identification of the scope of a markable. Thus, in the postprocessing step, we perform the latter (by the Markables Extractor module) as well as reformat the data for evaluation.

Another very important step during postpro­cessing is the selection of possible antecedents. In cases where more than one mention pair is classi­fied as coreferent, only the pair with highest con­fidence by TiMBL is selected. Since nouns can be discourse-new, they do not necessarily have a coreferent antecedent; pronouns however, require an antecedent. Thus, in cases where all possible antecedents for a given pronoun are classified as not coreferent, we select the closest subject as an­tecedent; or if this heuristic is not successful, the antecedent that has been classified as not corefer­ent with the lowest confidence score (i.e. the high­est distance) by TiMBL.