<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="1152"/><title>Simple Coreference Resolution with Rich Syntactic and Semantic Features</title><author surname="Haghighi" givenname="Aria"><org  name="UC Berkeley" country="USA" city="Berkeley"/></author><author surname="Klein" givenname="Dan"><org  name="UC Berkeley" country="USA" city="Berkeley"/></author></firstpageheader><frontmatter><p><b>Simple Coreference Resolution with Rich Syntactic and Semantic Features</b></p><p><b>Aria Haghighi and Dan Klein</b></p><p>Computer Science Division UC Berkeley</p><p>{aria42,  klein}@cs.berkeley.edu</p></frontmatter><abstract>Coreference systems are driven by syntactic, se­mantic, and discourse constraints. We present a simple approach which completely modularizes these three aspects. In contrast to much current work, which focuses on learning and on the dis­course component, our system is deterministic and is driven entirely by syntactic and semantic com­patibility as learned from a large, unlabeled corpus. Despite its simplicity and discourse naivete, our system substantially outperforms all unsupervised systems and most supervised ones. Primary con­tributions include (1) the presentation of a simple-to-reproduce, high-performing baseline and (2) the demonstration that most remaining errors can be at­tributed to syntactic and semantic factors external to the coreference phenomenon (and perhaps best addressed by non-coreference systems). </abstract></header><body><section number="1" title="Introduction"><p>The resolution of entity reference is influenced by a variety of constraints. Syntactic constraints like the binding theory, the i-within-i filter, and appos-itive constructions restrict reference by configura­tion. Semantic constraints like selectional compat­ibility (e.g. a <i>spokesperson </i>can <i>announce </i>things) and subsumption (e.g. <i>Microsoft </i>is a <i>company) </i>rule out many possible referents. Finally, dis­course phenomena such as salience and centering theory are assumed to heavily influence reference preferences. As these varied factors have given rise to a multitude of weak features, recent work has focused on how best to learn to combine them using models over reference structures (Culotta et al., 2007; Denis and Baldridge, 2007; Klenner and Ailloud, 2007).</p><p>In this work, we break from the standard view. Instead, we consider a vastly more modular system in which coreference is predicted from a determin­istic function of a few rich features. In particu­lar, we assume a three-step process. First, a self-contained syntactic module carefully represents syntactic structures using an augmented parser and extracts syntactic paths from mentions to potential antecedents. Some of these paths can be ruled in or out by deterministic but conservative syntactic constraints. Importantly, the bulk of the work in the syntactic module is in making sure the parses are correctly constructed and used, and this mod­ule's most important training data is a treebank. Second, a self-contained semantic module evalu­ates the semantic compatibility of headwords and individual names. These decisions are made from compatibility lists extracted from unlabeled data sources such as newswire and web data. Finally, of the antecedents which remain after rich syntac­tic and semantic filtering, reference is chosen to minimize tree distance.</p><p>This procedure is trivial where most systems are rich, and so does not need any supervised corefer­ence data. However, it is rich in important ways which we argue are marginalized in recent coref­erence work. Interestingly, error analysis from our final system shows that its failures are far more often due to syntactic failures (e.g. parsing mis­takes) and semantic failures (e.g. missing knowl­edge) than failure to model discourse phenomena or appropriately weigh conflicting evidence.</p><p>One contribution of this paper is the exploration of strong modularity, including the result that our system beats all unsupervised systems and ap­proaches the state of the art in supervised ones. Another contribution is the error analysis result that, even with substantial syntactic and semantic richness, the path to greatest improvement appears to be to further improve the syntactic and semantic modules. Finally, we offer our approach as a very strong, yet easy to implement, baseline. We make no claim that learning to reconcile disparate fea­tures in a joint model offers no benefit, only that it must not be pursued to the exclusion of rich, non-reference analysis.</p></section><section number="2" title="Coreference Resolution"><p>In coreference resolution, we are given a docu­ment which consists of a set of mentions; each <i>Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, </i>pages 1152-1161, Singapore, 6-7 August 2009. ©2009 ACL and AFNLP mention is a phrase in the document (typically an NP) and we are asked to cluster mentions ac­cording to the underlying referent entity.<page local="2" global="1153"/> There are three basic mention types: proper <i>(Barack Obama), </i>nominal <i>(president), </i>and pronominal <i>(he).<footnote anchor="1"/> </i>For comparison to previous work, we eval­uate in the setting where mention boundaries are given at test time; however our system can easily annotate reference on all noun phrase nodes in a parse tree (see Section 3.1.1).</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1152</doubt><subsection number="2.1" title="Data Sets"><p>In this work we use the following data sets: <b>Development:   </b>(see Section 3)</p><p><b>• ACE2004-ROTH-DEV </b>: Dev set split of the ACE 2004 training set utilized in Bengston and Roth (2008). The ACE data also annotates pre-nominal mentions which we map onto nominals. 68 documents and 4,536 mentions.</p><p><b>Testing:   </b>(see Section 4)</p><doubt alpha="65.2" length="158" tooSmall="False" monospace="0.0">• ACE2004-CULOTTA-TEST:Test set split of the ACE 2004 training set utilized in Culotta et al. (2007) and Bengston and Roth (2008). Consists of 107 documents.2</doubt><p><b>• ACE2004-N WIRE </b>: ACE 2004 News wire set to compare against Poon and Domingos (2008). Consists of 128 documents and 11,413 men­tions; intersects with the other ACE data sets.</p><p><b>• MUC-6-TEST: </b>MUC6 formal evaluation set consisting of 30 documents and 2,068 men­tions.</p><p><b>Unlabeled:   </b>(see Section 3.2)</p><p><b>• BLIPP: </b>1.8 million sentences of news wire parsed with the Charniak (2000) parser. No labeled coreference data; used for mining se­mantic information.</p><p><b>• WIKI: </b>25k articles of English Wikipedia ab­stracts parsed by the Klein and Manning (2003) parser.<footnote anchor="3"/> No labeled coreference data; used for mining semantic information.</p><footnote label="1">Other mention types exist and are annotated (such as pre-nominal), which are treated as nominals in this work.</footnote><footnote label="2">The evaluation set was not made available to non-participants.</footnote><footnote label="3">Wikipedia abstracts consist of roughly the first paragraph of the corresponding article</footnote></subsection><subsection number="2.2" title="Evaluation"><p>We will present evaluations on multiple corefer­ence resolution metrics, as no single one is clearly superior:</p><p>• Pairwise Fl: precision, recall, and Fl over all pairs of mentions in the same entity clus­ter. Note that this over-penalizes the merger or separation of clusters quadratically in the size of the cluster.</p><p><i>• b<footnote anchor="3"/> </i>(Amit and Baldwin, 1998): For each men­tion, form the intersection between the pre­dicted cluster and the true cluster for that mention. The precision is the ratio of the in­tersection and the true cluster sizes and recall the ratio of the intersection to the predicted sizes; Fl is given by the harmonic mean over precision and recall from all mentions.</p><p>• MUC (Vilain et al., 1995): For each true clus­ter, compute the number of predicted clusters which need to be merged to cover the true cluster. Divide this quantity by true cluster size minus one. Recall is given by the same procedure with predicated and true clusters reversed.<footnote anchor="4"/></p><p>• CEAF (Luo, 2005): For a similarity function between predicted and true clusters, CEAF scores the best match between true and pre­dicted clusters using this function. We use the <i>4&gt;s</i><i> </i>similarity function from Luo (2005).</p></subsection></section><section number="3" title="System Description"><p>In this section we develop our system and re­port developmental results on ACE2004-ROTH-dev (see Section 2.1); we report pairwise Fl fig­ures here, but report on many more evaluation metrics in Section 4. At a high level, our system resembles a pairwise coreference model (Soon et al., 1999; Ng and Cardie, 2002; Bengston and Roth, 2008); for each mention <i>rm, </i>we select ei­ther a single-best antecedent amongst the previ­ous mentions mi,..., <i>rrii-i, </i>or the NULL men­tion to indicate the underlying entity has not yet been evoked. Mentions are linearly ordered ac­cording to the position of the mention head with ties being broken by the larger node coming first.</p><footnote label="4">The MUC measure is problematic when the system pre­dicts many more clusters than actually exist (Luo, 2005; Finkel and Manning, 2008); also, singleton clusters do not contribute to evaluation.</footnote><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1153</doubt><page local="3" global="1154"/><p>While much research (Ng and Cardie, 2002; Cu­lotta et al., 2007; Haghighi and Klein, 2007; Poon and Domingos, 2008; Finkel and Manning, 2008) has explored how to reconcile pairwise decisions to form coherent clusters, we simply take the tran­sitive closure of our pairwise decision (as in Ng and Cardie (2002) and Bengston and Roth (2008)) which can and does cause system errors.</p><p>In contrast to most recent research, our pair-wise decisions are not made with a learned model which outputs a probability or confidence, but in­stead for each mention <i>rm, </i>we select an antecedent amongst mi,..., <i>m%-\ </i>or the null mention as follows:</p><p><b>• Syntactic Constraint: </b>Based on syntac­tic configurations, either force or disallow coreference between the mention and an an­tecedent. Propagate this constraint (see Fig­ure 4).</p><p><b>• Semantic/Syntactic Filter: </b>Filter the re­maining possible antecedents based upon compatibility with the mention (see Fig­ure 2).</p><p><b>• Selection: </b>Select the 'closest' mention from the set of remaining possible antecedents (see Figure 1) or the null antecedent if empty.</p><p>Initially, there is no syntactic constraint (im­proved in Section 3.1.3), the antecedent com­patibility filter allows proper and nominal men­tions to corefer only with mentions that have the same head (improved in Section 3.2), and pro­nouns have no compatibility constraints (improved in Section 3.1.2). Mention heads are determined by parsing the given mention span with the Stan­ford parser (Klein and Manning, 2003) and us­ing the Collins head rules (Collins, 1999); Poon and Domingos (2008) showed that using syntactic heads strongly outperformed a simple rightmost headword rule. The mention type is determined by the head POS tag: proper if the head tag is nnp or nnps, pronoun if the head tag is prp, prp$, wp, or wp$, and nominal otherwise.</p><p>For the selection phase, we order mentions mi,...,mj_i according to the position of the head word and select the closest mention that re­mains after constraint and filtering are applied. This choice reflects the intuition of Grosz et al. (1995) that speakers only use pronominal men­tions when there are not intervening compatible</p><doubt alpha="59.1" length="22" tooSmall="False" monospace="0.0">NP#1 VP NP PP VBD NP#3</doubt><doubt alpha="58.8" length="68" tooSmall="False" monospace="0.0">NNP      IN     NP#2announcedNP#1 NPNintendo    ofNNP PRP$     II NN</doubt><p><i>America its      new console</i></p><p>Figure 1 : Example sentence where closest tree dis­tance between mentions outperforms raw distance. For clarity, each mention NP is labeled with the underlying entity id.</p><p>mentions. This system yields a rather low 48.9 pairwise Fl (see base-flat in Table 2). There are many, primarily recall, errors made choos­ing antecedents for all mention types which we will address by adding syntactic and semantic con­straints.</p><subsection number="3.1" title="Adding Syntactic Information"><p>In this section, we enrich the syntactic represen­tation and information in our system to improve results.</p><subsubsection number="3.1.1" title="Syntactic Salience"><p>We first focus on fixing the pronoun antecedent choices. A common error arose from the use of mention head distance as a poor proxy for dis­course salience. For instance consider the exam­ple in Figure 1, the mention <i>America </i>is closest to <i>its </i>in flat mention distance, but syntactically <i>Nintendo of America </i>holds a more prominent syn­tactic position relative to the pronoun which, as Hobbs (1977) argues, is key to discourse salience.</p><p><b>Mapping Mentions to Parse Nodes: </b>In order to use the syntactic position of mentions to determine anaphoricity, we must associate each mention in the document with a parse tree node. We parse all document sentences with the Stanford parser, and then for each evaluation mention, we find the largest-span NP which has the previously deter­mined mention head as its head.<footnote anchor="5"/> Often, this re­sults in a different, typically larger, mention span than annotated in the data.</p><p>Now that each mention is situated in a parse tree, we utilize the length of the shortest tree path between mentions as our notion of distance. In<page local="4" global="1155"/></p><footnote label="5">If there is no NP headed by a given mention head, we add an NP over just that word.</footnote><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1154</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">s</doubt><doubt alpha="54.5" length="11" tooSmall="False" monospace="0.0">np-0rg#1 vp</doubt><p><i>regard     </i>np pp in pp s</p><p><i>the site   </i>in     np#2     <i>because   </i>to   np#1   <i>it is sacred as   a shrine to </i>prp</p><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">them</doubt><p>Figure 2: Example of a coreference decision fixed by agreement constraints (see Section 3.1.2). The pronoun <i>them </i>is closest to <i>the site </i>mention, but has an incompatible number feature with it. The clos­est (in tree distance, see Section 3.1.1) compatible mention is <i>The Israelis, </i>which is correct particular, this fixes examples such as those in Figure 1 where the true antecedent has many em­bedded mentions between itself and the pronoun. This change by itself yields 51.7 pairwise Fl (see base-tree in Table 2), which is small overall, but reduces pairwise pronoun antecedent selection er­ror from 51.3% to 42.5%.</p></subsubsection><subsubsection number="3.1.2" title="Agreement Constraints"><p>We now refine our compatibility filtering to in­corporate simple agreement constraints between coreferent mentions. Since we currently allow proper and nominal mentions to corefer only with matching head mentions, agreement is only a con­cern for pronouns. Traditional linguistic theory stipulates that coreferent mentions must agree in number, person, gender, and entity type (e.g. an-imacy). Here, we implement person, number and entity type agreement.<footnote anchor="6"/></p><p>A number feature is assigned to each mention deterministically based on the head and its POS tag. For entity type, we use NER labels. Ideally, we would like to have information about the en­tity type of each referential NP, however this in­formation is not easily obtainable. Instead, we opt to utilize the Stanford NER tagger (Finkel et al., 2005) over the sentences in a document and anno­tate each NP with the NER label assigned to that mention head. For each mention, when its NP is assigned an NER label we allow it to only be com­patible with that NER label.<footnote anchor="7"/> For pronouns, we deterministically assign a set of compatible NER values (e.g. personal pronouns can only be a per-</p><footnote label="6">Gender agreement, while important for general corefer­ence resolution, did not contribute to the errors in our largely newswire data sets.</footnote><footnote label="7">Or allow it to be compatible with all ner labels if the ner tagger doesn't predict a label.</footnote><p>Table 1 : Most common recall (missed-link) errors amongst non-pronoun mention heads on our de­velopment set. Detecting compatibility requires semantic knowledge which we obtain from a large corpus (see Section 3.2).</p><doubt alpha="60.0" length="55" tooSmall="False" monospace="0.0">np , np-appos#2is undersellingnnp    ,   np#1    jj nns</doubt><p><i>Giîano        </i>prp    <i>top brand </i><b>I</b></p><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">its</doubt><p>Figure 4: Example of interaction between the ap-positive and i-within-i constraint. The i-within-i constraint disallows coreference between parent and child NPs unless the child is an appositive. Hashed numbers indicate ground truth but are not in the actual trees.</p><p>son, but <i>its </i>can be an organization or loca­tion). Since the NER tagger typically does not label non-proper NP heads, we have no NER com­patibility information for nominals.</p><p>We incorporate agreement constraints by filter­ing the set of possible antecedents to those which have compatible number and NER types with the target mention. This yields 53.4 pairwise Fl, and reduces pronoun antecedent errors to 42.5% from 34.4%. An example of the type of error fixed by these agreement constraints is given by Figure 2.</p></subsubsection><subsubsection number="3.1.3" title="Syntactic Configuration Constraints"><p>Our system has so far focused only on improving pronoun anaphora resolution. However, a plurality of the errors made by our system are amongst non-pronominal mentions.<footnote anchor="8"/> We take the approach that in order to align a non-pronominal mention to an antecedent without an identical head, we require evidence that the mentions are compatible.</p><p>Judging compatibility of mentions generally re­quires semantic knowledge, to which we return later.   However, some syntactic configurations<page local="5" global="1156"/></p><footnote label="8">There are over twice as many nominal mentions in our development data as pronouns.</footnote><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1155</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>gore</p></td><td class="cell"><p>president</p></td><td class="cell"><p>florida</p></td><td class="cell"><p>state</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>bush</p></td><td class="cell"><p>governor</p></td><td class="cell"><p>lebanese</p></td><td class="cell"><p>territory</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>nation</p></td><td class="cell"><p>people</p></td><td class="cell"><p>arafat</p></td><td class="cell"><p>leader</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>inc.</p></td><td class="cell"><p>company</p></td><td class="cell"><p>aol</p></td><td class="cell"><p>company</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>nation</p></td><td class="cell"><p>country</p></td><td class="cell"><p>assad</p></td><td class="cell"><p>president</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">NP#1</doubt><doubt alpha="44.4" length="9" tooSmall="False" monospace="0.0">NP , NP#1</doubt><p>NN#1 NNP NNP , <i>subject ofthe [exhibition]?, painter   Pablo Picasso</i></p><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">(a)</doubt><doubt alpha="66.7" length="9" tooSmall="False" monospace="0.0">NP-PERS#1</doubt><doubt alpha="52.9" length="17" tooSmall="False" monospace="0.0">NP ,^^j^APros#i^^</doubt><p><i>subject of the [exhibition]?</i></p><p><i>painter        Pablo Picasso</i></p><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">(b)</doubt><p>Figure 3: NP structure annotation: In (a) we have the raw parse from the Klein and Manning (2003) parser with the mentions annotated by entity. In (b), we demonstrate the annotation we have added. NER labels are added to all NP according to the NER label given to the head (see Section 3.1.1). Appositive NPs are also annotated. Hashes indicate forced coreferent nodes guarantee coreference. The one exploited most in coreference work (Soon et al., 1999; Ng and Cardie, 2002; Luo et al., 2004; Culotta et al., 2007; Poon and Domingos, 2008; Bengston and Roth, 2008) is the appositive construction. Here, we rep­resent apposition as a syntactic feature of an NP indicating that it is coreferent with its parent NP (e.g. it is an exception to the i-within-i constraint that parent and child NPs <i>cannot </i>be coreferent). We deterministically mark a node as NP-appos (see Figure 3) when it is the third child in of a par­ent NP whose expansion begins with (NP , NP), and there is not a conjunction in the expansion (to avoid marking elements in a list as appositive).</p><p><b>Role Appositives: </b>During development, we dis­covered many errors which involved a variant of appositives which we call 'role appositives' (see <i>painter </i>in Figure 3), where an NP modifying the head NP describes the role of that entity (typi­cally a person entity). There are several challenges to correctly labeling these role NPs as being ap­positives. First, the NPs produced by Treebank parsers are flat and do not have the required inter­nal structure (see Figure 3(a)). While fully solving this problem is difficult, we can heuristically fix many instances of the problem by placing an NP around maximum length sequences of NNP tags or NN (and JJ) tags within an NP; note that this will fail for many constructions such as <i>U.S. Pres­ident Barack Obama, </i>which is analyzed as a flat sequence of proper nouns. Once this internal NP structure has been added, whether the NP immedi­ately to the left of the head NP is an appositive de­pends on the entity type. For instance, <i>Rabbi Ashi </i>is an apposition but <i>Iranian army </i>is not. Again, a full solution would require its own model, here we mark as appositions any NPs immediately to the</p><p>left of a head child NP where the head child NP is identified as a person by the NER tagger.<footnote anchor="9"/></p><p>We incorporate NP appositive annotation as a constraint during filtering. Any mention which corresponds to an appositive node has its set of possible antecedents limited to its parent. Along with the appositive constraint, we implement the i-within-i constraint that any non-appositive NP cannot be be coreferent with its parent; this con­straint is then propagated to any node its parent is forced to agree with. The order in which these constraints are applied is important, as illustrated by the example in Figure 4: First the list of pos­sible antecedents for the appositive NP is con­strained to only its parent. Now that all apposi­tives have been constrained, we apply the i-within-i constraint, which prevents <i>its </i>from having the NP headed by <i>brand </i>in the set of possible antecedents, and by propagation, also removes the NP headed by <i>Gitano. </i>This leaves the NP <i>Wal-Mart </i>as the closest compatible mention.</p><p>Adding these syntactic constraints to our system yields 55.4 Fl, a fairly substantial improvement, but many recall errors remain between mentions with differing heads. Resolving such cases will require external semantic information, which we will automatically acquire (see Section 3.2).</p><p><b>Predicate Nominatives: </b>Another syntactic con­straint exploited in Poon and Domingos (2008) is the predicate nominative construction, where the object of a copular verb (forms of the verb <i>be) </i>is constrained to corefer with its subject (e.g. <i>Mi­crosoft is a company in Redmond). </i>While much less frequent than appositive configurations (there are only 17 predicate nominatives in our devel-</p><footnote label="9">Arguably, we could also consider right modifying NPs (e.g., [Microsoft [Companyfi ]i) to be role appositive, but we do not do so here.</footnote><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1156</doubt><p>NP-APPOS«</p><p>NN NNP NNP<page local="6" global="1157"/></p><p>Figure 5: Example paths extracted via semantic compatibility mining (see Section 3.2) along with exam­ple instantiations. In both examples the left child NP is coreferent with the rightmost NP. Each category in the interior of the tree path is annotated with the head word as well as its subcategorization. The examples given here collapse multiple instances of extracted paths.</p><p>opment set), predicate nominatives are another highly reliable coreference pattern which we will leverage in Section 3.2 to mine semantic knowl­edge. As with appositives, we annotate object predicate-nominative NPs and constrain corefer­ence as before. This yields a minor improvement to 55.5 Fl.</p></subsubsection></subsection><subsection number="3.2" title="Semantic Knowledge"><p>While appositives and related syntactic construc­tions can resolve some cases of non-pronominal reference, most cases require semantic knowledge about the various entities as well as the verbs used in conjunction with those entities to disambiguate references (Kehler et al., 2008).</p><p>However, given a semantically compatible men­tion head pair, say <i>AOL </i>and <i>company, </i>one might expect to observe a reliable appositive or predicative-nominative construction involving these mentions somewhere in a large corpus. In fact, the Wikipedia page for <i>AOL<footnote anchor="10"/> </i>has a predicate-nominative construction which supports the compatibility of this head pair: <i>AOL LLC (for­merly America Online) is an American global In­ternet services and media company operated by Time Warner.</i></p><p>In order to harvest compatible head pairs, we utilize our BLIPP and WIKI data sets (see Sec­tion 2), and for each noun (proper or common) and pronoun, we assign a maximal NP mention node for each nominal head as in Section 3.1.1 ; we then annotate appositive and predicate-nominative NPs as in Section 3.1.3. For any NP which is annotated as an appositive or predicate-nominative, we ex­tract the head pair of that node and its constrained antecedent.</p><footnote label="10">http://en.wikipedia.org/wiki/AOL</footnote><p>The resulting set of compatible head words, while large, covers a little more than half of the examples given in Table 1. The problem is that these highly-reliable syntactic configurations are too sparse and cannot capture all the entity infor­mation present. For instance, the first sentence of Wikipedia abstract for <i>AI Gore </i>is:</p><p><i>Albert Arnold "AI" Gore, Jr. is an American environmental activist who served as the 45th Vice President of the United States from 1993 to 2001 under President Bill Clinton.</i></p><p>The required lexical pattern <i>X who served as Y </i>is a general appositive-like pattern that almost surely indicates coreference. Rather than opt to manu­ally create a set of these coreference patterns as in Hearst (1992), we instead opt to automatically ex­tract these patterns from large corpora as in Snow et al. (2004) and Phillips and Riloff (2007). We take a simple bootstrapping technique: given a set of mention pairs extracted from appositives and predicate-nominative configurations, we ex­tract counts over tree fragments between nodes which have occurred in this set of head pairs (see Figure 5); the tree fragments are formed by an­notating the internal nodes in the tree path with the head word and POS along with the subcatego­rization. We limit the paths extracted in this way in several ways: paths are only allowed to go be­tween adjacent sentences and have a length of at most 10. We then filter the set of paths to those which occur more than a hundred times and with at least 10 distinct seed head word pairs.</p><p>The vast majority of the extracted fragments are variants of traditional appositives and predicate-nominatives with some of the structure of the NPs<page local="7" global="1158"/></p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1157</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Path</p></td><td class="cell"><p>Example</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NP</p><p>NP-NNP PRN-NNP NP</p><p>NP-president   CC NP-NNP</p></td><td class="cell"><p><i>America Online Inc. (AOL) [President and C.E.O] Bill Gates</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><doubt alpha="62.5" length="16" tooSmall="False" monospace="0.0">ACE2004-ROTH-DEV</doubt><p><b>ACE2004-CULOTTA-TEST</b> <b>MUC6-TEST</b></p><doubt alpha="61.5" length="13" tooSmall="False" monospace="0.0">ACE2004-NWIRE</doubt><p>Table 2: Experimental Results (See Section 4): When comparisons between systems are presented, the largest result is bolded. The CEAF measure has equal values for precision, recall, and Fl.</p><p>specified. However there are some tree fragments which correspond to the novel coreference pat­terns (see Figure 5) of parenthetical alias as well as conjunctions of roles in NPs.</p><p>We apply our extracted tree fragments to our blipp and wiki data sets and extract a set of com­patible word pairs which match these fragments; these words pairs will be used to relax the seman­tic compatibility filter (see the start of the section); mentions are compatible with prior mentions with the same head or with a semantically compatible head word. This yields 58.5 pairwise Fl (see sem-compat in Table 2) as well as similar improve­ments across other metrics.</p><p>By and large the word pairs extracted in this way are correct (in particular we now have cov­erage for over two-thirds of the head pair recall errors from Table 1.) There are however word-pairs which introduce errors. In particular city-state constructions (e.g. <i>Los Angeles, California) </i>appears to be an appositive and incorrectly allows our system to have <i>angeles </i>as an antecedent for <i>California. </i>Another common error is that the % symbol is made compatible with a wide variety of common nouns in the financial domain.</p></subsection></section><section number="4" title="Experimental Results"><p>We present formal experimental results here (see Table 2). We first evaluate our model on the ace2004-culotta-test dataset used in the state-of-the-art systems from Culotta et al. (2007) and Bengston and Roth (2008). Both of these systems were supervised systems discrimi-natively trained to maximize <i>b<footnote anchor="3"/> </i>and used features from many different structured resources includ­ing WordNet, as well as domain-specific features (Culotta et al., 2007). Our best <i>b<footnote anchor="3"/> </i>result of 79.0 is broadly in the range of these results. We should note that in our work we use neither the gold men­tion types (we do not model pre-nominals sepa­rately) nor do we use the gold NER tags which Bengston and Roth (2008) does. Across metrics, the syntactic constraints and semantic compatibil­ity components contribute most to the overall final result.</p><p>On the muc6-test dataset, our system outper-<page local="8" global="1159"/></p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1158</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>MUC</p></td><td class="cell"><p></p></td><td class="cell"><p>fc<footnote anchor="3"/></p></td><td class="cell"><p></p></td><td class="cell"><p>Pairwise</p></td><td class="cell"><p>CEAF</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>System</p></td><td class="cell"><p>P         R Fl</p></td><td class="cell"><p>P</p></td><td class="cell"><p>R</p></td><td class="cell"><p>Fl</p></td><td class="cell"><p>P         R Fl</p></td><td class="cell"><p>P        R Fl</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>BASIC-FLAT</p></td><td class="cell"><p>73.5</p></td><td class="cell"><p>66.8</p></td><td class="cell"><p>70.0</p></td><td class="cell"><p>80.6</p></td><td class="cell"><p>68.6</p></td><td class="cell"><p>74.1</p></td><td class="cell"><p>63.6</p></td><td class="cell"><p>39.7</p></td><td class="cell"><p>48.9</p></td><td class="cell"><p>68.4</p></td><td class="cell"><p>68.4</p></td><td class="cell"><p>68.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>BASIC-TREE</p></td><td class="cell"><p>75.8</p></td><td class="cell"><p>68.9</p></td><td class="cell"><p>72.2</p></td><td class="cell"><p>81.9</p></td><td class="cell"><p>69.9</p></td><td class="cell"><p>75.4</p></td><td class="cell"><p>65.6</p></td><td class="cell"><p>42.7</p></td><td class="cell"><p>51.7</p></td><td class="cell"><p>69.8</p></td><td class="cell"><p>69.8</p></td><td class="cell"><p>69.8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>+SYN-COMPAT</p></td><td class="cell"><p>77.8</p></td><td class="cell"><p>68.5</p></td><td class="cell"><p>72.9</p></td><td class="cell"><p>84.1</p></td><td class="cell"><p>69.7</p></td><td class="cell"><p>76.2</p></td><td class="cell"><p>71.0</p></td><td class="cell"><p>43.1</p></td><td class="cell"><p>53.4</p></td><td class="cell"><p>69.8</p></td><td class="cell"><p>69.8</p></td><td class="cell"><p>69.8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>+SYN-CONSTR</p></td><td class="cell"><p>78.3</p></td><td class="cell"><p>70.5</p></td><td class="cell"><p>74.2</p></td><td class="cell"><p>84.0</p></td><td class="cell"><p>71.0</p></td><td class="cell"><p>76.9</p></td><td class="cell"><p>71.3</p></td><td class="cell"><p>45.4</p></td><td class="cell"><p>55.5</p></td><td class="cell"><p>70.8</p></td><td class="cell"><p>70.8</p></td><td class="cell"><p>70.8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>+SEM-COMPAT</p></td><td class="cell"><p>77.9</p></td><td class="cell"><p>74.1</p></td><td class="cell"><p>75.9</p></td><td class="cell"><p>81.8</p></td><td class="cell"><p>74.3</p></td><td class="cell"><p>77.9</p></td><td class="cell"><p>68.2</p></td><td class="cell"><p>51.2</p></td><td class="cell"><p>58.5</p></td><td class="cell"><p>72.5</p></td><td class="cell"><p>72.5</p></td><td class="cell"><p>72.5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>BASIC-FLAT   68.6   60.9 64.5 BASIC-TREE   71.2   63.2 67.0 +SYN-COMPAT   74.6   65.2 69.6 +SYN-CONSTR   74.3   66.4 70.2 +SEM-COMPAT   74.8   <b>77.7 79.6</b></p></td><td class="cell"><p>80.3 68.0 73.6 81.6 69.3 75.0 84.2 70.3 76.6 83.6 71.0 76.8 79.6   <b>78.5 </b>79.0</p></td><td class="cell"><p>57.1 30.5 39.8 60.1 34.5 43.9 66.7   37.2 47.8</p><p>66.4 38.0 48.3</p><p><b>57.5 </b>57.6 57.5</p></td><td class="cell"><p>66.5 66.5 66.5 67.9   67.9 67.9</p><p>69.2 69.2 69.2</p><p>69.6 69.6 69.6</p><p><b>73.3 73.3 73.3</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Supervised Results</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Culotta et al. (2007) Bengston and Roth (2008)   <b>82.7   </b>69.9 75.8</p></td><td class="cell"><p>86.7 73.2 79.3 <b>88.3   </b>74.5 <b>80.8</b></p></td><td class="cell"><p>55.4   <b>63.7 59.2</b></p></td><td class="cell"><p>-</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>+SEM-COMPAT</p></td><td class="cell"><p><b>87.2</b></p></td><td class="cell"><p><b>77.3</b></p></td><td class="cell"><p><b>81.9   </b>84.7   <b>67.3 75.0</b></p></td><td class="cell"><p><b>80.5</b></p></td><td class="cell"><p><b>57.8</b></p></td><td class="cell"><p><b>67.3   </b>72.0   72.0 72.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Unsupervised Results</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Poon and Domingos (2008)</p></td><td class="cell"><p>83.0</p></td><td class="cell"><p>75.8</p></td><td class="cell"><p>79.2 -</p></td><td class="cell"><p>63.0</p></td><td class="cell"><p>57.0</p></td><td class="cell"><p>60.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Supervised Results</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Finkel and Manning (2008)</p></td><td class="cell"><p>89.7</p></td><td class="cell"><p>55.1</p></td><td class="cell"><p>68.3   <b>90.9   </b>49.7 64.3</p></td><td class="cell"><p>74.1</p></td><td class="cell"><p>37.1</p></td><td class="cell"><p>49.5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>+SEM-COMPAT</p></td><td class="cell"><p><b>77.0</b></p></td><td class="cell"><p><b>75.9</b></p></td><td class="cell"><p><b>76.5   </b>79.4   74.5 76.9</p></td><td class="cell"><p><b>66.9</b></p></td><td class="cell"><p><b>49.2</b></p></td><td class="cell"><p><b>56.7   </b>71.5   71.5 71.5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Unsupervised Results</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Poon and Domingos (2008)</p></td><td class="cell"><p>71.3</p></td><td class="cell"><p>70.5</p></td><td class="cell"><p>70.9 -</p></td><td class="cell"><p>62.6</p></td><td class="cell"><p>38.9</p></td><td class="cell"><p>48.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Table 3: Errors for each type of antecedent deci­sion made by the system. Each row is a mention type and the column the predicted mention type antecedent. The majority of errors are made in the NOMINAL category.</p><p>forms both Poon and Domingos (2008) (an un­supervised Markov Logic Network system which uses explicit constraints) and Finkel and Manning (2008) (a supervised system which uses ILP in­ference to reconcile the predictions of a pairwise classifier) on all comparable measures.<footnote anchor="11"/> Simi­larly, on the ACE2004-NWIRE dataset, we also out­perform the state-of-the-art unsupervised system of Poon and Domingos (2008).</p><p>Overall, we conclude that our system outper­forms state-of-the-art unsupervised systems<footnote anchor="12"/> and is in the range of the state-of-the art systems of Cu­lotta et al. (2007) and Bengston and Roth (2008).</p></section><section number="5" title="Error Analysis"><p>There are several general trends to the errors made by our system. Table 3 shows the number of pairwise errors made on MUC6-TEST dataset by mention type; note these errors are not equally weighted in the final evaluations because of the transitive closure taken at the end. The most er­rors are made on nominal mentions with pronouns coming in a distant second. In particular, we most frequently say a nominal is NULL when it has an antecedent; this is typically due to not having the necessary semantic knowledge to link a nominal to a prior expression.</p><p>In order to get a more thorough view of the cause of pairwise errors, we examined 20 random errors made in aligning each mention type to an antecedent. We categorized the errors as follows:</p><p>• Sem. Compat: Missing information about the compatibility of two words e.g. <i>pay </i>and <i>wage. </i>For pronouns, this is used to mean that</p><footnote label="11">Klenner and Ailloud (2007) took essentially the same ap­proach but did so on non-comparable data.</footnote><footnote label="12">Poon and Domingos (2008) outperformed Haghighi and Klein (2007). Unfortunately, we cannot compare against Ng (2008) since we do not have access to the version of the ACE data used in their evaluation.</footnote><p>we incorrectly aligned a pronoun to a men­tion with which it is not semantically com­patible (e.g. <i>he </i>aligned to <i>board).</i></p><p>• Syn. Compat: Error in assigning linguistic features of nouns for compatibility with pro­nouns (e.g. disallowing <i>they </i>to refer to <i>team).</i></p><p>• Head: Errors involving the assumption that mentions with the same head are always com­patible. Includes modifier and specificity er­rors such as allowing <i>Lebanon </i>and <i>Southern Lebanon </i>to corefer. This also includes errors of definiteness in nominals (e.g. <i>the people in the room </i>and <i>Chinese people). </i>Typically, these errors involve a combination of missing syntactic and semantic information.</p><p>• Internal NP: Errors involving lack of inter­nal NP structure to mark role appositives (see Section 3.1.3).</p><p>• Prag. / Disc: Errors where discourse salience or pragmatics are needed to disambiguate mention antecedents.</p><p>• Process Error: Errors which involved a tok-enization, parse, or NER error.</p><p>The result of this error analysis is given in Ta­ble 4; note that a single error may be attributed to more than one cause. Despite our efforts in Sec­tion 3 to add syntactic and semantic information to our system, the largest source of error is still a combination of missing semantic information or annotated syntactic structure rather than the lack of discourse or salience modeling.</p><p>Our error analysis suggests that in order to im­prove the state-of-the-art in coreference resolu­tion, future research should consider richer syntac­tic and semantic information than typically used in current systems.</p></section><section number="6" title="Conclusion"><p>Our approach is not intended as an argument against the more complex, discourse-focused ap­proaches that typify recent work. Instead, we note that rich syntactic and semantic processing vastly reduces the need to rely on discourse effects or ev­idence reconciliation for reference resolution. In­deed, we suspect that further improving the syn­tactic and semantic modules in our system may produce greater error reductions than any other<page local="9" global="1160"/></p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1159</doubt><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>proper</p></td><td class="cell"><p>21/451</p></td><td class="cell"><p>8/20</p></td><td class="cell"><p>72/288</p></td><td class="cell"><p>101/759</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>nominal</p></td><td class="cell"><p>16/150</p></td><td class="cell"><p>99/432</p></td><td class="cell"><p>158/351</p></td><td class="cell"><p>323/933</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>pronoun</p></td><td class="cell"><p>29/149</p></td><td class="cell"><p>60/128</p></td><td class="cell"><p>15/97 1/2</p></td><td class="cell"><p>105/376</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Table 4: Error analysis on ACE2004-CULOTTA-TEST data by mention type. The dominant errors are in either semantic or syntactic compatibility of mentions rather than discourse phenomena. See Section 5.</p><p>route forward. Of course, a system which is rich in all axes will find some advantage over any sim­plified approach.</p><p>Nonetheless, our coreference system, despite being relatively simple and having no tunable pa­rameters or complexity beyond the non-reference complexity of its component modules, manages to outperform state-of-the-art unsupervised coref­erence resolution and be broadly comparable to state-of-the-art supervised systems.</p><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Mention Type</p></td><td class="cell"><p>Sem. Compat</p></td><td class="cell"><p>Syn. Compat</p></td><td class="cell"><p>Head</p></td><td class="cell"><p>intenal np</p></td><td class="cell"><p>Prag / Disc.</p></td><td class="cell"><p>Process Error</p></td><td class="cell"><p>Other</p></td><td class="cell"><p>Comment</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>nominal</p></td><td class="cell"><p>7</p></td><td class="cell"><p>-</p></td><td class="cell"><p>5</p></td><td class="cell"><p>6</p></td><td class="cell"><p>2</p></td><td class="cell"><p>2</p></td><td class="cell"><p>1</p></td><td class="cell"><p>2 general appos. patterns</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>pronoun</p></td><td class="cell"><p>6</p></td><td class="cell"><p>3</p></td><td class="cell"><p>-</p></td><td class="cell"><p>6</p></td><td class="cell"><p>3</p></td><td class="cell"><p>3</p></td><td class="cell"><p>3</p></td><td class="cell"><p>2 cataphora</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>proper</p></td><td class="cell"><p>6</p></td><td class="cell"><p>-</p></td><td class="cell"><p>3</p></td><td class="cell"><p>4</p></td><td class="cell"><p>4</p></td><td class="cell"><p>4</p></td><td class="cell"><p>1</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><references><p>B. Amit and B. Baldwin. 1998. Algorithms for scoring coreference chains. In <i>MUC7.</i></p><p>Eric Bengston and Dan Roth. 2008. Understanding the value of features for corefernce resolution. In <i>Em­pirical Methods in Natural Language Processing.</i></p><p>E. Charniak. 2000. Maximum entropy inspired parser.</p><p>In <i>North American Chapter of the Association of Computational Linguistics (NAACL).</i></p><p>Mike Collins. 1999. Head-driven statistical models for natural language parsing.</p><p>A Culotta, M Wick, R Hall, and A McCallum. 2007. First-order probabilistic models for coreference res­olution. In <i>NAACL-HLT.</i></p><p>Pascal Denis and Jason Baldridge. 2007. Global, Joint Determination of Anaphoricity and Corefer­ence Resolution using Integer Programming. In <i>HLT-NAACL.</i></p><p>Jenny Finkel and Christopher Manning. 2008. Enforc­ing transitivity in coreference resolution. In <i>Associ­ation of Computational Linguists (ACL).</i></p><p>Jenny Finkel, Trond Grenager, and Christopher Man­ning. 2005. Incorporating non-local information into information extraction systems by gibbs sam­pling. In <i>ACL.</i></p><p>Barbara J. Grosz, Aravind K. Joshi, and Scott Wein­stein. 1995. Centering: A framework for modelling the local coherence of discourse.</p><p>Aria Haghighi and Dan Klein. 2007. Unsupervised coreference resolution in a nonparametric bayesian model. In <i>Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics. </i>As­sociation for Computational Linguistics.</p><p>Marti Hearst. 1992. Automatic acquisition of hy-ponyms from large text corpora. In <i>Conference on Natural Language Learning (COLING).</i></p><p>J. R. Hobbs. 1977. Resolving pronoun references. <i>Lingua.</i></p><p>Andrew Kehler, Laura Kertz, Hannah Rohde, and Jef­frey Elman. 2008. Coherence and coreference re­visited.</p><p>D. Klein and C. Manning. 2003. Accurate unlexical-ized parsing. In <i>Association of Computational Lin­guists (ACL).</i></p><p>Manfred Klenner and Etienne Ailloud. 2007. Op­timization in coreference resolution is not needed: A nearly-optimal algorithm with intensional con­straints. In <i>Recent Advances in Natural Language Processing.</i></p><p>Xiaoqiang Luo, Abe Ittycheriah, Hongyan Jing, Nanda Kambhatla, and Salim Roukos. 2004. A Mention-Synchronous Coreference Resolution Al­gorithm Based on the Bell Tree. In <i>Association of Computational Linguists.</i></p><p>X Luo. 2005. On coreference resolution performance metrics. In <i>Proceedings of the conference on Hu­man Language Technology and Empirical Methods in Natural Language Processing.</i></p><p>Vincent Ng and Claire Cardie. 2002. Improving Ma­chine Learning Approaches to Coreference Resolu­tion. <i>In Association of Computational Linguists.</i></p><p>Vincent Ng. 2008. Unsupervised models of corefer­ence resolution. In <i>EMNLP.</i></p><p>W. Phillips and E. Riloff. 2007. Exploiting role-identifying nouns and expressions for information extraction. <i>In Recent Advances in Natural Language Processing (RANLP).</i></p><p>Hoifung Poon and Pedro Domingos. 2008. Joint unsu­pervised coreference resolution with Markov Logic.</p><p>In <i>Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing.</i></p><p>R. Snow, D. Jurafsky, and A. Ng. 2004. Learning syn­tactic patterns for automatic hypernym discovery. In <i>Neural Information Processing Systems (NIPS).</i></p><p>W.H. Soon, H. T. Ng, and D. C. Y. Lim. 1999. A machine learning approach to coreference resolution of noun phrases.</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1160</doubt><page local="10" global="1161"/><p>Marc Vilain, John Burger, John Aberdeen, Dennis Con­nolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In <i>MUC-6.</i></p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1161</doubt></references></body></article>