<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="141"/><title>Nested Named Entity Recognition</title><author surname="Finkel" givenname="Jenny Rose"><org  name="Stanford University" country="USA" city="Stanford"/></author><author surname="Manning" givenname="Christopher D."><org  name="Stanford University" country="USA" city="Stanford"/></author></firstpageheader><frontmatter><p><b>Nested Named Entity Recognition</b></p><p><b>Jenny Rose Finkel and Christopher D. Manning</b></p><p>Computer Science Department Stanford University Stanford, CA 94305 {jrfinkel <b>I </b>manning}@cs.stanford.edu</p></frontmatter><abstract>Many named entities contain other named entities inside them. Despite this fact, the field of named entity recognition has al­most entirely ignored nested named en­tity recognition, but due to technological, rather than ideological reasons. In this pa­per, we present a new technique for rec­ognizing nested named entities, by using a discriminative constituency parser. To train the model, we transform each sen­tence into a tree, with constituents for each named entity (and no other syntactic struc­ture). We present results on both news­paper and biomedical corpora which con­tain nested named entities. In three out of four sets of experiments, our model outperforms a standard semi-CRF on the more traditional top-level entities. At the same time, we improve the overall F-score by up to 30% over the flat model, which is unable to recover any nested entities. </abstract></header><body><section number="1" title="Introduction"><p>Named entity recognition is the task of finding en­tities, such as people and organizations, in text. Frequently, entities are nested within each other, such as <i>Bank of China </i>and <i>University of Wash­ington, </i>both <i>organizations </i>with nested <i>locations. </i>Nested entities are also common in biomedical data, where different biological entities of inter­est are often composed of one another. In the GENIA corpus (Ohta et al., 2002), which is la­beled with entity types such as <i>protein </i>and <i>DNA, </i>roughly 17% of entities are embedded within an­other entity. In the AnCora corpus of Spanish and Catalan newspaper text (Marti et al, 2007), nearly half of the entities are embedded. However, work on named entity recognition (NER) has almost en­tirely ignored nested entities and instead chosen to focus on the outermost entities.</p><p>We believe this has largely been for practical, not ideological, reasons. Most corpus designers have chosen to skirt the issue entirely, and have annotated only the topmost entities. The widely used CoNLL (Sang and Meulder, 2003), MUC-6, and MUC-7 NER corpora, composed of American and British newswire, are all flatly annotated. The GENIA corpus contains nested entities, but the JNLPBA 2004 shared task (Collier et al, 2004), which utilized the corpus, removed all embedded entities for the evaluation. To our knowledge, the only shared task which has included nested enti­ties is the SemEval 2007 Task 9 (Marquez et al, 2007b), which used a subset of the AnCora corpus. However, in that task all entities corresponded to particular parts of speech or noun phrases in the provided syntactic structure, and no participant di­rectly addressed the nested nature of the data.</p><p>Another reason for the lack of focus on nested NER is technological. The NER task arose in the context of the MUC workshops, as small chunks which could be identified by finite state models or gazetteers. This then led to the widespread use of sequence models, first hidden Markov mod­els, then conditional Markov models (Borthwick, 1999), and, more recently, linear chain conditional random fields (CRFs) (Lafferty et al., 2001). All of these models suffer from an inability to model nested entities.</p><p>In this paper we present a novel solution to the problem of nested named entity recognition. Our model explicitly represents the nested structure, allowing entities to be influenced not just by the labels of the words surrounding them, as in a CRF, but also by the entities contained in them, and in which they are contained. We represent each sen­tence as a parse tree, with the words as leaves, and with phrases corresponding to each entity (and a node which joins the entire sentence). Our trees look just like syntactic constituency trees, such as those in the Perm TreeBank (Marcus et al., 1993),</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">141</doubt><p><i>Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, </i>pages 141-150, Singapore, 6-7 August 2009. ©2009 ACL and AFNLP<page local="2" global="142"/></p><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">ROOT</doubt><p>PROT PROT PROT NN    NN NN</p><doubt alpha="54.5" length="11" tooSmall="False" monospace="0.0">PROT   , CC</doubt><doubt alpha="80.0" length="5" tooSmall="False" monospace="0.0">NN NN</doubt><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">PROT</doubt><p>NN NN NNS</p><doubt alpha="83.3" length="6" tooSmall="False" monospace="0.0">VBD DT</doubt><p>DNA PROT NN NN</p><doubt alpha="80.0" length="5" tooSmall="False" monospace="0.0">IN DT</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">DNA</doubt><p>NN PROT NN</p><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">NN</doubt><p>PEBP2 alpha A1 , alpha B1 , and alpha B2 proteins bound the PEBP2 site within the mouse GM-CSF promoter</p><p>Figure 1: An example of our tree representation over nested named entities. The sentence is from the GENIA corpus. <i>PROT </i>is short for <i>PROTEIN.</i></p><p>but they tend to be much flatter. This model allows us to include parts of speech in the tree, and there­fore to jointly model the named entities and the part of speech tags. Once we have converted our sentences into parse trees, we train a discrimina­tive constituency parser similar to that of (Finkel et al., 2008). We found that on top-level enti­ties, our model does just as well as more conven­tional methods. When evaluating on <i>all </i>entities our model does well, with F-scores ranging from slightly worse than performance on top-level only, to substantially better than top-level only.</p></section><section number="2" title="Related Work"><p>There is a large body of work on named en­tity recognition, but very little of it addresses nested entities. Early work on the GENIA cor­pus (Kazama et al., 2002; Tsuruoka and Tsujii, 2003) only worked on the innermost entities. This was soon followed by several attempts at nested al., 2004; Zhou et al., 2004) which built hidden Markov models over the innermost named enti­ties, and then used a rule-based post-processing step to identify the named entities containing the innermost entities. Zhou (2006) used a more elab­orate model for the innermost entities, but then used the same rule-based post-processing method on the output to identify non-innermost entities. Gu (2006) focused only on proteins and DNA, by building separate binary SVM classifiers for inner­most and outermost entities for those two classes.</p><doubt alpha="61.0" length="41" tooSmall="False" monospace="0.0">NER in GENIA (Shen et al., 2003; Zhang et</doubt><p>Several techniques for nested NER in GENIA where presented in (Alex et al., 2007). Their first approach was to layer CRFs, using the output of one as the input to the next. For inside-out lay­ering, the first CRF would identify the innermost entities, the next layer would be over the words and the innermost entities to identify second-level entities, etc. For outside-in layering the first CRF would identify outermost entities, and then succes­sive CRFs would identify increasingly nested en­tities. They also tried a cascaded approach, with separate CRFs for each entity type. The CRFs would be applied in a specified order, and then each CRF could utilize features derived from the output of previously applied CRFs. This technique has the problem that it cannot identify nested en­tities of the same type; this happens frequently in the data, such as the nested <i>proteins </i>at the begin­ning of the sentence in Figure 1. They also tried a joint labeling approach, where they trained a sin­gle CRF, but the label set was significantly ex­panded so that a single label would include all of the entities for a particular word. Their best results where from the cascaded approach.</p><p>Byrne (2007) took a different approach, on his­torical archive text. She modified the data by con­catenating adjacent tokens (up to length six) into potential entities, and then labeled each concate­nated string using the C&amp;C tagger (Curran and Clark, 1999). When labeling a string, the "previ­ous" string was the one-token-shorter string con­taining all but the last token of the current string. For single tokens the "previous" token was the longest concatenation starting one token earlier.</p><p>SemEval 2007 Task 9 (Marquez et al., 2007b) included a nested NER component, as well as noun sense disambiguation and semantic role la­beling. However, the parts of speech and syn­tactic tree were given as part of the input, and named entities were specified as corresponding to noun phrases in the tree, or particular parts of speech. This restriction substantially changes the task. Two groups participated in the shared task, but only one (Marquez et al., 2007a) worked on the named entity component. They used a multi­label AdaBoost.MH algorithm, over phrases in the<page local="3" global="143"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">142</doubt><p>NNparent=DNA,grandparent=ROOT</p><doubt alpha="100.0" length="5" tooSmall="False" monospace="0.0">mouse</doubt><p>DNAparent=ROOT @DNAparent=ROOT,prev=NN,first=PROT</p><p>PROTparent=DNA,grandparent=ROOT NNparent=DNA,grandparent=ROOT NNparent=PROT,grandparent=DNA</p><p>GM-CSF promoter</p><p>Figure 2: An example of a subtree after it has been annotated and binarized. Features are computed over this representation. An @ indicates a chart parser active state (incomplete constituent).</p><p>parse tree which, based on their labels, could po­tentially be entities.</p><p>Finally, McDonald et al. (2005) presented a technique for labeling potentially overlapping seg­ments of text, based on a large margin, multilabel classification algorithm. Their method could be used for nested named entity recognition, but the experiments they performed were on joint (flat) NER and noun phrase chunking.</p></section><section number="3" title="Nested Named Entity Recognition as Parsing"><p>Our model is quite simple - we represent each sen­tence as a constituency tree, with each named en­tity corresponding to a phrase in the tree, along with a root node which connects the entire sen­tence. No additional syntactic structure is rep­resented. We also model the parts of speech as preterminals, and the words themselves as the leaves. See Figure 1 for an example of a named entity tree. Each node is then annotated with both its parent and grandparent labels, which allows the model to learn how entities nest. We bina-rize our trees in a right-branching manner, and then build features over the labels, unary rules, and binary rules. We also use first-order horizon­tal Markovization, which allows us to retain some information about the previous node in the bina-rized rule. See Figure 2 for an example of an an­notated and binarized subtree. Once each sentence has been converted into a tree, we train a discrimi­native constituency parser, based on (Finkel et al., 2008).</p><p>It is worth noting that if you use our model on data which does not have any nested entities, then it is precisely equivalent to a semi-CRF (Sarawagi and Cohen, 2004; Andrew, 2006), but with no length restriction on entities. Like a semi-CRF, we are able to define features over entire entities of arbitrary length, instead of just over a small, fixed window of words like a regular linear chain CRF.</p><p>We model part of speech tags jointly with the named entities, though the model also works with­out them. We determine the possible part of speech tags based on distributional similarity clus­ters. We used Alexander Clarke's software,<footnote anchor="1"/> based on (Clark, 2003), to cluster the words, and then allow each word to be labeled with any part of speech tag seen in the data with any other word in the same cluster. Because the parts of speech are annotated with the parent (and grandparent) labels, they determine what, if any, entity types a word can be labeled with. Many words, such as verbs, cannot be labeled with any entities. We also limit our grammar based on the rules observed in the data. The rules whose children include part of speech tags restrict the possible pairs of adjacent tags. Interestingly, the restrictions imposed by this joint modeling (both observed word/tag pairs and observed rules) actually result in much faster infer­ence (and therefore faster train and test times) than a model over named entities alone. This is differ­ent from most work on joint modeling of multiple levels of annotation, which usually results in sig­nificantly slower inference.</p><subsection number="3.1" title="Discriminative Constituency Parsing"><p>We train our nested NER model using the same technique as the discriminatively trained, condi­tional random field-based, CRF-CFG parser of (Finkel et al., 2008).  The parser is similar to a<page local="4" global="144"/></p><footnote label="1"> http://www.cs.rhul.ac.uk/home/alexc/RHUL/Downloads.html</footnote><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">143</doubt><p><b>Local Features</b> <b>Pairwise Features</b></p><p>Table 1: The local and pairwise NER features used in all of our experiments. Consult the text for a full description of all features, which includes feature classes not in this table.</p><p>chart-based PCFG parser, except that instead of putting probabilities over rules, it puts <i>clique po­tentials </i>over local subtrees. These unnormalized potentials know what span (and split) the rule is over, and arbitrary features can be defined over the local subtree, the span/split and the words of the sentence. The inside-outside algorithm is run over the clique potentials to produce the partial deriva­tives and normalizing constant which are neces­sary for optimizing the log likelihood. Optimiza­tion is done by stochastic gradient descent.</p><p>The only real drawback to our model is run­time. The algorithm is <i>0(n<footnote anchor="3"/>) </i>in sentence length. Training on all of GENIA took approximately 23 hours for the nested model and 16 hours for the semi-CRF. A semi-CRF <i>with </i>an entity length re­striction, or a regular CRF, would both have been faster. At runtime, the nested model for GENIA tagged about 38 words per second, while the semi-CRF tagged 45 words per second. For compar­ison, a first-order linear chain CRF trained with similar features on the same data can tag about 4,000 words per second.</p></subsection></section><section number="4" title="Features"><p>When designing features, we first made ones sim­ilar to the features typically designed for a first-order CRF, and then added features which are not possible in a CRF, but are possible in our enhanced representation. This includes features over entire entities, features which directly model nested en­tities, and joint features over entities and parts of speech. When features are computed over each label, unary rule, and binary rule, the feature func­tion is aware of the rule span and split.</p><p>Each word is labeled with its distributional similarity cluster <i>(distsim), </i>and a string indicating orthographic information <i>(shape) </i>(Finkel et al., 2005). Subscripts represent word position in the sentence. In addition to those below, we include features for each fully annotated label and rule.</p><p><b>Local named entity features. </b>Local named en­tity features are over the label for a single word. They are equivalent to the local features in a linear chain CRF. However, unlike in a linear chain CRF, if a word belongs to multiple entities then the local features are computed for each entity. Local fea­tures are also computed for words not contained in any entity. Local features are in Table 1.</p><p><b>Pairwise named entity features. </b>Pairwise fea­tures are over the labels for adjacent words, and are equivalent to the edge features in a linear chain CRF. They can occur when pairs of words have the same label, or over entity boundaries where the words have different labels. Like with the lo­cal features, if a pair of words are contained in, or straddle the border of, multiple entities, then the features are repeated for each. The pairwise fea­tures we use are shown in Table 1.</p><p><b>Embedded named entity features. </b>Embedded named entity features occur in binary rules where one entity is the child of another entity. For our embedded features, we replicated the pairwise fea­tures, except that the embedded named entity was treated as one of the words, where the "word" (and other annotations) were indicative of the type of entity, and not the actual string that is the en­tity. For instance, in the subtree in Figure 2, we would compute <i>wordj+labelj-i+labelj </i>as <i>PROT-DNA-DNA </i>for 7 = 18 (the index of the word <i>GM-CSF). </i>The normal pairwise feature at the same po-<page local="5" global="145"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">144</doubt><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>label.</p></td><td class="cell"><p></p></td><td class="cell"><p>distsim; + distsini;_i + label;</p></td><td class="cell"><p>label;-1</p></td><td class="cell"><p>+ label;</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>word, +</p></td><td class="cell"><p>label;</p></td><td class="cell"><p>shape; + shape!+i + label;</p></td><td class="cell"><p>word; +</p></td><td class="cell"><p>label;_i +</p></td><td class="cell"><p>label;</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>wordj_i</p></td><td class="cell"><p>+ label;</p></td><td class="cell"><p>shape;-1 + shape; + label;</p></td><td class="cell"><p>word;_i</p></td><td class="cell"><p>+ label;-]</p></td><td class="cell"><p>+ label;</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>word!+i</p></td><td class="cell"><p>+ label;</p></td><td class="cell"><p>word;_i + shape; + label;</p></td><td class="cell"><p>word;+i</p></td><td class="cell"><p>+ label;_i</p></td><td class="cell"><p>+ label;</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>distsim;</p></td><td class="cell"><p>+ label.</p></td><td class="cell"><p>shape; + word;+i + label;</p></td><td class="cell"><p>distsim;</p></td><td class="cell"><p>+ label;-!</p></td><td class="cell"><p>+ label;</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>distsim;</p></td><td class="cell"><p>_i + label.</p></td><td class="cell"><p>words in a 5 word window</p></td><td class="cell"><p>distsim;</p></td><td class="cell"><p>_i + label;</p></td><td class="cell"><p>_i + label;</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>distsim;</p></td><td class="cell"><p>+1 + label;</p></td><td class="cell"><p>prefixes up to length 6</p></td><td class="cell"><p>distsim;</p></td><td class="cell"><p>+l + label;</p></td><td class="cell"><p>_! + label;</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>shape, 4</p></td><td class="cell"><p>- label;</p></td><td class="cell"><p>suffixes up to length 6</p></td><td class="cell"><p>distsim;</p></td><td class="cell"><p>_i + distsim; + label;-</p></td><td class="cell"><p>1 + label;</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>shape;_</p></td><td class="cell"><p>1 + label;</p></td><td class="cell"><p></p></td><td class="cell"><p>shape; 4</p></td><td class="cell"><p>- label;-!</p></td><td class="cell"><p>- label;</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>shapei+</p></td><td class="cell"><p>1 + label;</p></td><td class="cell"><p></p></td><td class="cell"><p>shape;_</p></td><td class="cell"><p>1 + label;-</p></td><td class="cell"><p>1 + label;</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>shape!+</p></td><td class="cell"><p>! + label;-</p></td><td class="cell"><p>1 + label;</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>shape;_</p></td><td class="cell"><p>1 + shape;</p></td><td class="cell"><p>+ label;-1 4</p></td><td class="cell"><p>label;</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>shape;-</p></td><td class="cell"><p>1 + shapeH</p></td><td class="cell"><p>_i + label j_</p></td><td class="cell"><p>1 + label;</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p><b>GENIA - Testing on AU Entities</b> <b>Nested NER Model Semi-CRF Model</b> <b>GENIA - Testing on Top-level Entities Only</b></p><p>sition would be <i>GM-CSF-DNA-DNA.</i></p><p><b>Whole entity features. </b>We had four whole en­tity features: the entire phrase; the preceding and following word; the preceding and following dis­tributional similarity tags; and the preceding dis­tributional similarity tag with the following word.</p><p><b>Local part of speech features. </b>We used the same POS features as (Finkel et al., 2008).</p><p><b>Joint named entity and part of speech features.</b></p><p>For the joint features we replicated the POS fea­tures, but included the parent of the POS, which either is the innermost entity type, or would indi­cate that the word is not in any entities.</p></section><section number="5" title="Experiments"><p>We performed two sets of experiments, the first set over biomedical data, and the second over Spanish and Catalan newspaper text. We designed our ex­periments to show that our model works just as well on outermost entities, the typical NER task, and also works well on nested entities.</p><subsection number="5.1" title="GENIA Experiments 5.1.1 Data"><p>We performed experiments on the GENIA v.3.02 corpus (Ohta et al., 2002). This corpus contains 2000 Medline abstracts (œ500k words), annotated with 36 different kinds of biological entities, and with parts of speech. Previous NER work using this corpus has employed 10-fold cross-validation for evaluation. We wanted to explore different model variations (e.g., level of Markovization, and different sets of distributional similarity cluster­ings) and feature sets, so we needed to set aside a development set. We split the data by putting the first 90% of sentences into the training set, and the remaining 10% into the test set. This is the exact same split used to evaluate part of speech tagging in (Tsuruoka et al, 2005). For develop­ment we used the first half of the data to train, and the next quarter of the data to test.<footnote anchor="2"/> We made the same modifications to the label set as the organiz­ers of the JNLPBA 2004 shared task (Collier et al, 2004). They collapsed all <i>DNA </i>subtypes into <i>DNA; </i>all <i>RNA </i>subtypes into <i>RNA; </i>all <i>protein </i>sub­types into <i>protein; </i>kept <i>cell line </i>and <i>cell type; </i>and removed all other entities. However, they also re­moved all embedded entities, while we kept them.</p><p>As discussed in Section 3, we annotated each word with a distributional similarity cluster. We used 200 clusters, trained using 200 million words from PubMed abstracts. During development, we found that fewer clusters resulted in slower infer-</p><p>This split may seem strange: we had originally intended a 50/25/25 train/dev/test split, until we found the previously used 90/10 split.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">145</doubt><table caption="Table 2: Named entity results on GENIA, evaluating on all entities." class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p># Test</p></td><td class="cell"><p>(train on all entities)</p></td><td class="cell"><p>(train on top-level entities)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Entities</p></td><td class="cell"><p>Precision</p></td><td class="cell"><p>Recall</p></td><td class="cell"><p>Fi</p></td><td class="cell"><p>Precision</p></td><td class="cell"><p>Recall</p></td><td class="cell"><p>Fi</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Protein</p></td><td class="cell"><p>3034</p></td><td class="cell"><p>79.04</p></td><td class="cell"><p>69.22</p></td><td class="cell"><p>73.80</p></td><td class="cell"><p>78.63</p></td><td class="cell"><p>64.04</p></td><td class="cell"><p>70.59</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>DNA</p></td><td class="cell"><p>1222</p></td><td class="cell"><p>69.61</p></td><td class="cell"><p>61.29</p></td><td class="cell"><p>65.19</p></td><td class="cell"><p>71.62</p></td><td class="cell"><p>57.61</p></td><td class="cell"><p>63.85</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>RNA</p></td><td class="cell"><p>103</p></td><td class="cell"><p>86.08</p></td><td class="cell"><p>66.02</p></td><td class="cell"><p>74.73</p></td><td class="cell"><p>79.27</p></td><td class="cell"><p>63.11</p></td><td class="cell"><p>70.27</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Cell Line</p></td><td class="cell"><p>444</p></td><td class="cell"><p>73.82</p></td><td class="cell"><p>56.53</p></td><td class="cell"><p>64.03</p></td><td class="cell"><p>76.59</p></td><td class="cell"><p>59.68</p></td><td class="cell"><p>67.09</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Cell Type</p></td><td class="cell"><p>599</p></td><td class="cell"><p>68.77</p></td><td class="cell"><p>65.44</p></td><td class="cell"><p>67.07</p></td><td class="cell"><p>72.12</p></td><td class="cell"><p>59.60</p></td><td class="cell"><p>65.27</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Overall</b></p></td><td class="cell"><p>5402</p></td><td class="cell"><p>75.39</p></td><td class="cell"><p><b>65.90</b></p></td><td class="cell"><p><b>70.33</b></p></td><td class="cell"><p><b>76.17</b></p></td><td class="cell"><p>61.72</p></td><td class="cell"><p>68.19</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 3: Named entity results on GENIA, evaluating on only top-level entities." class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p># Test</p></td><td class="cell"><p>(train on all entities)</p></td><td class="cell"><p>(train on top-level entities)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Entities</p></td><td class="cell"><p>Precision</p></td><td class="cell"><p>Recall</p></td><td class="cell"><p>Fi</p></td><td class="cell"><p>Precision</p></td><td class="cell"><p>Recall</p></td><td class="cell"><p>Fi</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Protein</p></td><td class="cell"><p>2592</p></td><td class="cell"><p>78.24</p></td><td class="cell"><p>72.42</p></td><td class="cell"><p>75.22</p></td><td class="cell"><p>76.16</p></td><td class="cell"><p>72.61</p></td><td class="cell"><p>74.34</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>DNA</p></td><td class="cell"><p>1129</p></td><td class="cell"><p>70.40</p></td><td class="cell"><p>64.66</p></td><td class="cell"><p>67.41</p></td><td class="cell"><p>71.21</p></td><td class="cell"><p>62.00</p></td><td class="cell"><p>66.29</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>RNA</p></td><td class="cell"><p>103</p></td><td class="cell"><p>86.08</p></td><td class="cell"><p>66.02</p></td><td class="cell"><p>74.73</p></td><td class="cell"><p>79.27</p></td><td class="cell"><p>63.11</p></td><td class="cell"><p>70.27</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Cell Line</p></td><td class="cell"><p>420</p></td><td class="cell"><p>75.54</p></td><td class="cell"><p>58.81</p></td><td class="cell"><p>66.13</p></td><td class="cell"><p>76.59</p></td><td class="cell"><p>63.10</p></td><td class="cell"><p>69.19</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Cell Type</p></td><td class="cell"><p>537</p></td><td class="cell"><p>69.36</p></td><td class="cell"><p>70.39</p></td><td class="cell"><p>69.87</p></td><td class="cell"><p>71.11</p></td><td class="cell"><p>65.55</p></td><td class="cell"><p>68.22</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Overall</b></p></td><td class="cell"><p>4781</p></td><td class="cell"><p><b>75.22</b></p></td><td class="cell"><p><b>69.02</b></p></td><td class="cell"><p><b>71.99</b></p></td><td class="cell"><p>74.57</p></td><td class="cell"><p>68.27</p></td><td class="cell"><p>71.28</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="6" global="146"/><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">ROOT</doubt><p><b>JNLPBA 2004 - Testing on Top-level Entities Only</b></p><p>Table 4: Named entity results on the JNLPBA 2004 shared task data. Zhou and Su (2004) was the best system at the shared task, and is still state-of-the-art on the dataset.</p><doubt alpha="57.1" length="70" tooSmall="False" monospace="0.0">SP  AQ     NC   FC ORGANIZATION        VS DA   AQ   FE FC VM PERSON FP</doubt><p>DA ORGANIZATION</p><p>A doble partido ,     el Barça es el favorito "   , afirma Makaay   , delantero del Deportivo</p><doubt alpha="66.3" length="98" tooSmall="False" monospace="0.0">At double match   ,    the Barça is the favorite "   ,   states Makaay   ,   attacker of Deportivo</doubt><p>PERSON PERSON</p><doubt alpha="60.6" length="33" tooSmall="False" monospace="0.0">NP    FC    NC    SP ORGANIZATION</doubt><doubt alpha="0.0" length="1" tooSmall="True" monospace="0.0">[</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">NP</doubt><figure caption="Figure 3: An example sentence from the AnCora corpus, along with its English translation."></figure><p>ence with no improvement in performance.</p><subsubsection number="5.1.2" title="Experimental Setup"><p>We ran several sets of experiments, varying be­tween all entities, or just top-level entities, for training and testing. As discussed in Section 3, if we train on just top-level entities then the model is equivalent to a semi-CRF. Semi-CRFs are state-of-the-art and provide a good baseline for per­formance on just the top-level entities. Semi-CRFs are strictly better than regular, linear chain CRFs, because they can use all of the features and strucutre of a linear chain CRF, but also utilize whole-entity features (Andrew, 2006). We also evaluated the semi-CRF model on all entities. This may seem like an unfair evaluation, because the semi-CRF has no way of recovering the nested en­tities, but we wanted to illustrate just how much information is lost when using a flat representa­tion.</p></subsubsection><subsubsection number="5.1.3" title="Results"><p>Our named entity results when evaluating on all entities are shown in Table 2 and when evaluat­ing on only top-level entities are shown in Table 3. Our nested model outperforms the flat semi-CRF on both top-level entities and all entities.</p><p>While not our main focus, we also evaluated our models on parts of speech. The model trained on just top level entities achieved POS accuracy of 97.37%, and the one trained on all entities achieved 97.25% accuracy. The GENIA tagger (Tsuruoka et al., 2005) achieves 98.49% accuracy using the same train/test split.</p></subsubsection><subsubsection number="5.1.4" title="Additional JNLPBA 2004 Experiments"><p>Because we could not compare our results on the NER portion of the GENIA corpus with any other work, we also evaluated on the JNLPBA corpus. This corpus was used in a shared task for the BioNLP workshop at Coling in 2004 (Collier et al, 2004). They used the entire GENIA corpus for training, and modified the label set as discussed in Section 5.1.1. They also removed all embedded entities, and kept only the top-level ones. They then annotated new data for the test set. This dataset has no nested entities, but because the training data is GENIA we can still train our model on the data annotated with nested entities, and then evaluate on their test data by ignoring all embed­ded entities found by our named entity recognizer.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">146</doubt><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p><b>Nested NER Model</b></p></td><td class="cell"><p><b>Semi-CRF Model</b></p></td><td class="cell"><p><b>Zhou &amp; Su (2004)</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p># Test</p></td><td class="cell"><p>(train on all entities)</p></td><td class="cell"><p>(train on top-level entities)</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Entities</p></td><td class="cell"><p>Precision</p></td><td class="cell"><p>Recall</p></td><td class="cell"><p>Fi</p></td><td class="cell"><p>Precision</p></td><td class="cell"><p>Recall</p></td><td class="cell"><p>Fi</p></td><td class="cell"><p>Precision</p></td><td class="cell"><p>Recall</p></td><td class="cell"><p>Fi</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Protein</p></td><td class="cell"><p>4944</p></td><td class="cell"><p>66.98</p></td><td class="cell"><p>74.58</p></td><td class="cell"><p>70.57</p></td><td class="cell"><p>68.15</p></td><td class="cell"><p>62.68</p></td><td class="cell"><p>65.30</p></td><td class="cell"><p>69.01</p></td><td class="cell"><p>79.24</p></td><td class="cell"><p>73.77</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>DNA</p></td><td class="cell"><p>1030</p></td><td class="cell"><p>62.96</p></td><td class="cell"><p>66.50</p></td><td class="cell"><p>64.68</p></td><td class="cell"><p>65.45</p></td><td class="cell"><p>52.23</p></td><td class="cell"><p>58.10</p></td><td class="cell"><p>66.84</p></td><td class="cell"><p>73.11</p></td><td class="cell"><p>69.83</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>RNA</p></td><td class="cell"><p>115</p></td><td class="cell"><p>63.06</p></td><td class="cell"><p>60.87</p></td><td class="cell"><p>61.95</p></td><td class="cell"><p>64.55</p></td><td class="cell"><p>61.74</p></td><td class="cell"><p>63.11</p></td><td class="cell"><p>64.66</p></td><td class="cell"><p>63.56</p></td><td class="cell"><p>64.10</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Cell line</p></td><td class="cell"><p>487</p></td><td class="cell"><p>49.92</p></td><td class="cell"><p>60.78</p></td><td class="cell"><p>54.81</p></td><td class="cell"><p>49.61</p></td><td class="cell"><p>52.16</p></td><td class="cell"><p>50.85</p></td><td class="cell"><p>53.85</p></td><td class="cell"><p>65.80</p></td><td class="cell"><p>59.23</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Cell type</p></td><td class="cell"><p>1858</p></td><td class="cell"><p>75.12</p></td><td class="cell"><p>65.34</p></td><td class="cell"><p>69.89</p></td><td class="cell"><p>73.29</p></td><td class="cell"><p>55.81</p></td><td class="cell"><p>63.37</p></td><td class="cell"><p>78.06</p></td><td class="cell"><p>72.41</p></td><td class="cell"><p>75.13</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Overall</b></p></td><td class="cell"><p>8434</p></td><td class="cell"><p>66.78</p></td><td class="cell"><p>70.57</p></td><td class="cell"><p>68.62</p></td><td class="cell"><p>67.50</p></td><td class="cell"><p>59.27</p></td><td class="cell"><p>63.12</p></td><td class="cell"><p><b>69.42</b></p></td><td class="cell"><p><b>75.99</b></p></td><td class="cell"><p><b>72.55</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="7" global="147"/><p><b>Nested NER Model Semi-CRF Model</b> <b>AnCora Spanish - Testing on AU Entities</b> <b>AnCora Spanish - Testing on Top-level Entities Only</b></p><p>Table 6: Named entity results on the Spanish portion of AnCora, evaluating on only top-level entities.</p><p>This experiment allows us to show that our named entity recognizer works well on top-level entities, by comparing it with prior work. Our model also produces part of speech tags, but the test data is not annotated with POS tags, so we cannot show POS tagging results on this dataset.</p><p>One difficulty we had with the JNLPBA exper­iments was with tokenization. The version of GE-NIA distributed for the shared task is tokenized differently from the original GENIA corpus, but we needed to train on the original corpus as it is the only version with nested entities. We tried our best to retokenize the original corpus to match the distributed data, but did not have complete suc­cess. It is worth noting that the data is actually to-kenized in a manner which allows a small amount of "cheating." Normally, hyphenated words, such as <i>LPS-induced, </i>are tokenized as one word. How­ever, if the portion of the word before the hyphen is in an entity, and the part after is not, such as <i>BCR-induced, </i>then the word is split into two to­kens: <i>BCR </i>and <i>-induced. </i>Therefore, when a word starts with a hyphen it is a strong indicator that the prior word and it span the right boundary of an en­tity. Because the train and test data for the shared task do not contain nested entities, fewer words are split in this manner than in the original data. We did not intentionally exploit this fact in our feature design, but it is probable that some of our orthographic features "learned" this fact anyway. This probably harmed our results overall, because some hyphenated words, which straddled bound­aries in nested entities and would have been split in the original corpus (and were split in our train­ing data), were not split in the test data, prohibiting our model from properly identifying them.</p><p>For this experiment, we retrained our model on the entire, retokenized, GENIA corpus. We also retrained the distributional similarity model on the retokenized data. Once again, we trained one model on the nested data, and one on just the top­level entities, so that we can compare performance of both models on the top-level entities. Our full results are shown in Table 4, along with the cur­rent state-of-the-art (Zhou and Su, 2004). Besides the tokenization issues harming our performance, Zhou and Su (2004) also employed clever post­processing to improve their results.</p></subsubsection></subsection><subsection number="5.2" title="AnCora Experiments 5.2.1 Data"><p>We performed experiments on the NER portion of AnCora (Marti et al., 2007). This corpus has Spanish and Catalan portions, and we evaluated on both. The data is also annotated with parts of speech, parse trees, semantic roles and word<page local="8" global="148"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">147</doubt><table caption="Table 5: Named entity results on the Spanish portion of AnCora, evaluating on all entities." class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p># Test Entities</p></td><td class="cell"><p><b>Nested NER Model</b></p><p>(train on all entities) Precision    Recall Fj</p></td><td class="cell"><p><b>Semi-CRF Model</b></p><p>(train on top-level entities) Precision    Recall Fj</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Person</p></td><td class="cell"><p>1778</p></td><td class="cell"><p>65.29</p></td><td class="cell"><p>78.91</p></td><td class="cell"><p>71.45</p></td><td class="cell"><p>75.10</p></td><td class="cell"><p>32.73</p></td><td class="cell"><p>45.59</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Organization</p></td><td class="cell"><p>2137</p></td><td class="cell"><p>86.43</p></td><td class="cell"><p>56.90</p></td><td class="cell"><p>68.62</p></td><td class="cell"><p>47.02</p></td><td class="cell"><p>26.20</p></td><td class="cell"><p>33.65</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Location</p></td><td class="cell"><p>1050</p></td><td class="cell"><p>78.66</p></td><td class="cell"><p>46.00</p></td><td class="cell"><p>58.05</p></td><td class="cell"><p>84.94</p></td><td class="cell"><p>13.43</p></td><td class="cell"><p>23.19</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Date</p></td><td class="cell"><p>568</p></td><td class="cell"><p>87.13</p></td><td class="cell"><p>83.45</p></td><td class="cell"><p>85.25</p></td><td class="cell"><p>79.43</p></td><td class="cell"><p>29.23</p></td><td class="cell"><p>42.73</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Number</p></td><td class="cell"><p>991</p></td><td class="cell"><p>81.51</p></td><td class="cell"><p>80.52</p></td><td class="cell"><p>81.02</p></td><td class="cell"><p>66.27</p></td><td class="cell"><p>28.15</p></td><td class="cell"><p>39.52</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Other</p></td><td class="cell"><p>512</p></td><td class="cell"><p>17.90</p></td><td class="cell"><p>64.65</p></td><td class="cell"><p>28.04</p></td><td class="cell"><p>10.77</p></td><td class="cell"><p>16.60</p></td><td class="cell"><p>13.07</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Overall</b></p></td><td class="cell"><p>7036</p></td><td class="cell"><p><b>62.38</b></p></td><td class="cell"><p><b>66.87</b></p></td><td class="cell"><p><b>64.55</b></p></td><td class="cell"><p>51.06</p></td><td class="cell"><p>25.77</p></td><td class="cell"><p>34.25</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p># Test</p></td><td class="cell"><p>(train</p></td><td class="cell"><p>on all entities)</p></td><td class="cell"><p>(train on</p></td><td class="cell"><p>top-level entities)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Entities</p></td><td class="cell"><p>Precision</p></td><td class="cell"><p>Recall</p></td><td class="cell"><p>Fi</p></td><td class="cell"><p>Precision</p></td><td class="cell"><p>Recall</p></td><td class="cell"><p>Fi</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Person</p></td><td class="cell"><p>1050</p></td><td class="cell"><p>57.42</p></td><td class="cell"><p>66.67</p></td><td class="cell"><p>61.70</p></td><td class="cell"><p>71.23</p></td><td class="cell"><p>52.57</p></td><td class="cell"><p>60.49</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Organization</p></td><td class="cell"><p>1060</p></td><td class="cell"><p>77.38</p></td><td class="cell"><p>40.66</p></td><td class="cell"><p>53.31</p></td><td class="cell"><p>44.33</p></td><td class="cell"><p>49.81</p></td><td class="cell"><p>46.91</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Location</p></td><td class="cell"><p>279</p></td><td class="cell"><p>72.49</p></td><td class="cell"><p>36.04</p></td><td class="cell"><p>48.15</p></td><td class="cell"><p>79.52</p></td><td class="cell"><p>24.40</p></td><td class="cell"><p>37.34</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Date</p></td><td class="cell"><p>290</p></td><td class="cell"><p>72.29</p></td><td class="cell"><p>57.59</p></td><td class="cell"><p>64.11</p></td><td class="cell"><p>71.77</p></td><td class="cell"><p>51.72</p></td><td class="cell"><p>60.12</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Number</p></td><td class="cell"><p>519</p></td><td class="cell"><p>57.17</p></td><td class="cell"><p>49.90</p></td><td class="cell"><p>53.29</p></td><td class="cell"><p>54.87</p></td><td class="cell"><p>44.51</p></td><td class="cell"><p>49.15</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Other</p></td><td class="cell"><p>541</p></td><td class="cell"><p>11.30</p></td><td class="cell"><p>38.35</p></td><td class="cell"><p>17.46</p></td><td class="cell"><p>9.51</p></td><td class="cell"><p>26.88</p></td><td class="cell"><p>14.04</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Overall</b></p></td><td class="cell"><p>3739</p></td><td class="cell"><p><b>50.57</b></p></td><td class="cell"><p><b>49.72</b></p></td><td class="cell"><p><b>50.14</b></p></td><td class="cell"><p>46.07</p></td><td class="cell"><p>44.61</p></td><td class="cell"><p>45.76</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p><b>Nested NER Model Semi-CRF Model</b> <b>AnCora Catalan - Testing on AU Entities</b> <b>AnCora Catalan - Testing on Top-level Entities Only</b></p><p>Table 8: Named entity results on the Catalan portion of AnCora, evaluating on only top-level entities.</p><p>senses. The corpus annotators made a distinction between <i>strong </i>and <i>weak </i>entities. They define <i>strong </i>named entities as "a word, a number, a date, or a string of words that refer to a single individual entity in the real world." If a strong NE contains multiple words, it is collapsed into a single token. <i>Weak </i>named entities, "consist of a noun phrase, being it simple or complex" and must contain a <i>strong </i>entity. Figure 3 shows an example from the corpus with both strong and weak entities. The entity types present are <i>person, location, organi­zation, date, number, </i>and <i>other. </i>Weak entities are very prevalent; 47.1% of entities are embedded.</p><p>For Spanish, files starting with 7-9 were the test set, 5-6 were the development test set, and the re­mainder were the development train set. For Cata­lan, files starting with 8-9 were the test set, 6-7 were the development test set, and the remainder were the development train set. For both, the de­velopment train and test sets were combined to form the final train set. We removed sentences longer than 80 words. Spanish has 15,591 train­ing sentences, and Catalan has 14,906.</p><subsubsection number="5.2.2" title="Experimental Setup"><p>The parts of speech provided in the data include detailed morphological information, using a sim­ilar annotation scheme to the Prague TreeBank (Hana and Hanovâ, 2002). There are around 250 possible tags, and experiments on the development data with the full tagset where unsuccessful. We removed all but the first two characters of each POS tag, resulting in a set of 57 tags which more closely resembles that of the Perm TreeBank (Mar­cus et al, 1993). All reported results use our mod­ified version of the POS tag set.</p><p>We took only the words as input, none of the extra annotations. For both languages we trained a 200 cluster distributional similarity model over the words in the corpus. We performed the same set of experiments on AnCora as we did on GENIA.</p></subsubsection><subsubsection number="5.2.3" title="Results and Discussion"><p>The full results for Spanish when testing on all en­tities are shown in Table 5, and for only top-level entities are shown in Table 6. For part of speech tagging, the nested model achieved 95.93% accu­racy, compared with 95.60% for the flatly trained model. The full results for Catalan when testing on all entities are shown in Table 7, and for only top-level entities are shown in Table 8. POS tagging results were even closer on Catalan: 96.62% for the nested model, and 96.59% for the flat model.</p><p>It is not surprising that the models trained on all entities do significantly better than the flatly trained models when testing on all entities. The story is a little less clear when testing on just top-level entities.<page local="9" global="149"/> In this case, the nested model does 4.38% better than the flat model on the Spanish data, but 2.45% worse on the Catalan data. The overall picture is the same as for GENIA: model­ing the nested entities does not, on average, reduce performance on the top-level entities, but a nested entity model does substantially better when evalu­ated on all entities.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">148</doubt><table caption="Table 7: Named entity results on the Catalan portion of AnCora, evaluating on all entities." class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p># Test Entities</p></td><td class="cell"><p>(train all entities) Precision    Recall Fj</p></td><td class="cell"><p>(train top-level entities only) Precision    Recall Fj</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Person</p></td><td class="cell"><p>1303</p></td><td class="cell"><p>89.01</p></td><td class="cell"><p>50.35</p></td><td class="cell"><p>64.31</p></td><td class="cell"><p>70.08</p></td><td class="cell"><p>46.20</p></td><td class="cell"><p>55.69</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Organization</p></td><td class="cell"><p>1781</p></td><td class="cell"><p>68.95</p></td><td class="cell"><p>83.77</p></td><td class="cell"><p>75.64</p></td><td class="cell"><p>65.32</p></td><td class="cell"><p>41.77</p></td><td class="cell"><p>50.96</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Location</p></td><td class="cell"><p>1282</p></td><td class="cell"><p>76.78</p></td><td class="cell"><p>72.46</p></td><td class="cell"><p>74.56</p></td><td class="cell"><p>75.49</p></td><td class="cell"><p>36.04</p></td><td class="cell"><p>48.79</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Date</p></td><td class="cell"><p>606</p></td><td class="cell"><p>84.27</p></td><td class="cell"><p>81.35</p></td><td class="cell"><p>82.79</p></td><td class="cell"><p>70.87</p></td><td class="cell"><p>38.94</p></td><td class="cell"><p>50.27</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Number</p></td><td class="cell"><p>1128</p></td><td class="cell"><p>86.55</p></td><td class="cell"><p>83.87</p></td><td class="cell"><p>85.19</p></td><td class="cell"><p>75.74</p></td><td class="cell"><p>38.74</p></td><td class="cell"><p>51.26</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Other</p></td><td class="cell"><p>596</p></td><td class="cell"><p>85.48</p></td><td class="cell"><p>8.89</p></td><td class="cell"><p>16.11</p></td><td class="cell"><p>64.91</p></td><td class="cell"><p>6.21</p></td><td class="cell"><p>11.33</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Overall</b></p></td><td class="cell"><p>6696</p></td><td class="cell"><p><b>78.09</b></p></td><td class="cell"><p><b>68.23</b></p></td><td class="cell"><p><b>72.83</b></p></td><td class="cell"><p>70.39</p></td><td class="cell"><p>37.60</p></td><td class="cell"><p>49.02</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p># Test</p></td><td class="cell"><p>(train all entities)</p></td><td class="cell"><p>(train top-level entities only)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Entities</p></td><td class="cell"><p>Precision</p></td><td class="cell"><p>Recall</p></td><td class="cell"><p>Fi</p></td><td class="cell"><p>Precision</p></td><td class="cell"><p>Recall</p></td><td class="cell"><p>Fi</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Person</p></td><td class="cell"><p>801</p></td><td class="cell"><p>67.44</p></td><td class="cell"><p>47.32</p></td><td class="cell"><p>55.61</p></td><td class="cell"><p>62.63</p></td><td class="cell"><p>67.17</p></td><td class="cell"><p>64.82</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Organization</p></td><td class="cell"><p>899</p></td><td class="cell"><p>52.21</p></td><td class="cell"><p>74.86</p></td><td class="cell"><p>61.52</p></td><td class="cell"><p>57.68</p></td><td class="cell"><p>73.08</p></td><td class="cell"><p>64.47</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Location</p></td><td class="cell"><p>659</p></td><td class="cell"><p>54.86</p></td><td class="cell"><p>67.68</p></td><td class="cell"><p>60.60</p></td><td class="cell"><p>62.42</p></td><td class="cell"><p>57.97</p></td><td class="cell"><p>60.11</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Date</p></td><td class="cell"><p>296</p></td><td class="cell"><p>62.54</p></td><td class="cell"><p>66.55</p></td><td class="cell"><p>64.48</p></td><td class="cell"><p>59.46</p></td><td class="cell"><p>66.89</p></td><td class="cell"><p>62.96</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Number</p></td><td class="cell"><p>528</p></td><td class="cell"><p>62.35</p></td><td class="cell"><p>70.27</p></td><td class="cell"><p>66.07</p></td><td class="cell"><p>63.08</p></td><td class="cell"><p>68.94</p></td><td class="cell"><p>65.88</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Other</p></td><td class="cell"><p>342</p></td><td class="cell"><p>49.12</p></td><td class="cell"><p>8.19</p></td><td class="cell"><p>14.04</p></td><td class="cell"><p>45.61</p></td><td class="cell"><p>7.60</p></td><td class="cell"><p>13.03</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Overall</b></p></td><td class="cell"><p>3525</p></td><td class="cell"><p>57.67</p></td><td class="cell"><p>59.40</p></td><td class="cell"><p>58.52</p></td><td class="cell"><p><b>60.53</b></p></td><td class="cell"><p><b>61.42</b></p></td><td class="cell"><p><b>60.97</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></subsubsection></subsection></section><section number="6" title="Conclusions"><p>We presented a discriminative parsing-based method for nested named entity recognition, which does well on both top-level and nested enti­ties. The only real drawback to our method is that it is slower than common flat techniques. While most NER corpus designers have defenestrated embedded entities, we hope that in the future this will not continue, as large amounts of information are lost due to this design decision.</p></section><section title="Acknowledgements"><p>Thanks to Mihai Surdeanu for help with the An­Cora data. The first author was supported by a Stanford Graduate Fellowship. This paper is based on work funded in part by the Defense Ad­vanced Research Projects Agency through IBM. The content does not necessarily reflect the views of the U.S. Government, and no official endorse­ment should be inferred.</p></section><references><p>Beatrice Alex, Barry Haddow, and Claire Grover. 2007. Recognising nested named entities in biomedical text. In <i>BioNLP Workshop at ACL 2007, </i>pages 65-72.</p><p>Galen Andrew. 2006. A hybrid markov/semi-markov con­ditional random field for sequence segmentation. In <i>Pro­ceedings of the Conference on Empirical Methods in Nat­ural Language Processing (EMNLP 2006).</i></p><p>A. Borthwick. 1999. <i>A Maximum Entropy Approach to Named Entity Recognition. </i>Ph.D. thesis, New York Uni­versity.</p><p>Kate Byrne. 2007. Nested named entity recognition in his­torical archive text. In <i>ICSC '07: Proceedings of the Inter­national Conference on Semantic Computing, </i>pages 589-596.</p><p>Alexander Clark. 2003. Combining distributional and mor­phological information for part of speech induction. In <i>Proceedings of the tenth Annual Meeting of the European Association for Computational Linguistics (EACL), </i>pages 59-66.</p><p>Nigel Collier, J. Kim, Y. Tateisi, T. Ohta, and Y. Tsuruoka, ed­itors. 2004. <i>Proceedings of the International Joint Work­shop on NLP in Biomedicine and its Applications.</i></p><p>J. R. Curran and S. Clark. 1999. Language independent NER using a maximum entropy tagger. In <i>CoNLL 1999, </i>pages 164-167.</p><p>Jenny Finkel, Shipra Dingare, Christopher Manning, Malv-ina Nissim, Beatrice Alex, and Claire Grover. 2005. Ex­ploring the boundaries: Gene and protein identification in biomedical text. In <i>BMC Bioinformatics 6 (Suppl. 1).</i></p><p>Jenny Rose Finkel, Alex Kleeman, and Christopher D. Man­ning. 2008. Efficient, feature-based conditional random field parsing. In <i>ACL/HLT-2008.</i></p><p>BaohuaGu. 2006. Recognizing nested named entities in GE­NIA corpus. In <i>BioNLP Workshop at HLT-NAACL 2006, </i>pages 112-113.</p><p>Jin Hana and Hana Hanovâ. 2002. Manual for morpholog­ical annotation. Technical Report TR-2002-14, UK MFF CKL.</p><p>Jun'ichi Kazama, Takaki Makino, Yoshihiro Ohta, and Jun'ichi Tsujii. 2002. Tuning support vector machines for biomedical named entity recognition. In <i>Proceedings of the Workshop on Natural Language Processing in the Biomedical Domain (ACL 2002).</i></p><p>John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional Random Fields: Probabilistic mod­els for segmenting and labeling sequence data. In <i>ICML 2001, </i>pages 282-289. Morgan Kaufmann, San Francisco, CA.</p><p>Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. <i>Computational Linguis­tics, </i>19(2):313-330.</p><p>L. Marquez, L. Padrè, M. Surdeanu, and L. Villarejo. 2007a. UPC: Experiments withjoint learning within semeval task 9. In <i>Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007).</i></p><p>L. Marquez, L. Villarejo, M.A. Marti, and M. Taulè. 2007b. Semeval-2007 task 09: Multilevel semantic annotation of Catalan and Spanish. In <i>Proceedings of the 4th Inter­national Workshop on Semantic Evaluations (SemEval-2007).</i></p><p>M.A. Marti, M. Taulè, M. Bertran, and L. Marquez. 2007. Ancora: Multilingual and multilevel annotated corpora. MS, Universität de Barcelona.</p><p>Ryan McDonald, Koby Crammer, and Fernando Pereira. 2005. Flexible text segmentation with structured mul-tilabel classification. In <i>HLT '05: Proceedings of the conference on Human Language Technology and Empiri­cal Methods in Natural Language Processing, </i>pages 987-994.</p><p>Tomoko Ohta, Yuka Tateisi, and Jin-Dong Kim. 2002. The GENIA corpus: an annotated research abstract corpus in molecular biology domain. In <i>Proceedings of the second international conference on Human Language Technology Research, </i>pages 82-86.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">149</doubt><page local="10" global="150"/><p>Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In <i>Proceedings of CoNLL-2003.</i></p><p>Sunita Sarawagi and William W. Cohen. 2004. Semi-markov conditional random fields for information extraction. In <i>In Advances in Neural Information Processing Systems 17, </i>pages 1185-1192.</p><p>Dan Shen, Jie Zhang, Guodong Zhou, Jian Su, and Chew-Lim Tan. 2003. Effective adaptation of a hidden markov model-based named entity recognizer for biomedical do­main. In <i>Proceedings </i><i>ofthe</i><i> ACL 2003 workshop on Nat­ural language processing in biomedicine. </i>Association for Computational Linguistics (ACL 2003).</p><p>Yoshimasa Tsuruoka and Jun'ichi Tsujii. 2003. Boost­ing precision and recall of dictionary-based protein name recognition. In <i>Proceedings </i><i>ofthe</i><i> ACL-03 Workshop on Natural Language Processing in Biomedicine, </i>pages 41­48.</p><p>Yoshimasa Tsuruoka, Yuka Tateishi, Jin-Dong Kim, Tomoko Ohta, John McNaught, Sophia Ananiadou, and Jun'ichi Tsujii. 2005. Developing a robust part-of-speech tag­ger for biomedical text. In <i>Advances in Informatics -10th Panhellenic Conference on Informatics, LNCS 3746,</i> pages 382-392.</p><p>Jie Zhang, Dan Shen, Guodong Zhou, Jian Su, and Chew-Lim Tan. 2004. Enhancing HMM-based biomedical named entity recognition by studying special phenomena. <i>Jour­nal of Biomedical Informatics, </i>37(6):411—122.</p><p>GuoDong Zhou and Jian Su. 2004. Exploring deep knowledge resources in biomedical name recognition. In <i>Joint Workshop on Natural Language Processing in Biomedicine and Its Applications at Coling 2004.</i></p><p>Guodong Zhou, Jie Zhang, Jian Su, Dan Shen, and Chewlim Tan. 2004. Recognizing names in biomedical texts: a machine learning approach. <i>Bioinformatics, </i>20(7):11781190.</p><p>Guodong Zhou. 2006. Recognizing names in biomedical texts using mutual information independence model and SVM plus sigmoid. <i>International Journal of Medical In­formatics, </i>75:456-467.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">150</doubt></references></body></article>