<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1"/><title>Experiments in Telugu NER: A Conditional Random Field Approach</title><pubinfo>Proceedings of the IJCNLP-08 Workshop on NER for South and South East Asian Languages,pages 105-110, Hyderabad, India, January 2008. ©2008 Asian Federation of Natural Language Processing</pubinfo><author surname="Shishtla" givenname="Praneeth M"><org  name="International Institute of Information Technology" country="India" city="Hyderabad"/></author><author surname="Gali" givenname="Karthik"><org  name="International Institute of Information Technology" country="India" city="Hyderabad"/></author><author surname="Pingali" givenname="Prasad"><org  name="International Institute of Information Technology" country="India" city="Hyderabad"/></author><author surname="Varma" givenname="Vasudeva"><org  name="International Institute of Information Technology" country="India" city="Hyderabad"/></author></firstpageheader><frontmatter><p><b>Experiments in Telugu NER: A Conditional Random Field Approach</b></p><p><b>Praneeth M Shishtla, Karthik Gali, Prasad Pingali and Vasudeva Varma</b></p><p>{praneethms,karthikg}@students.iiit.ac.in,{pvvpr,vv}@iiit.ac.in</p><p>Language Technologies Research Centre International Institute of Information Technology Hyderabad, India</p></frontmatter><abstract>Named Entity Recognition(NER) is the task of identifying and classifying tokens in a text document into predefined set of classes. In this paper we show our experiments with various feature combinations for Tel-ugu NER. We also observed that the prefix and suffix information helps a lot in find­ing the class of the token. We also show the effect of the training data on the perfor­mance of the system. The best performing model gave an <i>Fß</i>=1 measure of 44.91. The language independent features gave an <i>Fß=1</i>measure of 44.89 which is close to <i>Fß=1</i>measure obtained even by including the lan­guage dependent features. </abstract></header><body><section number="1" title="Introduction"><p>The objective of NER is to identify and classify all tokens in a text document into predefined classes such as person, organization, location, miscella­neous. The Named Entity information in a document is used in many of the language processing tasks. NER was created as a subtask in Message Under­standing Conference (MUC) (Chinchor, 1997). This reflects the importance of NER in the area of Infor­mation Extraction (IE). NER has many applications in the areas of Natural Language Processing, Infor­mation Extraction, Information Retrieval and speech processing. NER is also used in question answer­ing systems (Toral et al., 2005; Molla et al., 2006), and machine translation systems (Babych and Hart­ley, 2003). It is also a subtask in organizing and retrieving biomedical information (Tsai, 2006). The process of NER consists of two steps• identification of boundaries of proper nouns.</p><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">1G5</doubt><p>• classification of these identified proper nouns.</p><p>The Named Entities(NEs) should be correctly iden­tified for their boundaries and later correctly classi­fied into their class. Recognizing NEs in an English document can be done easily with a good amount of accuracy(using the capitalization feature). Indian Languages are very much different from the English like languages.</p><p>Some challenges in named entity recognition that are found across various languages are: Many named entities(NEs) occur rarely in the corpus i.e they belong to the open class of nouns. Ambiguity of NEs. Ex <i>Washington </i>can be a person's name or a place name. There are many ways of mentioning the same Named Entity(NE). In case of person names, <i>Ex: Abdul Kalam, A.P.J.Kalam, Kalam </i>refer to the same person. And, in case of place names <i>Waran-gal, WGL </i>both refer to the same location. Named Entities mostly have initial capital letters. This dis­criminating feature of NEs can be used to solve the problem to some extent in English.</p><p>Indian Languages have some additional chal­lenges: We discuss the challenges that are specific to Telugu. Absence of capitalization. Ex: The con­densed form of the person name <i>S.R.Shastry </i>is writ­ten as <i>S.R.S </i>in English and is represented as <i>srs </i>in Telugu. Agglutinative property of the Indian Lan­guages makes the identification more difficult. Ag­glutinative languages such as Turkish or Finnish, Telugu etc. differ from languages like English in the way lexical forms are generated.<page local="2" global="106"/> Words are formed by productive affixations of derivational and inflectional suffixes to roots or stems. <i>For example: warangal, warangal ki, warangalki, warangallo, warangal ni </i>etc .. all refer to the place Waran-gal. where <i>lo, ki, ni </i>are all postpostion markers in Telugu. All the postpositions get added to the stem hyderabad. There are many ways of represent­ing acronyms. The letters in acronyms could be the English alphabet or the native alphabet. Ex: <i>B.J.P </i>and <i>BaJaPa </i>both are acronyms of <i>Bharatiya Janata Party. </i>Telugu has a relatively free word order when compared with English. The morpohology of Tel-ugu is very complex. The Named Entity Recogni­tion algorithm must be able handle most of these above variations which otherwise are not found in languages like English. There are not rich and robust tools for the Indian Languages. For Telugu, though a Part Of Speech(POS) Tagger for Telugu, is avail­able, the accuracy is less when compared to English and Hindi.</p></section><section number="2" title="Problem Statement"><p><b>NER as sequence labelling task</b></p><p>Named entity recognition (NER) can be modelled as a sequence labelling task (Lafferty et al., 2001). Given an input sequence of words <i>Wn</i><i> </i>= w1w2w3...<i>w</i><i>n, </i>the NER task is to construct a label sequence L" = l1l2l3 ...ln , where label <i>li </i>either belongs to the set of predefined classes for named entities or is none(representing words which are not proper nouns). The general label sequence <i>l1 </i>has the high­est probability of occuring for the word sequence <i>W1 </i>among all possible label sequences, that is</p><doubt alpha="44.4" length="27" tooSmall="False" monospace="0.0">L? = argmax {Pr (L? | Wf) }</doubt></section><section number="3" title="Conditional Random Fields"><p>Conditional Random Fields (CRFs) (Wallach, 2004) are undirected graphical models used to calculate the conditional probability of values on designated out­put nodes given values assigned to other designated input nodes. In the special case in which the output nodes of the graphical model are linked by edges in a linear chain, CRFs make a first-order Markov inde­pendence assumption, and thus can be understood as conditionally-trained finite state machines(FSMs).</p><p>Let o = ( <i>O1,O2,...OT) </i>be some observed input data sequence, such as a sequence of words in text in a document,(the values on n input nodes of the graphical model). Let <b>S </b>be a set of FSM states, each of which is associated with a label, l e L. Let <b>s </b>= (s1 ,s2,... <i>sT</i>,) be some sequence of states,(the values on T output nodes). By the Hammersley-Clifford theorem CRFs define the conditional prob­ability of a state sequence given an input sequence to be</p><doubt alpha="50.0" length="2" tooSmall="False" monospace="0.0">1T</doubt><doubt alpha="41.7" length="48" tooSmall="False" monospace="0.0">P(s|o) = — *exp(       Akfk(st-1, St,o,t))Zot=1k</doubt><p>where <i>Zo </i>is a normalization factor over all state sequences, is an arbitrary feature function over its ar­guments, and <i>Ak</i><i> </i>is a learned weight for each feature function. A feature function may, for example, be defined to have value 0 or 1. Higher A weights make their corresponding FSM transitions more likely.</p><p>CRFs define the conditional probability of a la­bel sequence based on total probability over the state sequences, <i>P(l</i>|o) = Es:l(S)=lP(s|o) where l(s) is the sequence of labels corresponding to the labels of the states in sequence s. Note that the normalization fac­tor, <i>Z</i><i>o, </i>(also known in statistical physics as the parti­tion function) is the sum of the scores of all possible state sequences,</p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">T</doubt><doubt alpha="55.6" length="27" tooSmall="False" monospace="0.0">Zo=£*exp(Akfk(st-1,St,o,t))</doubt><p><i>s</i><i>e</i><i>ST        t</i><i>=1 </i><i>k</i></p><p>and that the number of state sequences is expo­nential in the input sequence length,T. In arbitrarily-structure CRFs, calculating the partition function in closed form is intractable, and approximation meth­ods such as Gibbs sampling, or loopy belief propa­gation must be used.</p></section><section number="4" title="Features"><p>There are many types of features used in general NER systems. Many systems use binary features i.e. the word-internal features, which indicate the presence or absence of particular property in the word. (Mikheev, 1997; Wacholder et al., 1997; Bikel et al., 1997). Following are examples of binary features commonly used. All-Caps (IBM), Internal capitalization (eBay), initial capital (Abdul Kalam), uncapitalized word (can), 2-digit number (83, 28), 4-digit number (1273, 1984), all digits (8, 31, 1228) etc.<page local="3" global="107"/> The features that correspond to the capitalization are not applicable to Telugu. We have not used any binary features in our experiments.</p><p>Gazetteers are used to check if a part of the named entity is present in the gazetteers. We don't have proper gazetteers for Telugu.</p><p>Lexical features like a sliding window [w-2,w-1,wo,w1,w2] are used to create a lexi­cal history view. Prefix and suffix tries were also used previously(Cucerzan and Yarowsky,1999).</p><p>Linguistics features like Part Of Speech, Chunk, etc are also used.</p><subsection number="4.1" title="Our Features"><p>We donot have a highly accurate Part Of Speech(POS) tagger. In order to obtain some POS and chunk information, we ran a POS Tagger and chunker for telugu (PVS and G, 2007) on the data. And from that, we used the following features in our experiments.</p><p>Language Independent Features</p><doubt alpha="61.3" length="181" tooSmall="False" monospace="0.0">current token: w0previous 3 tokens: w-3,w-2,w-1next 3 tokens: w1,w2,w3compound feature:w0w1compound feature:w-1w0prefixes (len=1,2,3,4) of w0: pre0suffixes (len=1,2,3,4) of w0: suf0</doubt><p>Language Dependent Features</p><p>POS of current word: POS0Chunk of current word: <i>Chunk0</i></p><p>Each feature is capable of providing some infor­mation about the NE.</p><p>The word window helps in using the context in­formation while guessing the tag of the token. The prefix and suffix feature to some extent help in cap­turing the variations that may occur due to aggluti­nation.</p><p>The POS tag feature gives a hint whether the word is a proper noun. When this is a proper noun it has a chance of being a NE. The chunk feature helps in finding the boundary of the NE.</p><p>In Indian Languages suffixes and other inflections get attached to the words increasing the length ofthe word and reducing the number of occurences of that word in the entire corpus. The character n-grams can capture these variations.</p></subsection></section><section number="5" title="Experimental Setup"><subsection number="5.1" title="Corpus"><p>We conducted the experiments on the developement data released as a part of NER for South and South­East Asian Languages (NERSSEAL) Competetion. The corpus in total consisted of 64026 tokens out of which 10894 were Named Entities(NEs). We di­vided the corpus into training and testing sets. The training set consisted of 46068 tokens out of which 8485 were NEs. The testing set consisted of 17951 tokens out of which 2407 were NEs. The tagset as mentioned in the release, was based on AUKBC's ENAMEX,TIMEX and NAMEX, has the follow­ing tags: NEP (Person), NED (Designation), NEO (Organization), NEA (Abbreviation), NEB (Brand), NETP (Title-Person), NETO (Title-Object), NEL (Measure) &amp; NETE (Terms).</p><doubt alpha="66.7" length="42" tooSmall="False" monospace="0.0">(Location), NETI (Time), NEN (Number), NEM</doubt></subsection><subsection number="5.2" title="Tagging Scheme"><p>The corpus is tagged using the IOB tagging scheme (Ramshaw and Marcus, 1995). In this scheme each line contains a word at the beginning followed by its tag. The tag encodes the type of named entity and whether the word is in the beginning or inside the NE. Empty lines represent sentence(document) boundaries. An example is given in table 1.</p><p>Words tagged with O are outside of named en­tities and the I-XXX tag is used for words inside a named entity of type XXX. Whenever two entities of type XXX are immediately next to each other, the first word of the second entity will be tagged B-XXX in order to show that it starts another entity. This tagging scheme is the IOB scheme originally put forward by Ramshaw and Marcus (1995).</p></subsection><subsection number="5.3" title="Experiments"><p>To evaluate the performance of our Named Entity Recognizer, we used three standard metrics namely precision, recall and f-measure. Precision measures the number of correct Named Entities(NEs) in the machine tagged file over the total number of NEs in the machine tagged file and the recall measures the number of correct NEs in the machine tagged file over the total number of NEs in the golden standard file while F-measure is the weighted harmonic mean of precision and recall:<page local="4" global="108"/></p><doubt alpha="37.5" length="16" tooSmall="False" monospace="0.0">=Q82+1)Rpp2R + P</doubt><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">with</doubt><doubt alpha="25.0" length="4" tooSmall="False" monospace="0.0">P =1</doubt><doubt alpha="63.8" length="94" tooSmall="False" monospace="0.0">where P is Precision, R is Recall and F is F-measure.W-n+n:A word window:w-n,w-n+1,.. ,w-1,w0,</doubt><p>w1, .. , <i>wn</i>-1, <i>wn.</i><i>POS</i><i>n: </i>POS <i>nth </i>token.</p><p><i>Ch</i><i>n: </i>Chunk of <i>nth </i>token.</p><p><i>pren:   </i>Prefix information of <i>nth </i>token. (prefix <i>sufn:</i><i>   </i>Suffix information of <i>nth </i>token. (suffix</p><doubt alpha="40.0" length="15" tooSmall="False" monospace="0.0">length=1,2,3,4)</doubt><p>The more the features, the better is the perfor­mance. The inclusion of the word window, prefix and suffix features have increased the <i>Fp</i>=1 mea­sure significantly. Whenever the suffix feature is included, the performance of the system increased. This shows that the system is able to caputure those agglutinative language variations. We also have ex­perimented changing the training data size. While varying the training data size, we have tested the performance on the same amount of testing data of 17951 tokens.</p></subsection></section><section number="6" title="Conclusion &amp; Future Work"><p>The inclusion of prefix and suffix feature helps in improving the <i>Fp</i>=1 measure (also recall) of the sys­tem. As the size of the training data is increased, the <i>Fp=1 </i>measure is increased. Even without the language specific information the system is able to perform well. The suffix feature helped improve the recall. This is due to the fact that the POS tagger also uses the same features in predicting the POS tags. Prefix, suffix and word are three non-linguistic features that resulted in good performance. We plan to experiment with the character n-gram approach (Klein et al., 2003) and include gazetteer informa­tion.</p><table caption="Table 1: IOB tagging scheme." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Token</p></td><td class="cell"><p>Named Entity Tag</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Swami</p></td><td class="cell"><p>B-NEP</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Vivekananda</p></td><td class="cell"><p>I-NEP</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>was</p></td><td class="cell"><p>O</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>born</p></td><td class="cell"><p>O</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>on</p></td><td class="cell"><p>O</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>January</p></td><td class="cell"><p>B-NETI</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>,</p></td><td class="cell"><p>I-NETI</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>12</p></td><td class="cell"><p>I-NETI</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>in</p></td><td class="cell"><p>O</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Calcutta</p></td><td class="cell"><p>B-NEL</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>O</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Features</p></td><td class="cell"><p>Precision</p></td><td class="cell"><p>Recall</p></td><td class="cell"><p><i>Fß</i>=1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Cho</p></td><td class="cell"><p>51.41%</p></td><td class="cell"><p>9.19%</p></td><td class="cell"><p>15.59</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>POSo</p></td><td class="cell"><p>46.32%</p></td><td class="cell"><p>9.52%</p></td><td class="cell"><p>15.80</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>POSo.Cho</p></td><td class="cell"><p>46.63%</p></td><td class="cell"><p>9.69%</p></td><td class="cell"><p>16.05</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>W-3+3.Cho</p></td><td class="cell"><p>59.08%</p></td><td class="cell"><p>19.50%</p></td><td class="cell"><p>29.32</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>W-3+3.POSo</p></td><td class="cell"><p>58.43%</p></td><td class="cell"><p>19.61%</p></td><td class="cell"><p>29.36</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Cho. <i>pre</i><i>n</i></p></td><td class="cell"><p>53.97%</p></td><td class="cell"><p>24.76%</p></td><td class="cell"><p>33.95</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>POSo. pre</i><i>n</i></p></td><td class="cell"><p>53.94%</p></td><td class="cell"><p>24.93%</p></td><td class="cell"><p>34.10</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>POSo.Cho. pre</i><i>n</i></p></td><td class="cell"><p>53.94%</p></td><td class="cell"><p>25.32%</p></td><td class="cell"><p>34.46</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>POSo.suf</i><i>n</i></p></td><td class="cell"><p>47.51%</p></td><td class="cell"><p>29.36%</p></td><td class="cell"><p>36.29</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>POS0.Ch0.sufn</i></p></td><td class="cell"><p>48.02%</p></td><td class="cell"><p>29.24%</p></td><td class="cell"><p>36.35</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Cho.suf</i><i>n</i></p></td><td class="cell"><p>48.55%</p></td><td class="cell"><p>29.13%</p></td><td class="cell"><p>36.41</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>W-3+3.POS0. pre</i><i>n</i></p></td><td class="cell"><p>62.98%</p></td><td class="cell"><p>27.45%</p></td><td class="cell"><p>38.24</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>W-3+3 <i>.POSo.Cho. pre</i><i>n</i></p></td><td class="cell"><p>62.95%</p></td><td class="cell"><p>27.51%</p></td><td class="cell"><p>38.28</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>W-3+3.Cho. <i>pre</i><i>n</i></p></td><td class="cell"><p>62.88%</p></td><td class="cell"><p>27.62%</p></td><td class="cell"><p>38.38</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>W-3+3.POSo.suf</i><i>n</i></p></td><td class="cell"><p>60.09%</p></td><td class="cell"><p>30.53%</p></td><td class="cell"><p>40.49</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>W-3+3 <i>.POSo.Cho.suf</i><i>n</i></p></td><td class="cell"><p>59.93%</p></td><td class="cell"><p>30.59%</p></td><td class="cell"><p>40.50</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>W-3+3.Cho.suf</i><i>n</i></p></td><td class="cell"><p>61.18%</p></td><td class="cell"><p>30.81%</p></td><td class="cell"><p>40.98</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>POSo.Cho. </i>pren.su<i> f</i><i>n</i></p></td><td class="cell"><p>57.83%</p></td><td class="cell"><p>34.57%</p></td><td class="cell"><p>43.27</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>POSo. pre</i><i>n</i><i>.suf</i><i>n</i></p></td><td class="cell"><p>57.41%</p></td><td class="cell"><p>34.73%</p></td><td class="cell"><p>43.28</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Cho. pre</i><i>n</i><i>.suf</i><i>n</i></p></td><td class="cell"><p>57.80%</p></td><td class="cell"><p>34.68%</p></td><td class="cell"><p>43.35</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>W-3+3.Cho. <i>pre</i><i>n</i><i>.suf</i><i>n</i></p></td><td class="cell"><p>64.12%</p></td><td class="cell"><p>34.34%</p></td><td class="cell"><p>44.73</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>W-3+3.POS0. pre</i><i>n</i><i>.suf</i><i>n</i></p></td><td class="cell"><p>64.56%</p></td><td class="cell"><p>34.29%</p></td><td class="cell"><p>44.79</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>W-3+3. <i>POSo.Cho. pre</i><i>n</i><i>.suf</i><i>n</i></p></td><td class="cell"><p>64.07%</p></td><td class="cell"><p>34.57%</p></td><td class="cell"><p>44.91</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Features</p></td><td class="cell"><p>Precision</p></td><td class="cell"><p>Recall</p></td><td class="cell"><p><i>Fß</i>=1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>w</p></td><td class="cell"><p>57.05%</p></td><td class="cell"><p>20.62%</p></td><td class="cell"><p>30.29</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>pre</p></td><td class="cell"><p>53.65%</p></td><td class="cell"><p>23.87%</p></td><td class="cell"><p>33.04</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>suf</p></td><td class="cell"><p>47.75%</p></td><td class="cell"><p>29.19%</p></td><td class="cell"><p>36.23</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>w.pre</p></td><td class="cell"><p>63.08%</p></td><td class="cell"><p>27.56%</p></td><td class="cell"><p>38.36</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>w.suf</p></td><td class="cell"><p>60.93%</p></td><td class="cell"><p>30.76%</p></td><td class="cell"><p>40.88</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>pre.suf</p></td><td class="cell"><p>57.94%</p></td><td class="cell"><p>34.96%</p></td><td class="cell"><p>43.61</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>w.pre.suf</p></td><td class="cell"><p>64.80%</p></td><td class="cell"><p>34.34%</p></td><td class="cell"><p>44.89</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 4: The effect of training data size on the performance of the NER." class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Number of Words</p></td><td class="cell"><p>Precision</p></td><td class="cell"><p>Recall</p></td><td class="cell"><p><i>Fß</i>=1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2500</p></td><td class="cell"><p>51.37%</p></td><td class="cell"><p>9.47%</p></td><td class="cell"><p>15.99</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>5000</p></td><td class="cell"><p>64.74%</p></td><td class="cell"><p>11.93%</p></td><td class="cell"><p>20.15</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>7500</p></td><td class="cell"><p>61.32%</p></td><td class="cell"><p>13.50%</p></td><td class="cell"><p>22.13</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>10000</p></td><td class="cell"><p>66.88%</p></td><td class="cell"><p>23.31%</p></td><td class="cell"><p>34.57</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>12500</p></td><td class="cell"><p>63.42%</p></td><td class="cell"><p>27.39%</p></td><td class="cell"><p>38.26</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>15000</p></td><td class="cell"><p>63.55%</p></td><td class="cell"><p>31.26%</p></td><td class="cell"><p>41.91</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>17500</p></td><td class="cell"><p>60.58%</p></td><td class="cell"><p>30.64%</p></td><td class="cell"><p>40.70</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>20000</p></td><td class="cell"><p>58.32%</p></td><td class="cell"><p>30.03%</p></td><td class="cell"><p>39.64</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>22500</p></td><td class="cell"><p>57.72%</p></td><td class="cell"><p>29.75%</p></td><td class="cell"><p>39.26</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>25000</p></td><td class="cell"><p>59.33%</p></td><td class="cell"><p>29.92%</p></td><td class="cell"><p>39.78</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>27500</p></td><td class="cell"><p>60.91%</p></td><td class="cell"><p>30.03%</p></td><td class="cell"><p>40.23</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>30000</p></td><td class="cell"><p>62.77%</p></td><td class="cell"><p>30.42%</p></td><td class="cell"><p>40.98</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>32500</p></td><td class="cell"><p>62.66%</p></td><td class="cell"><p>30.64%</p></td><td class="cell"><p>41.16</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>35000</p></td><td class="cell"><p>62.08%</p></td><td class="cell"><p>30.81%</p></td><td class="cell"><p>41.18</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>37500</p></td><td class="cell"><p>61.02%</p></td><td class="cell"><p>30.87%</p></td><td class="cell"><p>41.00</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>40000</p></td><td class="cell"><p>61.60%</p></td><td class="cell"><p>31.09%</p></td><td class="cell"><p>41.33</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>42500</p></td><td class="cell"><p>62.12%</p></td><td class="cell"><p>32.44%</p></td><td class="cell"><p>42.62</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>45000</p></td><td class="cell"><p>62.70%</p></td><td class="cell"><p>32.77%</p></td><td class="cell"><p>43.05</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>47500</p></td><td class="cell"><p>63.20%</p></td><td class="cell"><p>32.72%</p></td><td class="cell"><p>43.12</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>50000</p></td><td class="cell"><p>64.29%</p></td><td class="cell"><p>34.29%</p></td><td class="cell"><p>44.72</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><references><p>Bogdan Babych and Anthony Hartley. 2003. Improv­ing machine translation quality with automatic named entity recognition. In <i>Proceedings of Seventh Inter­national EAMT Workshop on MT and other language technology tools, </i>Budapest, Hungary.</p><p>Daniel M. Bikel, Scott Miller, Richard Schwartz, and Ralph Weischedel. 1997. Nymble: a high-performance learning name-finder. In <i>Proceedings of the fifth conference on Applied natural language pro­cessing, </i>pages 194-201, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.</p><p>Nancy Chinchor. 1997. Muc-7 named entity task defini­tion. Technical Report Version 3.5, Science Applica­tions International Corporation, Fairfax, Virginia.</p><p>Dan Klein, Joseph Smarr, Huy Nguyen, and Christo­pher D. Manning. 2003. Named entity recognition with character-level models. In <i>Proceedings ofthe seventh conference on Natural language learning at HLT-NAACL 2003, </i>pages 180-183, Morristown, NJ, USA. Association for Computational Linguistics.</p><p>John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In <i>Proc. 18th International Conf. on Machine Learning, </i>pages 282-289. Morgan Kaufmann, San Francisco, CA.</p><p>Andrei Mikheev. 1997. Automatic rule induction for unknown-word guessing. <i>Comput. Linguist., </i>23(3):405-123.</p><page local="5" global="109"/><p>Table 3: Average Precision,Recall and <i>Fp=1 </i>measure for different language independent feature combina­tions.</p><p>Diego Molla, Menno van Zaanen, and Daniel Smith. 2006. Named entity recognition for question answer­ing. In <i>Proceedings ofAustralasian Language Tech­nology Workshop 2006, </i>Sydney, Australia.</p><p>Avinesh PVS and Karthik G. 2007. Part-of-speech tag­ging and chunking using conditional random fields and transformation based learning. In <i>In Proceedings of SPSAL-2007 Workshop.</i></p><p>Lance Ramshaw and Mitch Marcus. 1995. Text chunking using transformation-based learning. In David Yarovsky and Kenneth Church, editors, <i>Proceedings ofthe Third Workshop on Very Large Corpora, </i>pages 82-94, Somerset, New Jersey. Association for Compu­tational Linguistics.</p><p>Antonio Toral, Elisa Noguera, Fernando Llopis, and Rafael Munoz. 2005. Improving question answering using named entity recognition. In <i>Proceedings ofthe lOth NLDB congress, </i>Lecture notes in Computer Sci­ence, Alicante, Spain. Springer-Verlag.</p><page local="6" global="110"/><p>Richard Tzong-Han Tsai. 2006. A hybrid approach to biomedical named entity recognition and semantic role labeling. In <i>Proceedings of the 2006 Conference of the North American Chapter of the Association for Com­putational Linguistics on Human Language Technol­ogy, </i>pages 243-246, Morristown, NJ, USA. Associa­tion for Computational Linguistics.</p><p>Nina Wacholder, Yael Ravin, and Misook Choi. 1997. Disambiguation of proper names in text. In <i>Proceed­ings of the fifth conference on Applied natural lan­guage processing, </i>pages 202-208, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.</p><p>Hanna M. Wallach. 2004. Conditional random fields: An introduction. Technical Report MS-CIS-04-21, Uni­versity of Pennsylvania, Department of Computer and Information Science, University of Pennsylvania.</p></references></body></article>