<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="171"/><title>CRFs-Based Named Entity Recognition Incorporated with Heuristic Entity List Searching</title><pubinfo>Sixth SIGHAN Workshop on Chinese Language Processing</pubinfo><author surname="Yang" givenname="Fan"><org  name="Chinese Academy of Sciences" country="China" city="Beijing"/></author><author surname="Zhao" givenname="Jun"><org  name="Chinese Academy of Sciences" country="China" city="Beijing"/></author><author surname="Zou" givenname="Bo"><org  name="Chinese Academy of Sciences" country="China" city="Beijing"/></author></firstpageheader><frontmatter><p><b>CRFs-Based Named Entity Recognition Incorporated with Heuristic</b></p><p><b>Entity List Searching</b></p><p><b>Fan Yang</b></p><p>National Laboratory of</p><p>Pattern Recognition Institute of Automation, Chinese Academy of Sciences fyang@nlpr.ia.ac.cn</p><p><b>Jun Zhao</b></p><p>Pattern Recognition Institute of Automation, Chinese Academy of Sciences jzhao@nlpr.ia.ac.cn</p><p><b>Bo Zou</b></p><p>Pattern Recognition Institute of Automation, Chinese Academy of Sciences bzou@nlpr.ia.ac.cn</p></frontmatter><abstract>Chinese Named entity recognition is one of the most important tasks in NLP. Two kinds of Challenges we confront are how to improve the performance in one corpus and keep its performance in another different corpus. We use a combination of statistical models, i.e. a language model to recognize person names and two CRFs models to recognize Location names and Organization names respectively. We also incorporate an efficient heuristic named entity list searching process into the framework of statistical model in order to improve both the performance and the adaptability of the statistical NER system. We participate in the NER tests on open tracks of MSRA. The testing results show that our system can performs well. </abstract></header><body><section number="1" title="Introduction"><p>Named Entity Recognition (NER) is one of the most important tasks in NLP, and acts as a critical role in some language processing applications, such as Information Extraction and Integration, Text Classification etc. Many efforts have been paid to improve the performance of NER.</p><p>NER task in Chinese has some differences from in English as follows. 1) There is no space between Chinese characters, which make boundary recognition more difficult. 2) In English, a capitalized letter at the beginning position of a word implies that the word is a part of a named entity. However, this kind of characteristic does not exist in Chinese.</p><p>In the paper, we will focus on two kinds of problems. 1) How to improve the performance of Chinese NER in one corpus, which contains boosting precision rate, recall rate and F-measure rate. 2) How to enhance the adaptability of a Chinese NER system, which means that a system can get a good performance on a testing set which has many differences from the training set. To solve the first problem, we should select a good model and adjust parameters carefully. But there is no framework that can solve the second problem completely.</p><p>Our goal is to find a way to solve these two problems. We select a language model to recognize Person names, and two CRFs models are used to recognize Location and Organization separately. We also try to incorporate a large-scale named entity list into the statistical model, where a heuristic searching method is developed to match the entities in the list quickly and efficiently.</p></section><section number="2" title="Framework of NER System"><p>The Input of the system is a raw text. We will apply some pre-processing such as code transformation. Then the heuristic searching will be executed to find the appearance of the entities in the named entity list. After that, two CRFs that have been trained before will be used to recognize Location and Organization based on the result of word segmentation, and a language model will be used to find Person names. All the results will be integrated at last.</p><page local="2" global="172"/><p>Un-segmenting test</p><p>Heuristic searching</p><p>Person recognition</p><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">5</doubt><p>Word segmentation</p><p>Location recognition</p><p>Output results</p><p>Organization recognition</p><figure caption="Figure 1. System Frameworks3    System Details"></figure><subsection number="3.1" title="Using heuristic method to search entity list"><p>NER task meets many difficulties which come from the complexity of the construction of named entities. Named entities have flexible internal styles and external environments. Building a good model to describe the condition precisely will have many troubles. The statistical models we common­ly used have some shortcomings, especially when they are adapted to a corpus of new domains or styles. We try to use an improved searching method to make up the relatively poor adaptability of the statistical models. The heuristic searching method is more flexible especially in the following two aspects. 1) Abbreviation can be matched. 2) Suffix in location and organization is universal, but it should not be taken in count when we search named entities.</p><p>The framework of the algorithm can be briefly described as follows:</p><p>1) Building an inverse index using Chinese characters as key term;</p><p>2) Using the text as a query to search for entities;</p><p>3) When comes terminal condition, a heuristic function is invoked to determine whether the character sequence is an entity;</p><p>4) When comes creation condition, a heuristic function is invoked to judge whether a new entity is created;</p><p><u>5)</u> <u>The labeled sequence is output.</u>_</p><table caption="Table 1. Heuristic Searching Method"></table><p>One advantage of heuristic searching method is that the heuristic function can be set to fit a special corpus. The heuristic searching method we used in Bakeoff-4 is as follows:</p><p>• Ignoring the suffix key word in Location and Organization names. For example "r§]y57Tongfang -^-h]/Corporation" and "I^Tj/Tongfang" will get same score under this heuristic rule.</p><p>• Ignoring the Location name as a prefix in an Organization name. For example, "US/American IftM/General Motors " and "ffiM/General Motors" will get same score under this heuristic rule</p><p>• Taking Abbreviation rules in consideration. For example"!!^ /Peking ^^/University" can be abbreviated as "Jb/Bei ^/Da" rather than        /Peking" or "^^/University"</p><p>Heuristic searching method also has such advan­tages as follows:</p><p>It is easy to be expanded to a corpus of new domain or style. We only need to add the entities in the new domain into list</p><p>Searching method will improve the recall performance remarkably</p><p>But the precision will be reduced for the ambi­guities, i.e. whether a sequence that matches an entity in the list really constructs an entity in the text. We will disambiguate it using statistical models.</p></subsection><subsection number="3.2" title="Conditional Random Fields Model"><p>Conditional Random Fields (CRFs) is an undirected graphical model that encodes a conditional probability distribution using a given set of features. Currently it is widely used as a discriminate model for sequence labeling:</p><doubt alpha="34.5" length="29" tooSmall="False" monospace="0.0">P(Y |X)= ^(^Af(yC1,yc2,X))(1)</doubt><p>CRFs is considered to be a very effective model to resolve the issue of sequence labeling for the following characteristics:</p><p>Because it uses a non-greedy whole sentence joint labeling method, high accuracy rate can be guaranteed and bias labeling can be avoided.</p><p>Any types of features can be integrated in the model flexibly.</p><p>Over-fitting can be avoided to some extent by integrating a priori with training data.</p><page local="3" global="173"/><p>As a discriminate model, CRT's inherits the advantages of both Hidden Markov Model (HMM) and Maximum Entropy Markov Model (MEMM) as well.</p></subsection><subsection number="3.3" title="Person names recognize"><p>We use language model to recognize Personal names. We use character-based rather than word-based model to avoid the word segmentation errors. We construct a context model and an entity model to respectively describe external and internal features of Personal names. The details of the model are as follows:</p><p>We use a tri-gram model as the context model:</p><doubt alpha="64.3" length="14" tooSmall="False" monospace="0.0">p(wc) ^rip(wci</doubt><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">i=1</doubt><p>Entity model:</p><doubt alpha="60.0" length="10" tooSmall="False" monospace="0.0">wci-2wci-1</doubt><doubt alpha="60.0" length="5" tooSmall="False" monospace="0.0">IB1M2</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(2)</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(3)</doubt><doubt alpha="61.5" length="26" tooSmall="False" monospace="0.0">=PK, IB)*fiPKM,&gt;Vd)xPK\Ek,</doubt><p>Where <i>B </i>means the beginning of the entity, <i>M </i>means middle, <i>E </i>means end.</p><p>Some expert knowledge is employed to assist the recognition process of language model.</p><p>• A Chinese family name list (476) and a Japanese family name list (9189) are used to restrict and select the generated candidates.</p><p>• A list of commonly used character in Russian and European name.</p><p>• Constrain of name length: A Chinese name cannot contain more than 8 characters.</p></subsection><subsection number="3.4" title="Location names recognition"><p>Location names have some composition characters. 1) There may be some key words as suffix, such as: "TtJ/Shi, tt/Zheng, $J/Hu, Lll/Shan" etc. 2) Other parts of Location names are always OOV words, such as "^ftjtt/Dagang Village, ^^PlS/Fuzi Temple". So the right boundary of Location can be determined easily. The mainly problem in Location recognition is on abbreviation, such as "IfJIIt-il/JinJiLuYu" is the combination of four location abbreviations. In our system, CRFs model can be supported by the heuristic searching method because it can match the abbreviation of entity in list. Using the searching method can boost the recall rate of location recognition significantly. We construct the recognition model based on the word-segmented texts.</p><p>To construct a CRFs model, we select the following features:</p><p>• A list of key word suffix is used to trigger the recognition processing.</p><p>• Using a list of indication words to restrict the boundary.</p><p>• Heuristic searching method is used to assist Location recognition.</p><p>The features we used in CRFs model is followed:</p><p>The indication words used in Location recognition include "for-indicate" and "back-indicate" words, where "for-indicate" denotes the indicating words that occur as the left neighbor of the candidate Location named entity, while "back-indicate" denotes the indicating words that occur as the right neighbor. "for-indicate" and "back-indicate" words are got from the training corpus. We calculate the mutual information between neighbor words and location entity, and get the top N words as indication words.</p><doubt alpha="45.5" length="22" tooSmall="False" monospace="0.0">MI( x, y) = p(x,y)log-</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(4)</doubt><doubt alpha="40.0" length="10" tooSmall="False" monospace="0.0">P(. x)p(y)</doubt><p>We select 1216 for-indicate words and 1227 back-indicate words. We also get 607 key words as location name suffix.</p></subsection><subsection number="3.5" title="Organization names recognition"><p>Organization name recognition is the most difficult part in NER task. The difficulties are as follows. 1) The composition of Organization name is very complex. For example: "^^/Dalian ^H/Shide <i>M^M</i>/Group", the first words in the entity is a location name. The second is a phonetic name<page local="4" global="174"/></p><table caption="Table 2. Features used in Location recognition Statement:" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>W0</p></td><td class="cell"><p>Current Word</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>W-l, W-2 , Wl, W2</p></td><td class="cell"><p>Two words before and behind</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>W-1W0, W0W1</p></td><td class="cell"><p>Bi-gram Features</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>POS0</p></td><td class="cell"><p>POS tag</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PRE-1; PRE0, PRE1</p></td><td class="cell"><p>Pre-Position reference words</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SRE-l, SRE0, SREl</p></td><td class="cell"><p>Suf-Position reference words</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Key</p></td><td class="cell"><p>Has Key suffix</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>DIC</p></td><td class="cell"><p>In Dictionary</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>• A list of key word suffix is used to trigger the recognition processing.</p><p>• Heuristic searching method is used to assist Location recognition.</p><p>which is also an OOV word and the last one is a key word as suffix.2) The boundary of organizat­ion name is hard to be classified, and the length of organization names is dynamitic. 3) Organization names are easily confused as Location names. We must use contextual information to determine its type. 4) To recognize the abbreviation of an organization is also a difficult task. So we choose the following features to solve the above problems.</p><p>• Using indication words to define the boundary of organization.</p><p>The features we used in the CRFs model are the same as used in Location model. We use the mutual information to select 513 for-indicate words and 1195 back-indicate words from training corpus. The number of key suffix words is 3129.</p></subsection></section><section number="4" title="Experiments"><p>We participate in the SigHAN Microsoft Research Asia (MSRA) corpus in open track. The table 3 is the official result of NER by our system.</p><p>The training corpora we used comes from 1) 1998 People's Daily corpus; 2) the training corpus supplied by MSRA for SigHAN bakeoff 4. These two corpora have many difference and we focus on how to get a good performance both on training corpus and testing corpus. We select some general features and get assistance from the heuristic searching method. A good list is very important, which has been proved by the experimental data. We collect nearly 1 million personal names, 40 thousand location names and more than 300,000 organization names.</p></section><section number="5" title="Conclusion"><p>In the paper, we give a presentation to our Chinese Named Entity Recognition System. It uses a language mode to recognize personal names, and two CRFs models to find Location and organization separately. We also have a flexible heuristic searching method to match entity in named entity list with text characters sequence. Our system achieves a good result in the open NER track of MSRA corpus.</p><p><b>Acknowledgement</b></p><p>The work is supported by the National High Technology Development 863 Program of China under Grants no. 2006AA01Z144, the National Natural Science Foundation of China under Grants No. 60673042, the Natural Science Foundation of Beijing under Grants no. 4052027, 4073043.</p><table caption="Table 3. SigHAN MSRA corpus test results" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>R</p></td><td class="cell"><p>P</p></td><td class="cell"><p>F</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Person</p></td><td class="cell"><p>0.9657</p></td><td class="cell"><p>0.9574</p></td><td class="cell"><p>0.9615</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Location</p></td><td class="cell"><p>0.9593</p></td><td class="cell"><p>0.9769</p></td><td class="cell"><p>0.968</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Organization</p></td><td class="cell"><p>0.8778</p></td><td class="cell"><p>0.9338</p></td><td class="cell"><p>0.9049</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>overall</p></td><td class="cell"><p>0.9377</p></td><td class="cell"><p>0.9603</p></td><td class="cell"><p>0.9489</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><references><p>J. Lafferty, A. McCallum, and F. Pereira. 2001.</p><p><i>Conditional random fields: Probabilistic models for segmenting and labeling sequence data, </i>In Proc.</p><p>ICML 2001.</p><p>F. Sha, F. Pereira. <i>Shallow parsing with conditional random fields. </i>In Proc. NAACL 2003</p><p>HP Zhang, HK Yu, DY Xiong, Q Liu and Liu Qun.</p><p><i>HHMM-based Chinese Lexical Analyzer ICTCLAS. </i>In Proc. Second of SIGHAN Workshop on Chinese Language Processing 2003.</p><p>Youzheng Wu, Jun Zhao, Bo Xu. <i>Chinese Named Entity Recognition Model Based on Multiple Features. </i>In Proceedings of HLT/EMNLP 2005</p><p>Youzheng Wu, Jun Zhao, Bo Xu. <i>Chinese Named Entity Recognition Combining a Statistical Model with Human Knowledge. </i>In Proceedings of ACL2003 Workshop on Multilingual and Mixed-language Named Entity Recognition</p></references></body></article>