This mental article is intended in Chinese to address, natural with a unified processing and pragmatic (NLP): What approach, is ‘word’ two funda- Chinese?, questions proach and How does language a computer most identify approaches Chinese words a from automatically? following Our in is unique ap- distinguished previous by the three ∗ Natural Beijing, 100080, Language China. Computing E-mail: jfgao@microsoft.com, Group, Microsoft Research muli@microsoft.com, Asia, 5F, Sigma Center, No. 49, Zhichun Road, microsoft.com. cnhuang@msrchina.research. + The work is reported andi.wu@grapecity.com. in this article was done while the author was at Microsoft Research. His current e-mail address Submission received: 17 June 2005. 22 November 2004; revised submission received: 20 April 2005; accepted for publication: © 2006 Association for Computational Linguistics Computational Linguistics Volume 31, Number 4 components that approach are integrated word into a single model: unknown a taxonomy word of Chinese words, a unified of word to breaking 1 and We will detection, of in and a customizable display Chinese segmentation. word is describe each word challenging because it these is often turn. segmentation in Chinese. Theoretical difficult to to define constitutes tried define Chinese what a using various criteria (e.g., linguists Packard have 2000). While each of words linguistic insights into in Chinese, not those criteria pro- vides valuable “word-hood” Fortunately, may not they do consistently issue in lead us to the same conclusions. where of this words be and a serious computational lin- guistics, one the definition processes words can vary in can depend to a large degree (Sproat upon how 2002). uses and these computer applications and Shih In this article, we define We the concept of Chinese in words which from Chinese the viewpoint into develop a taxonomy words of computational linguistics. one of named following five new types: words. lexicon 2 words, These morphologically can be categorized words, the of words derived factoids, properties entities, and and processed in different five ways types in Section are 3. Two of in our have different as will computational be not important in system, described detail to these five significant types, factoids in NLP. and named entities, Chinese are word theoretical involves linguists and word mainly but two are segmentation identification. In most research of issues: word boundary disambiguation to unknown two and with the current in be separate manner. tasks are dealt using different systems, these are considered or components a However, cascaded approached we consecutive believe simultaneously. that these In two issues are we not separate unified in nature approach and are better of word-level this article, Chinese NLP present the fundamental features (corresponding a to five of words (1) word (2) morphological to the five types described (4) named earlier): breaking, (NER), and (5) new word analysis, identifica- (3) factoid tion models (NWI). detection, in This approach is entity recognition which models based on a mathematical inspired framework source–channel of linear models mixture Chinese component model There are by the models. are basically The source two types model of is component models: of sentence generation. source probability a set of a and of channel word in which word used to to estimate of a sequence of the word given types, word a channel model We is each belongs one word the generative type. For each used to estimate the likelihood a character string, incorporate the wide type. of shall show that this in to a variety linguistic knowledge and framework is flexible models enough way. statistical words. In a unified While computer words applications, we are to more unambiguous concerned with segmentation units than units are supposed to be from and to static linguistic expected application. In entities, segmentation Chinese NLP-enabled are vary may application fact, dif- ferent of word applications have For different requirements that request different nition (ASR) granularities segmentation. example, automatic accuracy, speech whereas recog- systems prefer longer “words” to achieve higher inGao et al. Chinese Word Segmentation: A Pragmatic Approach formation (Wu 2003). retrieval (IR) systems prefer shorter “words” to obtain higher recall rates Therefore, mentation we do not assume We that an application-independent instead for of universal multiple word seg- standard exists. argue the each specific application. It is existence segmenta- tion standards, for a menter segmenters. A output better solution undesirable would be to to develop a set of application-specific with is provide develop a generic seg- customizable that to that is able either to predefined or alternative To specification we implied in segmentation units according the the (TBL; application Brill 1995) data. achieve this, in Section present 6. a transformation-based learning method, to be described ive We implement Chinese word the pragmatic approach MSRSeg. to Chinese It word segmentation of two in an adapt- segmenter is called on mixture consists model components: of (1) word a generic segmenter unknown that (2) word based the and linear that can adapt to domain-specific framework breaking and a of output detection adaptors for adapting the output of (1) vocab- ularies, and set Evaluation on with to different application-specific standards. five test sets different on standards shows It that the adaptive system achieves of state-of-the-art performance sets. demonstrates the possibility single adaptive Chinese word all the test is thus of multiple a segmenter that ous The capable supporting work remainder in of this Section article 3 is applications. organized introduces as follows. Section of 2 in study. Section the taxonomy 4 Chinese presents words previ- this field. background corpora on which we our used our presents some of and de- scribes the approach is Section 5 outlines the theoretical of Chinese word unified segmenter, MSRSeg, based. of the general architecture in the of and describes each where the appro- com- ponents we Section detail, 6 presenting presents of TBL a separate an method evaluation priate. the annotated of each component standards adaptation. we While in Section in Section methods of existence training corpus, focus 7 on 5 presume the the no creating We training data in a (semi-)automatic possibilities manner, of unsupervised with minimal or of Chinese human annotation. words. Section 8 thus demonstrate the presents evaluations of the system on learning different several in the different corpora, other each corresponding to Finally, a we segmentation standard, in Section comparison 9. with state-of-the-art systems. conclude the article

Many Tseng methods (1993); of Chinese Sproat word Shih segmentation (2002); and have Sun and been Tsou proposed: (2001). reviews These methods include Wu and many roughly as either and dictionary-based approaches. or statistically-based methods, while can be classified In state-of-the-art systems in dictionary methods, use hybrid dictionary-based identified. given an One input of character most string, popular only methods words is that mum the can be the maxi- are stored matching Studies (MM), usually augmented method or with minor heuristics to include deal with Chen ambiguities (1999) in segmentation. Nie, Jin, and Hannan that use (1994). this The of variants of which these methods et al. and performance a upon coverage dictionary, thus unfortunately may depends degree the the never to large many new words appear constantly. Therefore, in addition to the dictio- be complete nary, because systems also contain special components for unknown word identification. Computational Linguistics Volume 31, Number 4 In particular, or statistical methods have been mechanism widely applied because dictionary to they use a proba- bilistic These cost-based methods scoring rather Chang drawbacks. First, than some of a these methods segment (e.g., Lin the text. and Su have 1997) three identify OOV (out-of-vocabulary) words without et al. instance, might identify identifying 1993; For one string as but fail to identify their types. name. Second, many a a unit methods do not incorporate that it is a person (1999) into current statistical segmentation. For example, Thus, Teahan et identified al. (2000) linguistic knowledge effectively not OOV and use linguistic knowledge. the words Dai et al. do implausible, any and consequently, manual are be additional checking is likely to linguistically some Third, in many needed for subsequent identification is tasks such as parsing. process from current segmenters, Jiang separate segmentation (e.g., Chen OOV 2000; Chen considered Bai a 1998). For instance, Chen (2003) 2003; OOV Wu and or more and often assumes two long and segmented into that words are He usually characters to OOV are words of different single manner word detect types in characters. then uses different components a cascaded We after the basic from that the identification segmentation. believe word of We OOV words propose should not approach be treated as a problem separate problems simultaneously. segmentation. A work a unified on this is Sproat that et al. (1996), solves along line which both weighted finite-state previous on mechanism: transducers (FSTs). Our approach mixture is is based is different models. As similarly we motivated but models based provide a more linear incorporating shall of see, the information. a flexible Many framework in types of OOV for words various that are not kinds covered in Sproat’s lexical and statistical system can from be dealt discriminant with our system. widely The linear models we used are originally derived Hart, and Stork linear 2001) functions introduced used for into pattern NLP classification Duffy recently word tasks by Collins (Duda, (2001). Other and have been include maximum of entropy Chinese segmentation, which are similar to and models, frameworks models (Xue 2003) the linear (Peng, Feng, and McCallum 2004). They and conditional identification. use a unified approach random fields OOV also to word breaking and

Inthisarticle, weuse morphologically theterm “newwords” torefertoOOVwords words. words” otherthannamed mostly domain- en- tities, factoids, (e.g., and (e.g., Links’; ‘cellular’) derived and “New time-sensitive There political, are specific terms social, approaches or cultural terms into NWI. dictionary The ‘Three is new ^ x words ‘SARS’). to from have been two first word large Fung corpora and off-line general to acquire Chien 1997; Gao segmentation 2002). starts The other (e.g., is Wu 1994; and Nie, put Jin them Hannan a 1994; before new words on-line, and i.e., to new 2003; words in et al. a Jiang sentence 2000). on These the fly two during approaches the process to detect (e.g., Chen spot Wu of word segmentation we of and There them in our complement one another, and use both system. have been quite is a few Chinese methods word proposed for the off-line approach in which the basic assumption These methods that a metrics should appear mutual frequency, use that are based They on as a stable sequence in the corpus. information, statistical features such as The term new words and their mostly variants. proper nouns require and a reasonably Unfortunately, detected words, are our of other large training corpus. words. new relatively frequent under definition the term, may not be detected. Gao et al. Chinese Word Segmentation: A Pragmatic Approach of Fewer methods Some have been proposed in for an on-line on-line NWI approach, and of that is the approaches. recent advances explore NWI the use machine focus this article. For Li (2003) learning example, et al. machines define (SVM) as a binary classification problem and use support vector features whether Chinese to combine various linguistically sequence is word. Our motivated to determine of of Li a in character an that of word that NWI is not a a stand-alone process in method an et al. our is extension integral integration part in Section We system but segmentation. 5.5. shall show experimentally the benefit of the

As ous, described and earlier, while Chinese words we more are supposed to to are concerned with be well-defined, unambigu- static linguistic different entities, This inspires segmentation units that are expected of vary Chinese among word segmenter. computer applications. the development an However, adaptive most of the previous segmenters have been The developed that segmentation. only according to a standard of our assumes a single which display knowledge, of is correct adaptive the customizable output segmenter described in Wu system, (2003), to the in best the the segmentation can be customized by users. 3 in method we will in Section 6 can be viewed as an improved The adaptation describe that the rules (or version adaptation word via TBL is method transformations) not (Gao (see Palmer al. 2004). are acquired new et [1997]; Though automatically Hockenmaier the use of TBL from application Chinese data the Brew for [1998]), none of segmentation work is and the previous aimed at standards adaptation.

The performance However, of Chinese word segmenters across is generally reported in terms of and comparison systems could be difficult for preci- sion reasons. First, recall. a is not very clearly defined. It is two the “correct” segmentation a sentence there multiple plausible word common segmentations. As that for Sproat given To et (1996), are the problem, rate of agreement Fung and between Wu (1994) two human judges procedure is shown in 80%. al. with less than deal this mentation that uses blind judges’ standards. If we suggest set k = 1, it is a called nk-blind to n if it with one of sufficient for be judges. If a seg- considered correct agrees at least the n k = measure must agree. into Therefore, multiple nk-blind gives a more n, all judges 8.2, judges. will Similarly, our Sproat representative method et al. (1996) performance multiple by taking account In Section we cross-system also uses human We do judges. not use multiple present judges. Instead, we only for comparison. measures human different consider a set of that are lexicon-independent and less ambiguous among human judges The and systems. second reason ground that cross-system many comparisons are difficult For concerns test and rules by research papers. example, some the use of different sets precision and of 98% or 99%. But they either count only the words papers report in recall rates the or unrealistically simple data with low OOV that are Recently, stored ACL-SIGHAN–sponsored dictionary use First International Chinese a Word very Segmentation rate. the Computational Linguistics Volume 31, Number 4 Bakeoff alleviated the situation to some degree (Sproat and Emerson different 2003). standard, The and Bakeoff released four data sets, train–test 8.3. We each corresponding our to a evaluate segmenter using those four data sets in consistent splits. Sections 6.2 and

This which section de Chinese fines words Chinese words three into at levels. main We begin with a taxonomy processed categorized five types according to the way by are and are in realistic systems. Second, we develop the MSR they which is a Finally, set of used fic rules to guide human annotators in standard, sentences. we speci word the we development of segmenting a gold test set and how we Chinese Chinese describe Here, to evaluate segmenters. MSR use the standard, term on “gold top of test set ” to refer the manually annotated corpus, according to the the “test corpus ” that is the raw text corpus.

The taxonomy of into Chinese words is summarized in in Table (or 1, where Chinese words, words LWs), morpho- are categorized the words five (MDWs), types: entries (FTs), a lexicon named lexicon logically derived factoids entities (NEs), and new words Table Taxonomy 1 of Chinese words used in developing MSRSeg. LW Lexical Word Y , , Ø t , Ema Pho NE www P L NW O Affixation Reduplication (Pre fix, Suffix) Splitting Head Merging Factoid + word Particle Duration Date Time Money Percent and fraction Frequency, ordinal, integer, decimal, Age, weight, rate, etc. capacity, length, area, speed, temperature, E-mail angle, etc. WWW Phone, fax, telex Named Person name Entity Organization Location name New Word name ØØ tt í p ú C , 12.2 , , , 700 l ¤ ,2 l Ì , 1.8 Gao et al. Chinese Word Segmentation: A Pragmatic Approach (NWs). These processed in five types different of ways words in our have different For functions in Chinese word NLP and are in where Figure 1(a) of is system. shown in example, a plausible the the same figure. Figure 1(b) segmentation for our sentence words processed in different is ways: the output of system, different types are For LWs, word boundaries are detected, such as Y ‘professor ’. For FTs, normalized their types of and normalized time forms are detected, e.g., 12:30 (“tim” is indicates the form the expression A ¹ A a time expression). For NEs, name. their types are detected, e.g., Ê ‘Li Junsheng ’ is a person The (e.g., five types of words MDWs cannot be de fined LWs by any on consistent classi fication criteria the relation between does not and finition depends of Chinese the lexicon words. being We used); not intend the taxonomy therefore give a clear word to give a standard de finition de of Chinese words. Instead, we do for this article on a preprocessing step where treat Chinese segmentation as in the The best segmentation words consuming applications. in most five This word units depend how they of Chinese are used the is one types of represent the five types we that appear approach. applications. We on in the that of “a pragmatic of Chinese word ” focus two tasks the approach: reasons processing title this article optimized the (Sections five types 4 5), using 6). we and of and adapting our a uni fied framework that can be jointly (Section Now, word system in Table to different LWs each the five types 1 in applications (lexicon describe out Chinese word Although detail. words): without some previous of research has 1990; the use dictionaries (e.g., suggested Sproat carrying of Sun, Shen segmentation and Tsou 1998), For we believe in that machine a dictionary is an and Shih nent many essential it compo- applications. into example, LWs much a translation system, is de- sirable to segment of a words sentence as in as possible dictionary. so that the these looked bilingual Similarly, candidate translations would to can be into up LWs a in Chinese (TTS) we also like segment pronunciations a sentence in dictionary a text-to-speech system because the stored (for instance, the our we generated dynamically 98,668 by words, character-to-sound are usually much more precise than those including 22,996 rules). Chi- In system, used a lexicon containing Figure (a) A Chinese 1 sentence. Square Slashes indicate indicate word boundaries. word boundaries. (b) An + output indicates of our morpheme word segmentation boundary. system. brackets a Computational Linguistics Volume 31, Number 4 nese characters stored dictionaries as single-character Chinese words. This and lexicon is Thus, by linguists used in a combination of several authored LWs in similar to different all those described in Microsoft applications. i.e., theory are processor. that “salient and highly relevant to Packard (2000), linguistic units are the operation of the language MDWs ” (morphologically derived First, words): MDWs Chinese words of from this one type or have more the following (called two characteristics. MDW via productive morphological can be generated process. For example, in Figure LWs stems) a process. As ’ is generated in Table from 1, a stem Ø t ‘happy’ via an 2, the ØØ tt ‘happily main AABB of redu- plication processes, shown [2003] of which there are five categories has several subcategories, as detailed in morpho- logical (see Wu each Figure 2 for a detailed description): Affixation (MP and MS): Ë (friend - plural) ‘friends’; Reduplication (MR): Ø t ‘happy’ → ØØ tt ‘happily ’; Splitting (MS) (i.e., a set of words expressions that are separate words at the syntactic where level but single word at the semantic is level): m ‘already the split by the particle ‘already’; ate ’, bi-character m ‘eat’ Merging (MM): í ‘on duty’+ í ‘off duty’ → í ‘on-off duty ’; and Head ‘out’ Particle (MHP) ‘walk (i.e., out’. expressions that are verb + comp): p ‘walk’ + ú » → p ú » The second in characteristic That is, of MDWs is that they within form stable (of corpus. the components the MDWs Chinese are strongly character sequences the while both cor- related high with co-occurrence words Chinese outside frequency), the sequence. is measured We components at shall describe in Section ends have 5.2 low correlations of the qualitatively, how how the morph-lexicon ‘stability’ a Chinese sequence morphology and to construct a FTs (factoids): for There analysis. in Table are ten 1. All categories FTs of factoid words, such as time and date expressions, Therefore, as shown normalization can of be Machines. and FTs represented as regular expressions. the detection can be achieved by Finite State NEs names, (named entities): names, NEs organization are frequently names. used or NEs One Chinese names, including per- son location and of with cannot high develop accuracy, a regular grammar with that most rejects FTs. In accepts Section 5.3, the we constructions can shall describe how we as both heuristics and we do models for NER. use statistical NWs or (new words): nor NWs are morphological OOV words that are In neither particular, recognized we focus as including words, rules. occasional words, on named entities factoids new words, derived by newly mostly low- frequency NWs, words (Wu and of Jiang 2000). coined signi Many and time-sensitive current in most segmenters simply that ficance applications. However, ignore assuming that identi they are of little those words is unidenti fied word we argue the fication errors in surrounding critical words. because can the For a single cause segmentation it is more error NLP would applications whole that require full to parsing, even critical because a single cause a sentence fail. Gao et al. Chinese Word Segmentation: A Pragmatic Approach Figure Taxonomy 2 of morphologically derived words (MDWs) in MSRSeg.

The two taxonomy employed here has been speci fied of in detail in the MSR standard. There are general guidelines for the development the standard: 1. 2. The which standard should be applicable to a wide Chinese variety of input, NLP tasks, IR, TTS, of MT. some representative examples are text ASR, and The standard should be compatible Chinese with NE existing standards, in ET/ER-99, of which representative Mainland examples (GB/T), are Taiwan the ROCLING standards UPenn ’s Treebank standard (CNS14366; the Huang standard et al. 4 1997), and the Chinese (Xia 1999), as much as possible. We [ensures] are seeking a uniformity standard that (Huang is “linguistically 1997; felicitous, Sproat computationally Shih 2002). The feasible, and data word of of Chinese of ” et al. and MSR standard consists a set speci fic rules that aims at unambiguously of sentence, given a reference lexicon. The determining the segmentation we is an iterative a procedure, in next interacting with development the standard test (which will describe the section). We the development begin with initial of a gold set an set of Computational Linguistics Volume 31, Number 4 segmentation rules, based interannotator on which flict is four human annotators (automatically), label ever (e.g., con detected we a test corpus. more resolve process it When- an mostly The is by revising standard speci fic rules). iterated the conflict is by For adding we from with the via rule for until detecting morphological MDWs: no detected. example, begin LW process, “if a character sequence is can be derived MDW a We a observe both then the sequence treated as an candidate. morphological ‘already had a meal ” then ’ are derived is from that the LW m ‘already ate’ and m process of While m MDW, ‘eat’ via the is We splitting. “MDW m a reasonable with internal m debatable. then add An We a rule: candidates of a set of complex speci is fic rules to define what structures should internal be segmented. only is. ” also add example those rules “for MDW candidates of a complex structure four type MS, we One consider sequences we of that the approach are less is than characters continue to that the standard would long. ” drawback such speci fic rules, and people would become very making complicated errors. as We add controlled currently not have a systematic solution to this. The start clerical manually. do That is, new complexity has to be total number of is manageable. all added rules are recompiled by a linguist so that the rules

Several questions had to be answered when we were developing the gold test set for evaluation. 1. 2. 3. 4. 5. How should we construct a test corpus for reliable evaluation? Does the segmentation in the gold test set depend on a particular lexicon? Should we assume a single correct segmentation for a sentence? What are the evaluation criteria? How should we perform a fair comparison across different systems using the gold test set? We 3.5. answer First, the first three questions in this we section and leave the rest for Sec- tion proximately to conduct million a Chinese reliable evaluation, select a test domains distributions characters that have been in Table proofread corpus that contains ap- of half a The 2. The and balanced in terms and styles. are shown MSR gold test The to the in standard Table 3. Some via the iterative set is developed procedure by annotating in Section the test 3.2. corpus according of described in Figure 3, statistics the shown and are notation shown is presented in Table fragments As gold test set in are of many Section The 3.1, we the believe that the lexicon is 1. discussed which is of a critical component applications. segmentation of the gold test set depends upon in Microsoft a refer- ence lexicon, including the a Chinese combination input several lexicons (Chu MSR-NLP system Chinese (Gao that text et al. 2002), are used applications, 2001), The TTS 98,668 2003), and the parser morph-lexicon (Wu ASR (Chang and Jiang 2000). et al. et al. 50,963 lexicon consists of MDWs. entries. We will We also developed morph-lexicon a was , which contains in Section high-frequency 5.2. describe how the constructed Regarding the third question, Chinese though it is we common only that there single are multiple plausible segmentations for for a given The sentence, is simplicity. keep The a gold is segmentation for each sentence two reasons. first second due to the fact Gao et al. Chinese Word Segmentation: A Pragmatic Approach Table 2 Domain/style distribution in the MSR test corpus. Domain/Style (MB) Descriptive writing Expository writing Narrative writing Practical writing 5 Spoken Total (%) Culture Economic Literature Military Politics Science Society & Tech. Sport Computer Law Total (%) Table Words 3 in the MSR gold test set. Word type N LW MDW FT NE: Person Location name Organization name name that we currently do the applications. In effective particular, we using segmentations above-mentioned into words in segment When each sentence as multiple much as possible that are stored we the reference one lexicon. there are segmentations words. for a sentence, keep the that contains the fewest number of It We have approximately also manually 40 developed million Chinese a training set according from to the MSR standard. contains newspapers, novels, and magazines. In characters our various 90% of domains of text such is as model remaining experiments, 10% is the training set used for parameter estimation, and the a held-out set for tuning.

As of mentioned in Section (1) 1, MSRSeg is designed as an adaptive two generic segmenter that adapt to different segmenter that consists components: (2) of output a ularies, from can domain vocab- and a set adaptors, learned application data, for adapting to different application-speci fic standards. not know any way of multiple in Computational Linguistics Volume 31, Number 4 Figure 3 Fragments of the MSR gold test set. Therefore, in standard, we and evaluate MSRSeg using five corpora, each train –test splits, as shown in Table corresponding 4. MSR is to a differ- ent consistent Chinese and Segmentation the other four Bakeoff are standards used in for SIGHAN described previous sections, Word (or Bakeoff (Sproat ’s First In- ternational Emerson 2003). not In the Bakeoff in corpora, OOV is de fined brevity) as the set of words in and occurring the test corpus In we the training corpus. we experiments, predefined always consider the following which adaptation a standard according to we paradigm. Suppose of have general We word segmenter. Whenever create a we large amount training data. any then develop we a generic output of deploy the segmenter for application, customize the the segmenter to fic standard that be partially acquired from according an application-speci of (called can a given small amount The MSR application data standard in adaptation Section 3.2 data). on which described is used as the general The standard four Bakeoff in our experiments, the generic segmenter standard. We notice in Table into has which been are as 4 we developed. standards used specific standards wish (i.e., to adapt the general four Bakeoff that much the adaptation data sets MSR training corpora is smaller than the training set. Thus, for the standards) are otherwise of good simulation of the experimental In setting a article, we the adaptation shall by default report results on paradigm the the MSR described above. rest the data set unless stated.

As defined described independently earlier, we of argue that Chinese and words (or segmentation more units) cannot (i.e., be the applications, hence a flexible system an Table Standards 4 and corpora. Corpora (standards) Abbrev. # Training word # Test word OOV Beijing MSR standard U. Penn University Hong Kong Chinese City U. Treebank Academia Sinica MSR PK HK CTB AS .002 .181 .069 .071 .021 Gao et al. Chinese Word Segmentation: A Pragmatic Approach adaptive segmenter of such as MSRSeg) should objective and be adopted. rigorous However, of we are faced with the challenge In performing an of NLP is evaluation with such a system. general, the evaluation In systems we concerned MSRSeg is both the criteria the article, system in and standard data First, sets. this argue word-internal (e.g., provides that not of only word a better two regards. the generic segmenter Section tree structures MDWs, segmentation FTs, and NEs, but also in structures 6) the Ideally, as will be described provides of that cover all possible units segmentations. a where such a segmenter it superset needs. segmentation Second, output of each different MSRSeg application can find the subset (i.e., the units) adaptors Therefore, from can automatically pick different subsets segmentation for the superset an according to different applications. there is are two how criteria evaluating is. adaptive The segmenter: how the adaptation real evaluation will complete superset and effective (i.e., the sets texts used by different require some ever, application data not segmented data yet, and no other applications). How- such application is are no way available our system has undergone such evaluation, The so there methodology to we compare in system against is others in one adopted and this article a simulation. this fashion. evaluation the we On hand, developed a generic standard a corresponding many gold test set that simulates We the generic on superset that attempts to cover of as applications will data set the completeness the generic segmenter. as possible. other then evaluate we the we On the hand, show that can of which effectively adapt the generic segmenter to the four The different bakeoff measures data sets, is we each of measured use in simulates an application subset. evaluation MSRSeg through this study multiple are summarized in Table (P/R) 5. The performance F-measures words. (de fined as 2PR/(P+R)), of OOV each one word precision and for type. Riv –recall is is pairs, in-vocabulary Roov words. They to measure the recall of ing unknown ’s performance respectively. in the recall We ambiguities in word are used words, resolving segmentation and the segmenter proposed the statistical signi ficance of detect- Sproat also Emerson test Riv, by of OAS and (overlap (2003). results, using In the criterion addition to the errors number measure ambiguity string) and CAS (com- bination ambiguity string) in word are used to in more the segmenter Liang (1987) ’s performance fines OAS of resolving CAS ambiguities segmentation detail. de and as follows. Definition A 1 mented character into string words ABC is called AB/C an overlap or A/BC ambiguity (not string (OAS) on if it can be seg- two either as both), depending context. Table Evaluation 5 measures for Chinese word segmenter. Measures Remarks P/R/F Roov Riv # # OAS CAS errors errors Significant test Multiple P/R/F pairs, of NER each for one for type cross-system of words (i.e., LW, MDW, FT, NE); Test are Similar Test performance of used comparison the cross-bracketing, of detecting unknown the resolving for ambiguities words to performance cross-system in word segmentation Test See on set of 70 used high-frequency CASs in our comparison Sproat a and Emerson (2003). study Computational Linguistics Volume 31, Number 4 Definition A 2 character words. string AB is called a combination ambiguity string (CAS) if A, B, and AB are in Liang Chinese (1987) reports and that the relative frequency of CASs of is OASs text, relative frequency 12 times is 1.2 per 100 characters However, the most above single de finition, lower than in the relative Chinese frequency words of that CASs of OAS. according to the can be much higher because characters words CASs. can be by Consider word two-character ‘altitude’. Though can be However, themselves, this is not and as a result, almost all desirable. the Ø ¦ / both never Ø ‘high’ occurs and in reality. ¦ ‘degree’ problem, segmentation Ø ¦ almost To remedy are words by themselves, Sun and the Tsou (2001) finition this revise the de as follows: Definition A 3 AB character words, string AB is is called a one combination ambiguity which string (CAS) if A, B, are and least context under the segmentation A/B and there at syntactically. is plausible both semantically and Though the revision clari of fies the definition in principle, it where requires an a judgment of the syntactic and semantic sense different the segmentation —a task CAS easily in pilot among study. As will human annotators. Therefore, we agreement only cannot be reached measure is be described in Section 7, the number use of CAS the errors a estimated CASs. by counting the wrong segmentations of the prede fined 70 high- frequency While all the in measures cross-system in Table 5 can be used For in evaluating since MSRSeg, is used comparisons. example, the MSR most of them cannot be based our on reference other lexicon, some of the measures different are meaningless So in when gold test set a we compare system we to segmenters only P/R/F that use of NER lexicons. number of OAS comparing different (i.e., crossing systems, consider the measures and the errors single brackets), unambiguous because answer. these are lexicon-independent and there is always a

This of MSRSeg. section provides We some theoretical in Section background 4.1 Chinese on the word basis of the development work first present 4.2, source source models models of Chinese a segmentation uses –channel sentence mixture generation. Then, in frame- that we models in which wide Sec- tion variety These of generalize –channel models knowledge and statistical two models as linear modeling can be tools: incorporated (1) n-gram in a way. linguistic a uni fied models (LMs; Chen are constructed Goodman 1999), via and basic (2) finite (FSA; Roche language Schabes where 1997). More and fically, the LMs we state used are bigram and automata trigram backoff mod- and speci els, with the parameters are method, estimated using Goodman, and Miao (2001). called modi maximum-likelihood estimation (MLE) in Gao, a particular smoothing LMs fied to absolute discounting, information described (2) of word or are used character FTs, (3) sequence. FSAs capture statistical such as the likelihood are used to NE represent (1) the lexicon, the rules for detecting and the rules for generating candidates. Gao et al. Chinese Word Segmentation: A Pragmatic Approach

The task words of of MSRSeg different is to detect not only processed word boundaries in but Figure 1. also word Therefore, types so that Chinese word types can in be Table 1, we as shown fine Chinese following to a word class as a group of the taxonomy de are be generated according to the same distribution (or words that supposed manner) processed in the same as follows: 1. 2. 3. 4. 5. Each LW is de fined as a class; Each MDW is de fined as a class; Each type of FT (e.g., time expressions) is de fined as a class; Each type of NE (e.g., person names) is de fined as a class; and All NWs belong to a class. Notice floor-value that both words the LW and MDWclasses not in are open sets and In that we need word those that are stored the lexicons. particular, we to assign de fine a to MDWs follows. whose as One class is used to represent all unknown LWs and six unknown all unknown classes MDWs, one type for cannot be of detected. The other MM, MHP. The probabilities of types listed in five Table classes 1, i.e., are represent each the five MP/MS, used to MR, MS, unknown Good-Turing and method. these unknown word classes are estimated using Let the w = ww 1 2 ... A w n be a word class job is sequence, and most s be a Chinese word sentence that is a character sequence. segmenter into ’s w among all possible candidates which to choose the likely class sequence ∗ s could have been segmented: w ∗ = argmax P( w | s ) = argmax P( w )P( s | w ) w ∈ GEN ( s ) w ∈ GEN ( s ) (1) where Equation GEN ( s (1) ) denotes is the candidate of set given s . The models the basic form Chinese source –channel models is for Chinese follows: word segmen- tation. assume that of a (i.e., sentence word s generated ) to as probability sequence concepts P( classes w be output, First, a person chooses a to distribution of ); according the w then the person attempts probability to express cept sequence characters, according to the distribution each con- P( by ). choosing a s | w The source model –channel models probability also can of be word interpreted in another It indicates, way: P( given w ) is a stochastic how estimating word the occurs. For class sequence. a context, to occur likely a title class ‘professor example, ’. Consequently, person names . P( as ) is Y model how we are more likely P( ) before is model a such also refer to w as a context given s | w word a generative For estimating likely a character is string more generated to a name class. example, ‘Li Junsheng the character string N Ê likely common name in be China a person while Ì is than not. Ì So Ê P( | ) is ’ because N is a model. In family our we only one model (i.e., s w also referred to model) as class of system, models of use different context of which is a trigram class for one language and a set in Table 6. types, each class of words, as shown Computational Linguistics Volume 31, Number 4 Table 6 Context model, word classes, class models, and feature functions. Word class Model Feature functions, f( s , w ) Context model Word class trigram, P( w ) log(P( w )) LW MDW NE FT NW Lexicon TRIE Morph-lexicon TRIE Character/word one P( NE), bigram, s | where each for type, ′ s ′ FSA, of , and one forms an isa substring s NE. SVM each for type classi fier Number Number of of LW in MDW in w . Σ s A log(P( s | NE)), w . ′ ′ ∈ NE where of A is the set of in substrings . that are a particular type Number w Score of of FT SVM (of a particular fier. type) in w . classi It (e.g., NE should models be noted n-gram that different models class models on are constructed whereas in FT models different ways are trained model probabilities and have binary values). The corpora, (some dynamic not value ranges of use de- rivation rules different can different probabilities but scores) class inappropriate to be so models through are multiplication in Equation that it is (1). One way combine all simple to score is to introduce as (i.e., P( model) balance model these weight quantities for ) In our a weights λ to adjust optimized the class model each class model channel . score minimize P( s | w ) to s | w λ number models, of word experiments, these in Section errors on are 4.2. It training is worth data noticing under so as to the framework of the segmentation lin- ear models as described Section are rationale behind our Linear models e.g., that the system, the decoding process the source –channel 5.6 described in on optimization follows the framework. of model weights. are just another representation based the algorithm class

The framework of linear models introduced pattern fication (Duda, is derived Hart, from linear used classi and Stork discriminant 2001) functions widely for models into NLP Berger, tasks by Della Collins Pietra, and Duffy Della (2001). It is and has been and Pietra (1996), also related to (log-)linear recently Peng, described Feng, and in McCallum (2004). Xue (2003); Och (2003), and We use the following notation in the rest of the article. Training data are we a set of example input/output , pairs. R , for In Chinese 1 M, word segmentation, is input have Chinese training samples { s i w i } i = and R ... s sequence each w is where each i an (i.e., word character of . i the reference segmentation class sequence) s i We assume a set of D + 1 features ( that D 1 , where s , w ( ) f d , to ( s , w ),for d = map ( ) real values. ∈ 0 ( , ),f Using 0 ... D. The features in Table , w ) ( ) is f s w = s w and 1 ( is s , vector have + { f w ), fined ... ,f notation, are arbitrary we functions f s 6, D ( s , w ) } .As shown probability f 0 w of called the model base feature (i.e., word de as d , ),for d 1 D, the context class trigram model). the logarithm f ( s w = ... are defined for D word classes, respectively (i.e., Gao et al. Chinese Word Segmentation: A Pragmatic Approach they are basically of derived from of class models). Their probabilities of values are either the summation probabilistic the logarithm models, fined on or the heuristically. the only , will Here, corresponding assigned we omit for those features are s and denote them as f( w ). that de w Finally, one the parameters of the model model function, λ weights, in =( λ 0 , are λ Section , a 4.1. ... , vector D The ). of λ 1 , D ... , + 1 parameters, λ 1 λ λ D are in each for feature of fact class described written likelihood score a word class sequence w can be as D Score( w , s , λ ) = λ f ( w , s ) = ∑ λ dd f ( w , s ) · d = 0 (2) We see described in that Equation Section 4.1 (2) is by introducing yet another representation class weights (i.e., of the source adjust P( –channel | w )toP( models s s | w ) λ logarithm of probabilities. The of Equation (1) ) and taking the all decision rule can then be rewritten as w ∗ = argmax Score( w , s , λ ) · w ∈ GEN ( s ) (3) In what follows, an iterative we will procedure describe the of way of estimating λ under of the in framework direction of gradient minimizes descent: errors with adjusting the to parameters We λ and respect a loss function. will the that in loss the segmentation turn function the optimization present the algorithm. 4.2.1 Loss Function. it with Assume that we can measure the R number an of error segmentation errors w comparing in reference segmentation The w using function Er( in (i.e., by a our w R distance, over training criterion that directly minimizes , w ) editing errors case). is the segmentation the training data λ ∗ argmin ∑ Er( w R = i , w ( s i , λ )), λ i = 1 ... M (4) where . Equation w ( s i , λ ) is (4) the is segmentation determined minimum by Equation (3), w referred to the sample risk (MSR; where Gao it is denoted as ∗ hereafter. Notice approximately without as criterion on the “true” distribution This is of et al. 2005) that knowing the data, the best λ can of be chosen minimization based (ERM; training Vapnik samples. 1998): If known as were the principle empirical risk MSR it would to Bayes the (minimal the criterion, converge a risk segmenter trained using However, error exactly Er(.) as is the training size goes to in finity. performance rate) poor a piecewise optimization constant any function of the model numerical parameter λ , and thus For a candidate for by simple gradient-type Er( ) is search. example, with the gradient to cannot , be computed many explicitly Therefore, respect an λ and there are local minima because on the error error . not differ- entiable (5), we ) is defined loss function, in Equation minimum (2), where squared (MSE) surface. where use Score( alternative in equation . s has been suppressed Computational Linguistics Volume 31, Number 4 for simplicity. MSELoss( λ λ and ) is simply the of squared difference incorrect one, between the over score of the correct segmentation the score the summing all training samples. MSELoss( λ ) ∑ (Score( w R , λ ) Score( w , λ )) 2 = i − i · i = 1 ... M (5) It is useful imations maximum to note that the MSE solution, under The certain of conditions, leads to approximation approx- to form of likelihood solution. the discriminant quality functions (e.g., the Equation (2)). Due depends upon the properties, linear literature, and MSE to its ap- pealing theoretical many the solution criterion procedures has received considerable (Duda, Hart, attention and in the 2001). there are available Stork 4.2.2 Optimization an unthresholded Algorithm. This perceptron, section discusses the in delta rule, Mitchell a (1997). training The algorithm for in its is following the description delta rule component form λ d = λ d − η G( λ d ), (6) where η is the step loss size, and of G is the gradient with of MSELoss. 6 (7) to G can be estimated by differentiating the function equation respect λ d as G( λ d ) = ∂ MSELoss( λ ) [BAR] ∂ λ d = ∑ (Score( w R i , λ ) − Score( w i , λ ))(f d ( w R i ) − f d ( w i )) · (7) i = 1 ... M However, word the objective function many local of Equation minima, (5) in the context of our task (i.e., Chinese finding segmentation) has minimum. We and thus gradient descent cannot guaran- tee Whereas the global therefore use a stochastic approximation to gradient descent. over the gradient descent in Equation computes (7), parameter updates samples the stochastic approximation after summing all training parameters incrementally, as shown in following calculation of the error method updates Equation the (8). for each indi- vidual training sample, as shown G( λ d ) (Score( w R i , λ ) − Score( w i , λ ))(f ( w R = d i ) − f d ( w i )) (8) The stochastic MSELoss approximation ( ) defined method individual can be viewed as optimizing a distinct loss function i λ for each training sample i as follows MSELoss ( λ ) (Score( w R , λ ) Score( w , λ )) 2 i = i − i · (9) Gao et al. Chinese Word Segmentation: A Pragmatic Approach The T optimization over algorithm we used All in our experiments is initially shown to in passes training set. parameters are set be 1. Figure The 4. It takes model the in simple λ does change during training. Parameters Class model context parameter 0 parameters are updated a to additive MSELoss ( fashion: ). are altered according to the gradient with respect In our i λ rors,so implementation, Score( ,s , R ) we Score( approximate ,s , ) 0, where w R as the w in GEN ( holds s ) with only the when fewest R er- λ w λ w the equality w = That The is, the model are updated when the sentence is wrongly w. parameters increases for word whose segmented. update rule , (i.e., the parameter values R value f( s , w ) classes “underestimated feature is models )), ” expected less mated” s w decreases the parameter R)). values whose than Empirically, models observed were f( and were feature value (i.e., f( iterated s , ) is larger than f( s , w the sequence of “overesti- when w over provides approximation these up- dates, all Although method gradient with training samples, to respect to the original a reasonable descending the optimal loss function of modeling cannot guarantee a globally solution, it Equation is (5). our this because of its ef and because it chosen for ficiency achieved the best results in our experiments. The difference algorithm is is similar to instead of the perceptron algorithm of Equation described in the rule (8) Collins (as (2002). key shown in The that, using delta 1 line 5 of Figure 4), Collins (2002) updates parameters using the rule: λ t + ( Our pilot d λ t d + f d ( w R ). our i ) f d w i study shows that algorithm achieves slightly better results. 4.3 Discussions on Robustness The But training methods described in Section The 4.2 aim at issue minimizing errors in test be different. robustness how well a training set. imal error sets rate in can set in concerns the test set. According to Dudewicz the min- and Mishra (1988), the training and the MSE preserves function from in general is not very robust because One of many possible those training samples far away from it is not bounded boundary. can be contaminated for improving robustness is the decision introduce a margin in training procedure solutions of Figure 4. The the difference the its margin) basic idea is to to the score (or score (i between , a ; correct GEN ( ), R (i.e., enlarge segmentation ). According w R incorrect ) and competing segmentations e w w s w = Equation if (8), is perceptron . . training correctly. algorithm The robustness of Figure 4 does not w to the improved adjust parameters the sentence segmented margin could be and if to the between the segmentation the top we con- tinued enlarge score if input correct even sentence had been correctly segmented, until com- peting candidate the the Figure The perceptron 4 training algorithm for Chinese word segmentation. Computational Linguistics Volume 31, Number 4 margin has exceeded a preset threshold. More speci fically, we can modify Equation (8) as follows G( λ d ) (Score( w R i , λ ) − Score( w i , λ ) − δ )(f ( w R = d i ) − f d ( w i )), (10) where δ is the desired of margin that can either be (Su an absolute Lee value 1994). or The a quantity modi propor- tional to the is score the to correct segmentation and margins proposed fied We perceptron algorithm with by Krauth training algorithm Mèzard similar (1987). the the of the algorithm future work. and Readers leave evaluation margin-based can to Duda, Hart, and Stork (2001) to also refer for a detailed description of learning algorithms.

MSRSeg We consists of two components: in a generic segmenter and and a set of output the component this leave the to adaptors. The describe first section in 4. It has been on basis of second Section generic developed the the mathematical 6. segmenter Section of in Figure 5. models described consists several components, as shown 1. 2. Sentence Segmenter: . ,?and!. To detect sentence boundaries using punctuation clues such as Word Candidate and Generator: in Given word an input Each string s , to generate its candidate P( ), where is all word candidates word store them model a lattice. log assigned is to class and the class score, e.g., s ′ of . | w s ’ any substring s Figure Overall 5 architecture of MSRSeg. Gao et al. Chinese Word Segmentation: A Pragmatic Approach 3. Decoder: To select *) the best from (or the N best) word segmentation Equations (4) (i.e., and word class sequence w Viterbi (or A*) the lattice according to (5), using the search algorithm. 4. Wrapper: To output MDW segmentation FT of results using some normalized prede fined canonical forms, e.g., Section 3.1. and are their form, as described in Here, we will (and describe candidate morph-lexicon) generators TRIEs for different LW (or word MDW) classes. They (2) are (1) the lexicon models to NE to generate (3) finite-state candidates, (FSA) the FT candidates, the NW automaton to NE class generate (4) generate candidates, and the classi fier to generate candidates.

Computational Linguistics Volume 31, Number 4 and construct the morph-lexicon, The where each entry morph-lexicon is tagged with 50,963 its morphological pattern and stem(s). resulting contains MDWs.

We consider organization four types names of named (ONs), and entities: person names of (PNs), foreign location names names (FNs). (LNs), in transliterations number principle be for a named more entity of one or more Because we types, in any character string can of order in two effective search, for generate named to limit the given candidates input a First, we entity candidates (which an string steps: and each type, FSAs) use to a set of constraints only “most are compiled by linguists Second, are of represented as generate those model likely probability. ” candidates. Class models each fined the generated candidates on are de MLE, as generative models is as- signed a class estimated corresponding named lists with that are their in Section entity 4.1.1. using We will together a backoff smoothing schema, as models described The Chinese describe model brie is modi fly the con- straints of and the class in Sproat here. (1996). Other person-name NE models novel, a fied version some that described with et Chinese al. model. are though they share similarities the person-name 5.3.1 Chinese Person assume F Chinese Names. PN There of are two main name constraints. that a family F (1) PN and given name patterns: consists a G , We + G . Both F G one or a (2) Family name and PN are two characters F long. list: (which We is of the only pattern and consider 373 candidates in our that begin with an stored in the family name list contains ity P( Given entries PN) is PN system). a candidate, which is a character model string s , the class s by character bigram follows: (1) model probabil- | name computed , with a probability P( F); as (2) generate probability P( P( s F | and generate the given name the family substring , with s F the sub-string name, s G with the probability P( s G | G)(or ,G2). For s G1 | G1)); (3) generate the ity of the it s G2 is | s G1 string given PN would example, the generative the that be estimated P( probabil- second given N Ê a as N Ê | PN) = P( N | F)P( Ê | G1)P( | Ê , G2). 5.3.2 LN Location is Names. Unlike given PNs, (which there are is no patterns 10 for LNs. We assume (1) is less than in LN characters (which long), if one that of an candidate generated is s 30,000 the following LNs); (2) conditions in satis fied: in 120-entry s an LN entry the The keyword P( LN) is a (e.g., list ends keyword list ‘city’). contains s a 9 probability ‘river’. ³ ‘Shamir s river | probability ’. It computed is an LN by a character bigram candidate because it model. ends in an LN Consider a string s ³ The P( of the string, P( given P( that it P( is an LN, /LN | = | < > s | | s ³ | P( would keyword generative LN) P( LN ) ) ) ) < of /LN be estimated where LN as s ³ respectively. > are symbols denoting the beginning and the end an LN, > | ³ ), < > and < 5.3.3 ONs Organization nested Names. named ONs are more For dif ficult to identify ON than PNs and LNs because China are Corporation usually an entities. LN ‘China’. example, the - ýý E * z l ø ‘Air ’ contains - ý Gao et al. Chinese Word Segmentation: A Pragmatic Approach Like the identi fication (less of LNs, 15 an ON candidate if it is only generated in given character characters long), keyword in a string 1,355-entry ON s than (e.g., ’). ends To a a probability of keyword nested list ON, we l ø ‘corporation C: P( In introduce word estimate the a segmentations P( , the ON) class P( ON)P( recovers P( of generative ON class model ,ON). Since s | ON) over s, w, as hidden variables. sible s ON) principle, s w | =Σ w | s | w P( all pos- | =Σ w w s | w ,ON) = P( approximated s | w ), we have P( single s | ON) =Σ a of w P( w | ON)P( terms P( s | w ). We ∗ | ON)P( then s | w ∗ ), assume w where that the w ∗ probable by pair discovered That is, we is sum is word Equation (5). the most our class find segmentation *, by to the –channel models also estimated on use system w but source are the ON list. Consider is the earlier LN, example. Assuming probability P( that ON) w * = an LN/ would ý E / * z / l ø , where - ý tagged model as as: P( the s | P( ýý E * z where > LN) is | LN) P( ø | ON) ≈ P(LN/ be estimated - | ON - )P( * model z | ý )P( ý E / using l probability E l ø of | * z )P( * z / a word class bigram LN)= P(LN < ý E < /ON l ø | ON) P( ý | LN, LN), P( > | l - | - ý ON, ON /ON | the class - ý given that is an respectively. < > and < > are symbols denoting the beginning and the end of it ø ) ý and an 5.3.4 Transliterations of Foreign Names. Chinese As described in Sproat whose et al. (1996), FNs pronunciation are usually mimics transliterated source using original pronunciation pronunciation character is of strings the the unlimited, name. Since sequential language FNs can be of of any and effectively the recognition names length their can be tricky. Fortunately, in only a few hundred Chinese such there are characters that are particularly Therefore, common in FN transliterations. an candidate would name be generated given (which s if it contains 618 Chinese only char- acters stored Notice The in our probability a transliterated P( FN FN) is character list contains characters). s | estimated PN, using LN, or a character an FN can be a named an an ON, bigram model. that one category, Then, system an depending on the context. given are in candidate, three with entity the class probabilities candidates, each P( s | PN) for generated the lattice, = P( s | LN) = P( s | ON) = P( s | FN). where In other words, model we is postpone the determination of its type to the decoding phase the context used. 5.3.5 Abbreviations. NE For the sake within of completeness, our we This describe is ongoing below the basic ideas for we tackling not abbreviations Gao results more yet. Readers framework. where can marginal refer improvements to Zhu research, et al. (2003) and Sun, do Zhou, have conclusive (2003) and and for details, have been reported. For different of Chinese types of PNs NEs, different LNs strategies single-character can be NEs. adopted. The PN We and LN find models that most abbreviations and are described previously in cannot way handle from them very of multicharacter well because NEs, (1) single-character of different from that and (2) NEs are generated a NEs is different multicharacter ones. For the context single-character ‘China-Russia NEs ’. But appear adjacent to one another, example, such as - and Ä in single- character usually is not for multicharacter NEs. - Ä 8 noted We trade thus SCPN fine this the and SCLN, single-character respectively. PNs case de We and LNs as two separate word SCPN assume that a character is classes, de- by (or SCLN) only when it is included in fined SCPN (or SCLN) a candidate which of a prede list, Computational Linguistics Volume 31, Number 4 contains models 151 (or 177) characters. The class model probabilities are assigned by unigram as P(s | SCPN) C(s) ,P(s | SCLN) C(s) = [BAR] C(s ) = [BAR] ∑ ∑ C(s ) , 1 N i 1 N i i = i = ... ... (11) where of C(s) is the SCPN (or count of the SCLN) SCPN In (or SCLN) model, s in an annotated we differentiate training set and N isthe size (or the LN) SCPN (or list. SCLN). the Therefore, context SCPN also SCLN between PN in and and should be labeled explicitly the It training is ONAs much more data. dif ficult to multiple-character detect abbreviations ON usually strings and of ONs (denoted generated from by ONA) are can be ONA of abbreviation of ' f ‘Peking University ' f ’is ‘Tsinghua their original because arbitrarily. while For example, the ' . We University ’ is the abbreviation is only ¬ in if following generated given a character string s (fewer assume than 6 that an candidate of (2) conditions are satis fied: (1) An ON characters long), both the and from ON has been detected the same fined document, in Table 7. Since s can is be no derived the for ONA using a generative pattern we de to there training Consider a function . The probability ONA data the score assign each of candidate a would approximated ONA) P( the string, ON) P( given class model class model, construct a ' generative it is score. string where P( , P( an ONA, be as ON) is defined ' | ≈ ¬ ' (0.8 f | in our ' | ¬ ' more from ' f as a constant experiments) if f , ON), ' | ¬ ON is ¬ ' f using any generative pattern of Figure 7; 0, otherwise. ' can If be derived than one ONA detected (e.g., previously in the ‘North same document University and ON derive is the into candidate We notice ¹ ' f ’), only can be used to account. that ONAs and ONs occur in the closest So we taken not differentiate also in model. similar contexts. do them the context 5.4 Factoids The types of of factoids we handled in MSRSeg are of shown in Table 1 of The Section 3.1. For each type factoid, generate from into a grammar single FSA. regular Given an expressions. expressions are compiled output input ten regular then FT a when matches string one of s, we scan it left to right, and a candidate a substring the ten Table Zhou Generative 7 Gao patterns 2003). of ONA, where s ij denotes the j-th character of the i-th word of ON (Sun, and Condition (ON) Generated patterns (ONA) Examples ON = w 1 ... w N (N ≥ 2) ON = ON ww 1 2 ,and ,and w 1 is not = 1 2 3 w 1 is not an an LN www LN ON = www 1 2 3 ,and w 1 is an LN ONA = ONA s 11 s 21 ,or = ONA s 11 s 21 s 31 ,or = ONA s 11 s 21 ... s N1 = ONA w 1 = ONA w 1 ,or = ONA ww 1 2 = w 2 ¬ / / ® 5 / / ' → ® ... ¬ ® 5 ' → ® ' / / ' / 5 / 5 ¬ ý ù ù / → / l → ù / l → ù / 5 → ý Gao et al. Chinese Word Segmentation: A Pragmatic Approach regular other expressions. of We also remove those FT Consider candidates that in are Figure substrings 1; only of any candidates is the same type. FT the (a example ¹ candidate not the string A A or accepted ). as a time expression), the substrings (e.g., A ¹ The A ¹ use of A on FSA is motivated internal by the properties fact and that the without detection on of most FTs is This based exclusively in their of justi fied by experiments. only FSA is As shown in relying with Table of 8, context. the overall can be principle FT performance detection information using of FT is comparable (i.e., in that MSRSeg, using FT simultaneously with other FSA MSRSeg where the contextual the considered words). are used If we as feature functions, and the carefully, we are detected of information (in MSRSeg) read the results can lower see that the use small context difference. achieves a higher precision but a recall —a but signi ficant

New words or in this nor section refer to OOV words morphological that are neither These words recognized and/or Links ’, are mostly as named entities factoids time-sensitive, derived by new not such as ‘Three rules. of ^ x ‘SARS’. domain- specific words It is The identi on studied extensively of word before. an important fi- cation issue such that substantial impact has been approximately has 30% of OOV words the performance in SIGHAN PK segmentation. in For Table ex- ample, new words of There the work on ’s Chinese corpus an has been manner previous detecting new words 4 are from this type. a large corpus However, in our off-line approach is and updating in able to detect new the words dictionary on-line, before i.e., to word segmentation. new words sentence on fly during the process of word segmentation, where spot widely a the used statistical features such as mutual information or term frequency are not 5.5.1 Class Model. We use a classi fier (SVM light number of [Joachims 2002] estimate two which adjacent characters forming four a new in our to likelihood we word. experiments) the of with motivated Of the great features experimented, They are due to their effectiveness and linguistically features chosen are word probability (IWP), availability anti-word pair (AWP), for on-line detection. independent word formation analogy Table FT 8 10 detection of results on the MSR gold in Table test set. 1, which The amount ‘All’ column 6630 shows the results of in detecting types described to factoids, shown Table 3. all factoids, as as [BAR] All (P/R) dat (P/R) tim (P/R) mea (P/R) mon (P/R) MSRSeg FSA 0.908/0.896 0.906/0.905 0.980/0.851 0.967/0.870 0.973/0.967 0.967/0.967 0.913/0.898 0.911/0.919 0.994/0.975 0.991/0.975 Computational Linguistics Volume 31, Number 4 (WFA), In and Section morphological 5.5.2, we productivity (MP). way We now describe each is the the training data (new feature word in turn. shall describe list) for the classifier IWP created by sampling. (independent word probability) independent is a words real-valued or feature. Most of Chinese char- acters can words, or be used either The IWP of as single is component of parts multicharacter both. independent word in a character (Wu the Jiang likelihood 2000): this character to appear as an texts and IWP(x) = C(x,W), [BAR] C(x) (12) , MP(x) n N(x) 1 (x) = [BAR] . (13) MP is strongly related to the found Good-Turing in estimate. Here, number N is of the number of of tokens nouns of a particular of in - construction , and is the number a corpus, . Intuitively, of e.g., types of the tokens all ending nouns n 1 in - that construction, (one of a higher multicharacter value of MP indicates e.g., the number probability unique ending of) a to higher word. For that Sproat the component Shih (2002) parts a MP string of appears Chinese noun be a example, and productive verb - Ç are 0.20 MP and 0.04, show respectively, that the affix - and indicating values more affix while of nouns that which - is a much af fix, the value single-character belong to Gao et al. Chinese Word Segmentation: A Pragmatic Approach a closed and intuition. nonproductive Similarly, we find class is close to 0. These productive results are in agreement in very words characters with high MP with our 13 our some training only there are 236 values. set, that contain the character = , For example, which occur once. among 5.5.2 The Context is to Model. The motivations for information. using a context For model for NWI new words are twofold. first to capture nouns useful contextual pronouns, POS is example, is than and tagging context-sensitive. The are more likely more be important. As in other Section 4, with a context model, (e.g., NWI second described with word word can be per- formed simultaneously morphological in segmentation manner. tasks breaking, NER, and However, it is dif analysis) to a uni fied “we ficult not develop know a what training we don corpus where new know.” words Our solution are We of new words from our ’t is anno- tated Carlo because simulation. usually do Monte We —the probability sample a set that any lexical word from w would dictionary new-word-annotated new according word-segmented be a word P(NW to the distribution | w). then Now generate we a way words P(NW | w) corpus is a of estimated. It is text corpus. new words describe the whose probability in reasonable new to assume is that are those document. lexical In words. our Let P (k) we be probability appearing of word a document i the probability assume K in P(NW new i ) w i that occurs lower in than general a that | w can be approximated k times of occurring experiments, by the w i less than times a document: (14) where The the constant K (7 in our experiments) is dependent P (k) on the size of the document: larger the models document, (Chapter the larger 15.3 in the Manning value. i Sch can be estimated [1999]). Following using several (2000), K-Mixture and ütze P (k) Gao term Lee distribution we (Katz 1996) which and use estimates i as K1 P(NW | w i ) ≈ ∑ P i (k), k = 0 P (k) (1 a) δ k,0 + a 1 ( β 1 ) k i = − [BAR] [BAR] β + β + , (15) where observed δ k,0 mean =1 if k=0; 0, otherwise. observed inverse α and document β are parameters that IDF can be fit using the λ and the frequency as follows: (16) where number cf of is the total in number of occurrence in which of word occurs, w i in training N is data, total df number is the of documents In our implementation, training data w i and approximately the documents. the training domain, data contain time. 40,000 documents that have been balanced among style, and Computational Linguistics Volume 31, Number 4 P R F All features w/o w/o IWP w/o AWP w/o MP WFA P R F All w/o features w/o IWP w/o AWP w/o MP WFA sifier. 5.5.3 Evaluation This Results. section of NWI. discusses First, we investigate factors that believe have the of four the performance motivated in NWI. Second, we the relative contribution methods where the we linguisti- cally (i.e., features SVM post-processor compare use the NWI component models an of Equation classi (4). fier) as a versus as a feature function in the linear Table NWI 11 results on PK and CTB corpora, NWI as post-processor versus uni fied approach. PK CTB MSRSeg Roov F P R Roov F P R w/o w/ NWI NWI w/ NWI (post-processor) (uni fied approach) .746 .741 .781 .953 .956 .955 .947 .949 .952 .960 .963 .959 .722 .690 .746 .899 .892 .904 .886 .875 .895 .912 .910 .914 Table NWI 12 results on HK and AS corpora, NWI as post-processor versus uni fied approach. HK AS MSRSeg Roov F P R Roov F P R w/o w/ NWI NWI w/ NWI (post-processor) (uni fied approach) .728 .694 .746 .952 .947 .954 .944 .937 .948 .959 .958 .960 .549 .436 .584 .955 .951 .958 .950 .958 .955 .959 .943 .961 The two NWI component we has been constructed most as impact an SVM on clasGao et al. Chinese Word Segmentation: A Pragmatic Approach feature. Tables 11 and 12 approaches show (i.e., results of NWI NWI on four Bakeoff test sets. We can outperform the component as a feature function) signi see that unified consistently, in of approaches using both Roov (i.e., P/R/F of NWI ficantly consecutive using the component overall word as a post-processor) empirically terms and of the segmentation. approach Chinese bene word fits using the context model in 5.5.2. for NWI This demonstrates the and the unified to segmentation, as described

The decoding process follows the source –channel framework. It consists of three steps: Step 1: Throughout lattice, which is initialized the process, we maintain empty. an array of word class candidates, called a to be Step 2: Given a Chinese sentence, all possible words models of different in types For the corresponding channel described Sections are 5.2 generated simultaneously by in Table 6, TRIE LW to 5.5. example, as shown NW the lexicon so on. All generates candidates; the SVM classifier generates Each candidates, in and is 5-tuple the ( generated w,i,l,t,s), candidates the position lattice in a l is where are the length of w, w is added to the lattice. is element the of is the word candidate, i is starting model w of the sentence, its t in Table the word class Some tag, and s the class in Figure score 5. w assigned by feature function 6. examples are shown Step 3: The Viterbi (or A*) algorithm in is used to search for the best among all segmentations lattice, according to Equations word class (2) and sequence, candidate the (3). For efficiency, Thus, word we sometimes need to control the are ranked normalized search space. Given tence by their class model an input sen- s, all number candidates of (i.e., of is controlled score two λ f( . ). the candidates the size the lattice) by parameters: Number given threshold: The maximum number of candidates cannot be larger than a threshold; Score threshold: The difference and bottom-ranked between the class model score of the top-ranked given candidates the candidates cannot be larger than a threshold.

This output section of describes the second component of MSRSeg: a set of adaptors for adjusting the We the generic segmenter to different application-speci Suppose fic standards. consider the following fined standards ourselves. adaptation We paradigm. we have a general standard prede by have also created a large training segmented according this general standard. We amount of data that a word are segmenter, i.e., to the system described in Section 5. Whenever then de- velop generic we Computational Linguistics Volume 31, Number 4 deploy the segmenter for any application, we need to customize which the is output itly an application-speci fic standard, not of the segmenter according defined. to However, it is often implicitly fined in given always of explic- (called from de tion data which a amount applica- adaptation data) the speci fic standard can be partially acquired.

In MSRSeg, ordered standards of adaptation is on conducted output by of a postprocessor that performs —removing an list transformations word inserting the new the generic segmenter extraneous meets boundaries standard. and boundaries —to obtain a word segmen- tation The that method a we different initial use is Transformation-Based Learning into (TBL; initial a of which Brill we [1995]), an goal segmentation wish which requires segmentation, (i.e., to transform the segmentation, Under and a space allowable transformations tion above-mentioned adaptation paradigm, the initial transforma- templates). is The the output of the generic segmenter. make The goal to words segmentation (i.e., is segmen- tation the adaptation data. well transformation some templates as prede fined can types (i.e., reference lexicalized tem- plates) below. as class-type based templates), as described We from notice that words most variability not in word segmentation in across dictionary. different comes typically stored the Those standards those in nature and that are through productive morphological words are processes. dynamic In study, we are For this on usually formed word focus three categories: MDW, to belongs is to these with categories, ‘word we NE, and de fine FT. each Wu (2003). class The that root an internal structure similar other nodes. structure There a 30 tree class As ’ as the in and the types. shown Figure ‘component types’ word as Af are component two Date types: Pre nine fix, —3 Stem, non-terminals and Suf fix. Similarly, 6, the PersonName class fixation has three component These has internal component types and has words as and 6 as terminals. structures are assigned to by the generic segmenter at Figure Word internal 6 structure and class-type transformation templates. Gao et al. Chinese Word Segmentation: A Pragmatic Approach Table 13 Comparison scores for PK open and CTB open. PK (# of Tr. Word = 1.1M) CTB (# of Tr. Word = 250K) P R F Roov Riv P R F Roov Riv .854 .959 .946 .963 .963 .839 .955 .929 .959 .953 .320 .781 .524 .799 .743 .861 .972 .977 .975 .980 .799 .895 .805 .907 .891 .818 .914 .874 .916 .911 .809 .904 .838 .912 .901 .624 .746 .521 .766 .736 .861 .950 .952 .949 .949 run time. form: The transformation templates for words of the above three categories are of the Condition: word class Actions: Insert: Place a new boundary between two component types. Delete: Remove an existing boundary between two component types. is Since the application on word of the and transformations makes derived to from the above we templates conditioned class reference Some component types, in Figure call we templates. examples are shown 6. the templates In class-type transformation addition, also use lexicalized transformation templates as follows: Insert: Place a new boundary between two lemmas. Delete: Remove an existing boundary between two lemmas. Here, morphological lemmas refer process. to those They basic lexical mostly words single that cannot be two-character formed by any words, pro- ductive 4-character idioms. are characters, and

The Tables results 13 and of 14. standards 10 A adaptation of on four Bakeoff open test is sets are shown set each standard learned TBL in from our corresponding Bakeoff transformations set. For for training each test set, we using the with without (Rows 1 report improves standards across adaptation in and 2). It results turns out using system and that performance For we dramatically maximum matching) also include in the board all four each table the results of test sets. ward comparison, using (Row the 2 segmenter as a generic segmenter 3), FMM and (for- top Segmentation (sorted F) greedy that (Rows are reported 4 in 5). SIGHAN 11 We ’s First International with Chinese the Word scores by Bakeoff and can see that adaptation, Computational Linguistics Volume 31, Number 4 Table 14 Comparison scores for HK open and AS open. HK (# of Tr. Word = 240K) AS (# of Tr. Word = 5.8M) P R F Roov Riv P R F Roov Riv .822 .960 .823 .958 .909 .820 .954 .821 .956 .886 .593 .746 .591 .788 .579 .840 .977 .841 .971 .935 .832 .955 .930 .894 .853 .838 .961 .947 .915 .892 .835 .958 .939 .904 .872 .405 .584 .160 .426 .236 .847 .969 .964 .926 .906 Table Size 15 of training data set and the adaptation results on AS open. Size of training set(# words) R P F Roov Riv .961 .959 .954 .832 .947 our generic its segmenter can over achieve other state-of-the-art For performance is no different single standards, showing in Bakeoff superiority that top-2 systems. notice Tables achieved 13 and 14 ranks in example, there all four test sets (Sproat and Emerson segmenter the We in that quality of adaptation to 2003). outperformed size of Bakeoff adaptation data (indicated the by # of Tr. Word in seems depend largely upon the in AS of the tables): size of we the while best we worse systems in the CTB set because of the large our the set because the small size of the adaptation data, of To are verify hypothesis, we the adaptation data. AS of different and evaluated observed the adaptation results using subsets in Table 15. the However, training set with much sizes the same (e.g., trend, 250K as we smaller adaptation data set words), shown outperform even Bakeoff a still the best results.

This section describes model (semi-)automatic probability methods P( ) (i.e., of creating probability) the training data in Equation based on the Ideally, estimated given class an where w trigram each sentence is segmented into words (1). annotated corpus, are MLE. classes, the trigram word class probabilities that tagged Unfortunately, by their is can be calculated using building such annotated training corpora very expensive.

Our basic of solution is the (1) bootstrapping Initially, we approach described word in Gao et al. (2002). It consists and three obtain steps: initial model use a greedy on initial segmenter annotated to annotate the corpus we an context obtained based models; the (3) we corpus; (2) reannotate the corpus using the and retrain the context .956 .950 .945 .838 .935 .958 .954 .949 .835 .941 .603 .623 .591 .405 .595 .968 .966 .962 .847 .955 on Gao et al. Chinese Word Segmentation: A Pragmatic Approach Figure (a) A Chinese 7 whose OASs OAS ý . (b) Two with sentences in the training OAS set, which . (Li contain 2003). the OAS and (c) have been replaced the single tokens < > et al. Table 16 Methods of resolving OAs in word segmentation, on the MSR test set. Methods Accuracy FMM BMM Rule Rule + + FMM Ours BMM model of using approximation the system converges. This approach is also named are of EM Viterbi iterative training, an In of approach, training. the above initial the annotated quality of the context which model is, however, depends not to a large degree upon problems. the quality First, the corpus, many greedy segmenter cannot deal with satis fied due to two ities, the Second, many even after iterations, these ambiguities can only the segmentation ambigu- and named identi be partially resolved. segmenter, which factoids is and on dictionary. entities cannot be fied using the greedy word ities To solve based in initially the problem, the first we use two methods We to resolve word segmentation ambigu- into the segmented respectively, overlap training data. to OAS (OA) classify segmentation ambi- guities two classes: ambiguity CAS, fined and combination in Section 3.5. ambiguity (CA), corresponding, To OA, OAS we . An identify OASs and all is in de resolve the in training Figure 7. data By and doing replace we them with a single token of < > example shown training that likely to contain OA errors. We so, remove the portion model data are not OASs. thus train using training set that does contain Intuitively, a context the model reduced context would the ambiguities. The method any the resulting MSR maximum Our main resolve results —BMM) are shown method in Table 16. only We has been tested set. can see that the FMM on the OAs, (or test backward while our method, matching model can resolve 94.3% 73.1% from resulting ones to of (or 71.1%) using context can resolve the For different the previous that use lexicalized rules resolve OAS. OAs. of Our method is example, Sun and Zuo (1998) report that over 90% of OAs can be disambiguated the reannotated corpus. 12 Steps 2 and 3 iterated until the performance Computational Linguistics Volume 31, Number 4 Figure 8 Results method of 70 high-frequency two-character more CASs. ‘Voting’ of indicates CAS. the ‘ME’ accuracy of of maximum-entropy always chooses the frequent case a given indicates the baseline that fier. ‘VSM’ indicates of method of the VSM accuracy the disambiguation. classi the accuracy the using for simply 90.7% (or by 91.3%) rules. We of reimplemented OAs in their MSR method in our experiments and Sun not the test set ours. can Therefore, The found the be resolved. result is that ilar to method Zou ’s outperforms but still as good as approaches. we sim- and our signi is ficantly the rule-based Another conclude that our method it is an approach that requires no advantage of Readers that to Li unsupervised (2003) more human annota- tion. To can CA, refer we et al. / able’), 70 for details. resolve select high-frequency in Figure two-character 8. For CASs CAS, we (e.g., M ý ‘talent’ and M ý ‘just as shown sentences contain the CAS each train a binary classifier using segmenter. Then, that occurrence and of that CAS have in been the for each the initial segmented using greedy Our fier is a used 95.7% to of determine CAs whether segmented training data, the corresponding classi the CAS should Detailed be segmented. experiments in Figure 8, show where that ‘Voting’ the indicates can be resolved. results method are shown more of the accuracy ‘VSM’ chooses the frequent case a given CAS, of the baseline classifier, which indicates that always and will the accuracy of the 13 VSM-inspired Suppose we (vector CAS, space model) in is i. We described its six surrounding here. words have in a s, sentence i 3, whose binary be 3 We fine of w positions i 2, position IDF 2, a use i 1, i i features. then de a set feature functions . In particular, to simulate TF1( the ,w)(or TF- i + 1, + and + TF2( , w)) Each as scores. feature function is of a mapping in f( s ,w) Similarly, term frequency w the case that s is a 1-word (or let 2-word) s s be 2-word) IDF1 the ( s ,w) (or IDF2 occurs ( s ,w)) in be both 1 if w only occurs IDF1 ( in string. 1-word (or let If ,w)= the case IDF2 that ( ,w)= s is a 0.25. We weight string. w cases, let also for position empirically (i.e., in our s s assign λ and each Then we experiments, λ =1, 2 = λ 2 =4, λ 1 = λ 1 =7). can calculate the score of we have 1-word 3 = or λ + 2-word 3 λ Score2( Equations + two words (18) and otherwise. (19), + respectively. The CAS is a single word s to be if Score1( s ) by > s ), and Score 1( s ) = ∑ TF 1( s ,w i )IDF 1( s ,w i ) λ i i = 3 + 3,i = ... 0 (17) Score 2( s ) = ∑ TF 2( s ,w i )IDF 2( s ,w i ) λ i i = 3 + 3,i = ... 0 (18) Gao et al. Chinese Word Segmentation: A Pragmatic Approach Our experiments show that the VSM-inspired fiers this task. For classi fier outperforms example, maximum entropy other well- known classi for particular For features achieved problem of an NE overall and FT accuracy of a 94.1%. classi fier using the same the second approach, in Section detection, 5.4, though, FTs in we can simply initial use the FSA- based our method of NER as described in initial (i.e., to detect 1) is the more segmented sophisticated. corpus; manually the named step on step a little annotate small subset (called the seed set) of First, we Then we improve obtain entities model a a is on the seed on set (called the training data. context model initial the seed model). We thus the interpolating 2 it context 3 with that trained of the seed model. Finally, the We we annotated use the in improved training corpus by in and the show the next context model steps (e.g., bootstrapping. 150K words) is shall subsection that a relatively model small initialization. seed set enough to get a reasonably good context for

To justify different the methods initial just described, For we of built initial a large number of context models using Viterbi corpora. iterative procedure each the corpora, i.e., a context model using until convergence, the improvement is trained word the mentation The of resulting system, is of the segmentation performance of are shown initial in Table the 17, where Row 1 (FMM) less than a preset threshold. results in Row segmented 2, we by a greedy word presents the seg- results using the corpus (overlap) segmenter — the basic solution Rows on 8, top of described earlier; the corpus in Row 1; we resolve then tag FTs segmentation in Rows 3 and 4. From am- biguities 5 to several NE annotated seed sets of different approximately human million cost. In Rows sizes For 1 are to 8, we used, showing the trade-off between performance and 50 use include containing characters. comparison, we the raw training in set Row 9 manually of MSRSeg, whose context model has been trained on also 20-million-word the results The a annotated corpus. experimental results reveal several facts. Table Comparison 17 iterative of performance with different of MSRSeg: initial The versions (Rows that on training sets 1 are to 8) trained using (semi-)supervised training annotated corpus of 20 million words (Row 9). versus the version that is trained Computational Linguistics Volume 31, Number 4 Table 18 Precision of initialized person name recognition with on the different MSR test set, using Viterbi iterative training, by four seed sets sizes. Seed set (M char) w/o iteration w/1 iteration w/2 iterations .745 .757 .783 .815 .790 .796 (+1.6%) .824 (+1.8%) .835 (+5.2%) (+2.5%) .800 .802 (+0.7%) .835 (+0.5%) .840 (+1.3%) (+0.6%) Although the in greedy word segmenter (FMM) can resolve in around Table 16, 90% of ambiguities is much segmentation, words worse MSRSeg as shown the still because a large number resulting segmenter than Viterbi of unknown cannot be detected correctly even after iterative learning. The method method of we not resolving OA brings marginal improvements. human annotation, Row 2 Since the does require in our any shows the best results approaches. achieved experiments using unsupervised learning Factoid rules, although simple, bring substantial improvements. The way Viterbi of iterative training method Rows ambiguities 4, word in word does not turn out segmentation or of to be an effective words. resolving In 2 1 detecting new or to 3 iterations, the with segmentation improvement. performance (Row an initial little F-measure For always saturates FMM after two 1) segmentation of example, iterations, achieves it and with 0.8773. 0.8771, and after saturates ends up Table Precision 19 initialized of location name recognition with different on the MSR test set, using Viterbi iterative training, by four seed sets sizes. [BAR] Seed set (M char) w/o iteration w/1 iteration w/2 iterations .893 .883 .894 .895 .901 .895 (+0.9%) (+1.4%) .901 .902 (+0.8%) (+0.8%) .897 .900 (+0.2%) ( − .900 ( 0.1%) − .903 (+0.1%) 0.1%) Table Precision 20 initialized of organization name with recognition different on the MSR test set, using Viterbi iterative training, by four seed sets sizes. Seed set (M char) w/o iteration w/1 iteration w/2 iterations .695 .737 .745 .757 .770 .786 (+10.8%) .790 (+6.6%) .796 (+6.0%) (+5.2%) .780 .794 (+1.3%) .800 (+1.0%) .802 (+1.3%) (+0.8%) Gao et al. Chinese Word Segmentation: A Pragmatic Approach The Viterbi without iterative training for is effective (e.g., in boosting the precision of NER when great sacri fices of recall Row 5 in the when of Rows 6 Table recall 8). 17, As or remains almost the same using the seed set becomes in a little Tables 18 worse using with the seed of sets of to shown NER, which sets is different sizes with and –20, we start of a series seed of achieve MSRSeg, a reasonable accuracy iterations. comparable that after two The use of a small NE annotated seed set (e.g., in Row 5) would achieve the best trade-off iterations, between performance of NER is and human to effort, while of very close that of because after two the accuracy is using much larger seed sets, the human effort creating the seed set less.

Our system is designed on or off so that components we such investigate as the FT detector and NE recognizer can be “switched overall MDW word ” so that can the relative We performance. To contribution date, work. we of have not done each component to the of segmentation a separate The main evaluation recognition. (Row results are of shown in Table 21. For leave that to future 1) comparison, (FMM) we also include in Section in the table Row 2 the results using the of our greedy segmenter shows the results system, where only described find, in Rows 1 2, the lexicon is used. It 7. interesting to baseline and is the dictionary-based methods is that not methods already achieve quite good identify recall, unknown but the words precision not very in good because those cannot that lexicon, such factoids and named cor- rectly entities. We models find are the that even using the same approach lexicon, (with our approach as mixture also outperforms based on the linear difference) the most promising the use property of greedy context of our model a slight approach resolves is more but statistically significant The because ambiguities mixture in segmentation. models provide where wide of that the linear and statistical models a flexible framework in uni a variety can fied way. As linguistic shown in Rows knowledge when be combined overall are switched on in a turn by activating corresponding class models, 3 to 6, components the word segmentation performance increases consistently. Table MSRSeg 21 system results for the MSR test set. Word segmentation FT PN LN ON Segmenter P R P R P R P R P R Baseline FMM 2+FT 4+LN 3+PN 5+ON .863 .837 .950 .913 .969 .955 .947 .927 .972 .961 .979 .975 .905 .904 .905 .905 .898 .898 .899 .899 .858 .790 .870 .906 .905 .906 .892 .794 .855 .860 .816 .654 Computational Linguistics Volume 31, Number 4 We also conduct an error analysis, which NE shows that 85% of errors come from and especially the abbreviations, although the tokens of NER factoid word detection, new to only 8.3% in the MSR test set. The remaining 15% of these types mainly amount words. errors are due to

We 14 compare our system with three other Chinese word segmenters on the MSR test set: 1. 2. 3. The MSWS Microsoft (as system is of one of Windows the APIs). best available MSWS products. It is released a first conducts word by MM (augmented set using heuristic rules for disambiguation), and breaking by NER then conducts factoid detection and using rules. The It is LCWS system is one Beijing of Language the best University. research systems The in mainland to MSWS, by system works China. released has larger dictionary more PNs and LNs. similarly but a containing The PBWS output system is word a rule-based Chinese parser It (Wu and Jiang 2000) that can also segmentation results. for Chinese explores word high-level linguistic knowledge NER. such as syntactic structure segmentation and mentioned As mentioned earlier, only to achieve in a of fair NER comparison, we OAS errors. However, systems terms precision and compare recall and the previously of four it is we find that to the number due to the different annotation automatically. speci fications used by these systems, ‘Beijing still very city dif ficult compare their results inconsistently For example, / ¬ ‘Beijing ? city + government ’or ’ has / been segmented ‘Beijing’ + as ¬ in ? ’ in the same system. Therefore, Worse ‘government still, we some LNs ¬ ? manually tagged in one ‘city government ’ even ONs another system are We tagged as 933 system. 22,833 have words to (including check 329 the PNs, results. ONs) at random We containing did not differentiate LNs and ONs in 617 LNs, picked 435 sentences That and is, interchangeable. we only for testing. The word also boundaries of LNs and ONs and evaluation. checked the in Table 22. We treated in both small tags as MSRSeg results are overall shown of NER can and see that this test set, OASs. achieves the best performance the best performance of resolving

Table others 23 presents indicates previously. the comparison The results sites of of MSRSeg on in the table 1st SIGHAN follows (Peng, four Bakeoff International Feng, and test McCallum sets with 2004). SXX reported layout Segmentation Bakeoff participating (Sproat Emerson the 2003). CRFs indicates word Chinese Word in Peng, Feng, and McCallum and (2004), which models the of linear-chain segmenter reported uses conGao et al. Chinese Word Segmentation: A Pragmatic Approach Table 22 Cross-system comparison results. # OAS LN PN ON Segmenters errors P R F P R F P R F MSWS LCWS PBWS MSRSeg .935 .854 .767 .876 .442 .720 .736 .864 .600 .782 .752 .870 .907 .945 .780 .830 .744 .781 .787 .897 .818 .856 .784 .862 .642 .713 .817 .799 .469 .131 .216 .617 .600 .222 .342 .696 Table 23 Comparisons 1 SIGHAN International against other Chinese segmenters: In Word Column 1, SXX Segmentation indicates Bakeoff, participating CRFs indicates sites in word the st in (Peng 2004). In Columns 2 to 5, entries and contain F-measure the segmenter reported on different open et al. with in bold. Column the Site-Avg is of each segmenter F-measure over runs, on the which best performance of open the average bolded entry indicates the data sets outperforms a segmenter MSRSeg. reported Column results over segmenter where bolded Our-Avg is runs, where a F-measure of MSRSeg the other the same data sets, a entry indicates the that MSRSeg average outperforms the segmenter. [BAR] ASo ASc CTBo CTBc HKo HKc PKo PKc Site-Avg Our-Avg S01 S02 S03 S04 S05 S06 S07 S08 S09 S10 S11 S12 CRFs MSRSeg ditional random open runs, fields indicated XXo, with contain the the performance in each different Avg is F-measure over by best of the runs, average where the indicates data sets on which bold. Column Site- open a segmenter outperforms reported MSRSeg. results Column bolded Our-Avg indicates is a bolded entry the F-measure outperforms of MSRSeg the segmenter average MSRSeg over other the same segmenter. data sets, For where a entry we include that in Table 23 of the only results material closed runs, indicated com- pleteness, In also one the from by a use training the training data for XXc. closed test, can Since corpus being tested MSR on. No other for material was allowed (Sproat and Emerson the particular 2003). MSRSeg uses the corpus training, our results are of open tests. .930 .943 .904 .955 .955 .955 .930 .943 .943 .943 on .872 .904 .958 .957 .942 .945 .961 .956 .959 .881 .829 .912 .884 .901 .894 .904 (CRFs). Entries .881 .874 .732 .829 .831 .849 .886 .956 .879 .946 .954 .901 .924 .904 .928 .916 .953 .925 .937 .938 .940 .886 .959 .946 .955 F-measure of .951 .894 .939 .924 .936 .947 .946 .941 .917 .878 .912 .937 .947 .940 .930 .888 .936 segmenter Computational Linguistics Volume 31, Number 4 Several conclusions can be drawn from Table 23. First, for the same system, generally material. results than no closed single tests due to the use of open tests achieve 16 better training Third, Second, MSRSeg is additional there segmenter that performs best For MSRSeg consistently high performance across in all four data sets. achieves all four data sets. example, achieves that results on better average performance all (i.e., S03, than three four data S11, the other segmenters particular, MSRSeg report outperforms on sets F-measure data set. There CRF). In them every ours. One is S02, which are two segmenters that on achieve CTB better only. The other average is S10, which than on CTB reported we MSRSeg results and PK. results is an reported word From these results, of-the-art conclude that on adaptive segmenter that achieves state- performance different data sets, corresponding to different domains and standards. As roughly described in grouped Section into 2.1, most segmenters, ones including the ones in Table approach 23, be that rule-based and ones can that approaches. use a approach. two categories: statistical Though MSRSeg is use a (e.g., a hybrid S08, S10, system that S11 takes in Table advantage 23) of both rule-based systems make and of increasingly can achieve reasonably good weak results, in unknown they cannot and in word effectively Table 23) detection and use models adaptation. Some large training data menters and (e.g., are word S01 S07 generative such as HMM statistical for Chinese seg- into However, (approximated or it is use segmentation. very dif ficult to incorporate process of Chinese linguistic knowledge neath the which models assumed) generation Discriminative models (e.g., sentences, under- in MSRSeg, where the are developed. models from the linear discriminatively components are derived generative models, models though all they provide are combined mathematical using trained incorporate weights) are free from this issue and They a flexible not underlying framework to process. arbitrary Instead, linguistic knowl- edge. do and assume any from generation they assume form of that the training (i.e., test sets are process) generated is the same If we distribution, Chinese but word the the distribution generative ones, classi fication problem, i.e., to discriminate unknown. view segmenta- tion as a we may models to between “good” models. segmentations Intuitively, it is and “bad” from to ones find prefer discriminative generative sufficient (as directly in discriminative the desired models). features that It is, can however, differentiate not necessary good segmentations distributions bad which Chinese (or to estimate the based upon sentences use estimated distributions to are generated segmentations) first, and then models). the As pointed out Vapnik (1998): construct directly to solving by more “When the desired features problem solving an a given problem, (as in generative solve it and try avoid a general as intermediate step.” in Peng, Our models Feng, and are McCallum similar to the (2004) maximum in entropy models models in give Xue (2003) and CRFs incorporate that all these Our the models flexibility to novel in arbitrary many features and can be discriminatively that functions derived from probabilistic trained. or heuristic models are Section inspired feature by Therefore, source –channel models are 4.3. of Chinese not sentence only potentially generation, much feature functions are more as described in maximum entropy more models informative these in NLP. than, for instance, the binary features used in reasonable but also standard Gao et al. Chinese Word Segmentation: A Pragmatic Approach We also notice word that many from segmenters word (e.g., S03 and S04 Though in Table would 23) make separate unknown of detection the easier, it segmentation. to this flawed solution in reality, the de- velopment earlier. segmenter The of seems integrating be both a as we Table fits tasks has also been shown empirically dis- cussed 23. bene in

This article presents a pragmatic First, approach we to Chinese Chinese word words segmentation. Our main contributions definition is are threefold. in nature view on how as segmentation and units whose (differently) in pragmatic and while depends they are used we theoretical linguists mathematical de fine processed realistic Second, applications, purely propose words using Chinese linguistic word criteria. where problems a pragmatic of word framework (i.e., word for morphological segmentation, various approach. The approach factoid on detection, NER, models and segmentation breaking, analysis, is NWI) where are solved in models a unified are inspired source –channel models based in Chinese of Chinese linear component by word segmenter, sentence MSRSeg. generation. This Third, we de- scribe of detail two an adaptive (1) is on pragmatic consists segmenter that based the mathematical system of word components: a generic framework segmentation and unknown (2) word of detection, output and that for can adapt to different output of domain-speci fic vocabularies, and a set adaptors the different application-speci fic standards. Evaluation adapting on the with former to five test sets different on standards shows that the adaptive system achieves state-of-the-art performance One of all our the test sets. area We future work is to apply MSRSeg in a wide range of practical applications. into believe For instance, that some in application-speci MT, it would interesting fic features can investigate also be integrated the optimize framework. of both word be and to word how to jointly MSRSeg As one is of the performances impressive, reviewers it is pointed one out, segmentation alignment. the of though most the reliable high performance with of richest Hence, by far interesting the complex of our systems work is access to the whether resources. is another We attributed to area future a superior architecture or to explore the performance simpli of MSRSeg, simply to resources. developed fied called S-MSRSeg. the richer not have use 4, morph-lexicon a only and is version trained using (i.e., LWs one- fifth of the MSR It does in Table the which not Inter- training data estingly, S-MSRSeg are partially 4. similar (or labeled Table This very slightly worse) are annotated). in achieves of performance test demonstrates again the potential our on the pragmatic approach five to Chinese sets word segmentation. of another The work reported of Chinese in this word article represents Toward not an end but a S-MSRSeg beginning and its yet training and view segmentation. this http://research.microsoft.com/ end, test data sets of are publicly available we jfgao) for the sake encouraging others (e.g., to improve at ∼ upon the work have carried out. Acknowledgments The work The reported number of in this article intimately was a team effort. involved in project people this We only is far name too large for now. us to We enumerate group the manager here. MSRA and teams NLG of a few thank the NISD, NLC at Ming Zhou, the Hong-Jiang at especially Zhang, Jun Liu, and project. Kai-Fu We Lee, for initiating Ashley Chang, and funding Hongqiao the Qin, Li, Jianfeng Sun, Li, thank Haowei Jian Xinsong Jianghong Xia, Wenfeng Li, Yang, in Ye Zhang, and Xiaodan Zhu for their help of our the experimentation for and evaluation We system, John and Chen, data Nianwen annotation. also thank Xue, Computational Linguistics Richard Sproat, Keh Yi Su, Hisami Suzuki, and the three anonymous on reviewers of for their comments early drafts this article. References Aho, Alfred Ullman. V., Ravi 1986. Sethi, and Compilers: Jeffrey Principles, D. Techniques, Boston. and Tools. Addison-Wesley, Baayen, R. Harald. 1989. Morphological A Corpus-based Approach Statistical to Productivity: Interpretation. Analysis Ph.D. and Psycholinguistic Amsterdam. thesis, Free University, Berger, Adam, Vincent J. Stephen Della Della Pietra. Pietra, 1996. A and entropy approach processing. to natural maximum Computational Linguistics language 22(1):39 –71. , Brill, error-driven Eric. 1995. Transformation-based processing: learning A and natural language Part-of-Speech case Computational study in Linguistics , 21(4):543 tagging. Chang, Huang. Eric, 2001. Jianlai Speech Zhou, –565. Yu in Shi, box: and A Chao Mandarin toolbox lab to a speech In Eurospeech Jumpstart speech 2799 related –2782, research. Chang, Jing-Shin and Aalborg, Keh-Yih Denmark. ’2001, pages iterative method Su. for 1997. Chinese An unsupervised new Computational lexicon extraction. Linguistics International Chinese Journal of Language Processing , 2(2):97–148. and Chen, Aitao. 2003. Chinese In The Second minimal word segmentation using SIGHAN linguistic knowledge. July Workshop 11–12, on Sapporo, Chinese Language Japan. Processing . Chen, Unknown Keh-Jiann word and Ming-Hong Chinese Bai. 1998. corpus-based detection for by International a Journal learning Computational method. Linguistics Chinese of 3(1):27–44. and Language Processing , Chen, An Stanley F. and Joshua of smoothing Goodman. 1999. empirical study techniques Computer Speech for language Language modeling. Cheng, Kowk-Shing, and Gilbert H. Yong, , 13:359–394. Kam-Fai Wong. 1999. A on and word-based integral-bit study and JASIS, Chinese text compression 50(3):218 –228. algorithms. Chien, Lee-Feng. 1997. PAT-tree-based keyword information extraction for In Chinese retrieval. Proceedings of the Volume 31, Number 4 Gao et al. Chinese Word Segmentation: A Pragmatic Approach Gao, Li, Jianfeng, and Joshua Kai-Fu Lee. Goodman, approach 2002. Mingjing Toward a uni fied to Chinese. statistical language Transactions modeling for Language ACM Information Processing on , Asian Gao, Jianfeng, Mu Li, 1(1):3–33. Huang. 2003. and Improved Chang-Ning model for Chinese source-channel Proceedings word 41st segmentation. Meeting In of the Computational Annual Linguistics of the Association July Gao, 7–12, Sapporo, for Andi Japan, Wu, Mu Li, pages 272 , Jianfeng, –280. Huang, Haowei Hongqiao Qin. 2004. Li, Chang-Ning Xinsong Adaptive Xia, and In Chinese Proceedings word segmentation. Meeting of the 42nd Annual Computational Linguistics of the Association , July 21–26, for Barcelona, Gao, Jianfeng, pages Hao 462 Yuan. 2005. Yu, –469. Peng Minimum Xu, risk and Wei methods Proceedings Conference modeling. sample for language Language of the Empirical on Human In Natural Technology Language Processing and Frakes, B. and Ricardo Baeza-Yates . Methods in (eds.). William Hall, Englewood 1992. Information Cliffs, NJ. Retrieval . Prentice Fung, Pascale and of Dekai Chinese Wu. 1994. machine- Statistical augmentation Second dictionary. a readable Workshop In Proceeding Very Large of the Corpora, Annual S. Kyoto, 1987. Estimation pages 180 on Katz, M. of –181. from probabilities sparse of data for the recognizer. language model component 35(3):400 a –401. speech IEEE Katz, ASSP words S. M. and 1996. Distribution in of content modeling. Natural phrases Language text and Engineering language Kneser, 1996(2):15 –59. , Improved Reinhard and Hermann for m-gram Ney. 1995. modeling. backing-off In Proceeding IEEE language International Signal Conference of the Processing , on Acoustics, 181 –184. Speech, Krauth, and W. with M. optimal M pages and èzard. 1987. Learning algorithm networks. Journal Physics stability , 20:745–752. in neural Hockenmaier, Julia of Chris Brew. A Error and of Chinese. 1998. Communications driven segmentation COLIPS , 8(1):69–84. In Huang, Chu-Ren, Keh-Jiann of Lili, Feng-Yi Chen. 1997. Chen, Segmentation Chang and standard processing. for International Chinese natural Journal language Computational Linguistics Chinese of Language Processing , 2(2):47–62. and Joachims, Text Using Thorsten. Support 2002. Vector Learning Machines to . Classify Kluwer. Li, Hongqiao, Gao, Chang-Ning SVM Xiaozhong Huang, Fan. 2004. Jianfeng and for Chinese new word The identi use of In Proceedings Conference the First fication. of Natural International Sanya City, Language Joint on Hainan Island, Processing China, , March Li, Mu, 22–24, Jianfeng Gao, pages 497 –504. Jianfeng Li. Chang-Ning 2003. Huang, and Unsupervised training for overlapping in Chinese word ambiguity resolution In Proceedings Second SIGHAN segmentation. Workshop Chinese of the July 11–12, on Language Sapporo, Japan, Processing 1 –7. , Liang, Nanyuan. ( CDWS ) 1987. pages W C (A written f Chinese b I í ê Í û ß – automatic segmentation (Journal system). Chinese In Information - á o f Processing ¥ ), of Lin, Ming-Yu, Tung-Hui 2:44–52. Su. 1993. A Chiang, on and Keh-Yi word preliminary In problem study unknown Proceedings in Chinese ROC word segmentation. Computational Linguistics Conference of the 119–141. VI , Manning, Schutze. Christopher 1999. Foundations D. and Hinrich Natural Language Processing of . Statistical Cambridge, MA. MIT Press, Mitchell, McGraw-Hill, Tom M. New 1997. York. Machine Learning . Nie, Mareie-Louise Jian-Yun, Wanying Hannan. Jin, 1994. and approach word A hybrid to unknown of Chinese. In detection Proceedings and segmentation International Conference Chinese of the Och, Computing 2003. , pages Minimum 326 –335, on Franz. error Singapore. in machine rate training In Proceedings statistical 41st translation. of the Computational Annual Meeting of Linguistics the Association , July 7–12, for 160 –167. Sapporo, Japan, Packard, pages Chinese: Jerome. Linguistics 2000. The Cognitive Morphology of Cambridge A University and Press, Cambridge. Approach Palmer, David D. 1997. word A trainable rule- based In Proceedings algorithm for 35th segmentation. of the Computational Annual Linguistics Meeting of Madrid, the Association for Peng, Fuchun, pages Fangfang 321 –328. McCallum. 2004. Chinese Feng, and Andrew new word segmentation and detection In COLING using 2004 conditional random 562 fields. –568. , Roche, pages Finite-State Emmanuel Language and Yves Processing Schabes. . MIT 1997. Cambridge, MA. Press, . , Computational Linguistics Rosenfeld, Xiaojin Roland, Zhu. Stanley 2001. F. Whole Chen, and sentence exponential language models: A vehicle for linguistic Computer statistical Sproat, Speech integration. Richard and Language Tom , 15(1):55 Emerson. 2003. –73. and international Chinese word The first In Proceedings segmentation bakeoff. SIGHAN Workshop of the Second Processing , July Chinese 11–12, on Language Sproat, Richard Sapporo, Corpus-based Chilin methods Shih. Japan. and in 2002. morphology Chinese 2002 and phonology. International In Conference Proceedings of Computational the Linguistics . on Sproat, Richard, Chen, Alan Shankar W. Kumar, Black, Mari Stanley Christoper Richards. Ostendorf, and Normalization Computer of 2001. Sproat, Speech non-standard and Languages words. Richard, Chilin Shih, William , 15:287–333. Nancy Chang. 1996. A Gale, and finite-state word-segmentation stochastic Chinese. Computational Linguistics algorithm for 22(3):377 –404. , Sproat, Computation Richard. . MIT 1992. Press, Morphology Cambridge, and Sproat, Richard Chilin Shih. 1990. A MA. method and statistical in Chinese for finding Computer word boundaries Processing Chinese text. 4:336–351. of and Oriental Languages Su, Keh-Yih and Chin-Hui weighted Lee. HMM 1994. Speech recognition projection using approaches. IEEE and subspace Trans., 2(1):69–79. Speech and Audio Processing , Sun, Zhou, Jian, and Jianfeng Chang-Ning Gao, Lei Huang. Zhang, 2002. Ming Chinese named entity identi model. fication using Proceedings class-based 2002 language Conference Computational the International In of 967 –973. on Linguistics , Sun, pages 2003. Jian, Chinese Ming Zhou, named and Jianfeng identi Gao. entity model. fication using International class-based Journal language Linguistics 8(2):1–28. Chinese of Computational and Language Processing , Sun, Tsou Maosong. ( ( ). Y (A f 2001. ~ ) and Benjamin K. ¹ and I í on ê Í v Ä ð review of evaluation Chinese). automatic segmentation (Contemporary Linguistics), S 3(1):22 ã í –32. f , Volume 31, Number 4 Sun, K. Maosong, Tsou. Dayang 1998. Shen, Chinese and word Benjamin without segmentation using lexicon In and hand-crafted training data. International Proceedings Conference of the 17th Linguistics , Montreal, on Computational Sun, 1265 –1271. Quebec, Canada, pages Maosong and Overlapping Zhengping in Zuo. 1998. Chinese) Quantitative ambiguity Chinese Computational text. (In Studies 323 Chinese –338. Language and on the , HK, Teahan, pages W. Ian J., Yingying Witten. 2000. Wen, A Rodger McNab, and compression- based algorithm for Chinese word segmentation. 26(3):375 –393. Computational Linguistics , Uchimoto, Hitoshi Kiyotaka, Isahara. Satoshi 2001. The Sekine, and word of Japanese problem: morphological unknown a maximum entropy analysis dictionary. using aided 2001 by a Conference In EMNLP Proceedings of the Vapnik, Vladimir 91–99. on , pages Theory. John Wiley N. 1998. Sons, Statistical Wu, and Zixin Jiang. & Inc. Learning Statistically-enhanced Andi new 2000. identification In in word Proceedings a rule-based Second Chinese system. Chinese Processing of Workshop the ACL 41–66. , Wu, pages of Andi. morphologically 2003. Customizable words segmentation Chinese. In International derived Journal in Computational Linguistics Chinese of Language Processing , 8(1):1–27. and Wu, Chinese Zimin and Gwyneth Tseng. 1993. text segmentation problems. for text JASIS, retrieval achievements 532–542. and 44(9): Xia, Chinese Fei. 1999. Treebank Segmentation Project . Technical guideline, University of Pennsylvania. report, Xue, Nianwen. 2003. Chinese word segmentation International Journal as character Computational tagging. Linguistics Chinese of Processing , and 8(1):29–48. Language Zhu, Chang-Ning Xiaodan, Mu Huang. Li, Jianfeng Single Gao, and Chinese named character Proceedings entity Second recognition. SIGHAN In Workshop of Chinese the July 11–12, on Sapporo, Language Japan, Proceeding 125 –132. , pages