Chinese Word Segmentation based on Maximum Matching and Word Binding Force

Abstract

ally, all but one of these alternatives arc syntactically and/or semantically incorrect. This is l;he A Chinese word Seglnentation algorithm case because unlike texts in English, Chinese texl;s based on forward icnaxinnlln matching have no word nlarkers. A tirst step towmds buihtand word binding force is t)roposed in ing a language model based on N-gram statistics is this pai)er. This algorithm Iilays a key to de, vek)p an etIMent lexical analyser to id(!ntify role in post-processing the outtmt of a all the words in the, corpus. character or st/eech recognizer in deterWord segmentation algorithlns behmg to one of mining the proper word sequence c(/rretwo types ill general, viz., the structural (Wang et st)onding to an input line of cha.raeter al., 1991) and the statistical type (Lua, 1990)(Lua images or a speech wav(~,fol'tn. ~FO supand Gan, 1994)(Sproat and Shih, 1990) rt;specport this algorithm, a text; (:orims of over tively. A structural algorithm resolves segmenta63 millions characters is employed to ention mnbiguities by examining the structural rclarich an 80,O00-words lexi(:on in terlns of tionships between words, while a statistical algo-its word entries and word binding forces. rithm compares the usage flequencies of the words As it stands now, given an input line of and their ordered combinations inste, ad. Both aptext, the word segmentor can proce, ss on proaches ln~ve serious liinitat;ions. the average 210,000 characters per se(:ond when running on an IBM RISC Sys2 Maximum Matching Method for tem/6000 3BT workstation with a col Segmentation rect word identitication rate of 99.74%. Maximum matching (l,iu et al., /994) is one of the most I)opular structural segmentation algorithms 1 Introduction for Chinese texts. This method favours long words A language model as a t)ost-processor isesse, ntial an(1 is a gree(ty algorithm by (lesign, hen(:e, subto a recognizer of speech or characters in order optimal. Segmenl;ation may start from either end to determine the approi)riate word se, que, n(:e and of the line without any difference in segmentahenc.e the semantics of an inI)ut line of text or uttion results. In this paper, the forward direction terance. It is well known that an N-gram statistics is adopted. The major advantage of inaximum language model is just as effective as, t)ut nmch matching is its etHciency while its segmentation more eificient than, a syntactk:/semantic analyser accuracy can be expected to lie around 95%. in determining the correct word sequence. A necessary condition to successflfl collection of N-gram 3 Word Frequency Method for statistics is the existence of a coInprehensive l , xSegmentation icon and a large text corpus. The latter must tie lexically analysed in order to identify all the In this statistical approach in terms of word frewords, from which, N-gram statistics can be dequencies, a lexicon needs not only a rich repertoire rived. of word entries, lint also the usage frequency of About 5,000 characters are being used in mode, ach word. To segment a line of text, each possiern Chinese and they are the building blocks of all ble segmentation alternative is ewduated accordwor(ls. Ahnost every character is a word and inost ing to the product of the word fi'equencies of the words are of one or two characters long but there words Seglnented. The word sequence, with the are also abundant wor(ls longer than two charachighest fi'equency product is accepted a.s correct. ters. Before it; is seginented into words, a line of This method is simple but its a(:curacy (h,,lmnds text is just a sequence of characters and there are heavily on the accuracy of the usage fi'equencies. numerous word segmentation alternatives. UsuThe usage frequency of a word differs greatly from