<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="61"/><title>Which Performs Better on In-Vocabulary Word Segmentation: Based on Word or Character?</title><pubinfo>Sixth SIGHAN Workshop on Chinese Language Processing</pubinfo><author surname="Huang" givenname="Changning"><org  name="Which Performs Better on In-Vocabulary Word Segmentation:"/></author><author surname="Zhu" givenname="Jingbo"><org  name="Which Performs Better on In-Vocabulary Word Segmentation:"/></author><author surname="Wang" givenname="Zhenxing"><org  name="Which Performs Better on In-Vocabulary Word Segmentation:"/></author></firstpageheader><frontmatter><p><b>Which Performs Better on In-Vocabulary Word Segmentation:</b></p></frontmatter><abstract></abstract></header><body><section title="Based on Word or Character?"><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">12 2 1</doubt><p><b>Zhenxing Wang ' , Changning Huang and Jingbo Zhu</b></p><p>1 Institute of Computer Software and Theory, Northeastern University, Shenyang, China, 110004 2 Microsoft Research Asia, 49, Zhichun Road, Haidian District, Beijing, China, 100080</p><p>zxwang@ics.neu.edu.cn<u> </u>v-cnh@microsoft.com<u> </u>zhujingbo@mail.neu.edu.cn</p><p><b>Abstract*</b></p><p>Since the first Chinese Word Segmenta­tion (CWS) Bakeoff on 2003, CWS has experienced a prominent flourish be­cause Bakeoff provides a platform for the participants, which helps them rec­ognize the merits and drawbacks of their segmenters. However, the evaluation metric of bakeoff is not sufficient enough to measure the performance tho­roughly, sometimes even misleading. One typical example caused by this in­sufficiency is that there is a popular be­lief existing in the research field that segmentation based on word can yield a better result than character-based tag­ging (CT) on in-vocabulary (IV) word segmentation even within closed tests of Bakeoff. Many efforts were paid to bal­ance the performance on IV and out-of-vocabulary (OOV) words by combining these two methods according to this be­lief. In this paper, we provide a more de­tailed evaluation metric of IV and OOV words than Bakeoff to analyze CT me­thod and combination method, which is a typical way to seek such balance. Our evaluation metric shows that CT outper­forms dictionary-based (or so called word-based in general) segmentation on both IV and OOV words within Bakeoff * The work is done when the first author is working in MSRA as an intern.</p><p>closed tests. Furthermore, our analysis shows that using confidence measure to combine the two segmentation results should be under certain limitation.</p></section><section number="1" title="Introduction"><p>Chinese Word Segmentation (CWS) has been witnessed a prominent progress in the last three Bakeoffs (Sproat and Emerson, 2003), (Emer­son, 2005), (Levow, 2006). One of the reasons for this progress is that Bakeoff provides stan­dard corpora and objective metric, which makes the result of each system comparable. Through those evaluations researchers can recognize the advantage and disadvantage of their methods and improve their systems accordingly. Howev­er, in the evaluation metric of Bakeoff, only the overall F measure, precision, recall, IV (in-vocabulary) recall and OOV (out-of-vocabulary) recall are included and such a metric is not suffi­cient to give a completely measure on the per­formance, especially when the performance on IV and OOV word segmentation need to be eva­luated. An important issue is that segmentation based on which, word or character, can yield the better performance on IV words. We give a de­tailed explanation about this issue as following.</p><doubt alpha="66.5" length="328" tooSmall="False" monospace="0.0">Since CWS was firstly treated as a character-based tagging task (we call it "CT" for short he­reafter) in (Xue and Converse, 2002), this me­thod has been widely accepted and further de­veloped by researchers (Peng et al., 2004), (Tseng et al., 2005), (Low et al., 2005), (Zhao et al.,   2006).   Relatively   to dictionary-based</doubt><page local="2" global="62"/><p>segmentation (we call it "DS" for short hereaf­ter), CT method can achieve a higher accuracy on OOV word recognition and a better perfor­mance of segmentation in whole. Thus, CT has drawn more and more attention and became the dominant method in the Bakeoff 2005 and 2006.</p><p>Although CT has shown its merits in word segmentation task, some researchers still hold the belief that on IV words DS can perform bet­ter than CT even in the restriction of Bakeoff closed test. Consequently, many strategies are proposed to balance the IV and OOV perfor­mance (Goh et al., 2005), (Zhang et al., 2006a). Among these strategies, the confidence measure used to combine the results of CT and DS is a straight-forward one, which is introduced in (Zhang et al., 2006a). The basic assumption of such combination is that DS method performs better on IV words and Zhang derives this belief from the fact that DS achieves higher IV recall rate as Table 1 shows. In which AS, CityU, MSRA and PKU are four corpora used in Ba­keoff 2005 (also see Table 2 for detail). We pro­vide a more detailed evaluation metric to ana­lyze these two methods, including precision and F measure of IV and OOV respectively and our experiments show that CT outperforms DS on both IV and OOV words within Bakeoff closed test. The precision and F measure are existing metrics and the definitions of them are clear. Here we just employ them to evaluate segmenta­tion results. Furthermore, our error analysis on the results of combination reveals that confi­dence measure in (Zhang et al., 2006a) has a representation flaw and we propose an EIV tag method to revise it. Finally, we give an empiri­cal comparison between existing pure CT me­thod and combination, which shows that pure CT method can produce state-of-the-art results on both IV word and overall segmentation.</p><p>The rest of this paper is organized as fol­lows. In Section 2, we give a brief introduction to Zhang's DS method and subword-based tag­ging, which is a special CT method. And by comparing the results of this special CT method and DS according our detailed metric, we show that CT performs better on both IV and OOV. We review in Section 3 how confidence measure works and indicate its representation flaw. Fur­thermore, an "EIV" tag method is proposed to revise the confidence measure. In Section 4, the experimental results of existing pure CT method are demonstrated to compare with combination result, based on which we discuss the related work. In Section 5, we conclude the contribu­tions of this paper and discuss the future work.</p></section><section number="2" title="Comparison between DS and CT Based on Detailed Metric"><p>We proposed a detailed evaluation metric for IV and OOV word identification in this section and experiments based on the new metric show that CT outperforms DS not only on OOV words but also on IV words with F-measure of IV. All the experiments in this paper conform to the constraints of closed test in Bakeoff 2005 (Emerson, 2005). It means that any resource beyond the training corpus is excluded. We first review how DS and CT work and then present our evaluation metric and experiment results. There is one thing should be emphasized, by comparing DS and CT result we just want to verify that our new metric can show the performance on IV words more objectively. Since either DS or CT implementation has specific setting here we should not extend the comparison result to a general sense between those generative models and discriminative models.</p><subsection number="2.1" title="Dictionary-based segmentation"><p>For the dictionary-based word segmentation, we collect a dictionary from training corpus first. Instead of Maximum Match, trigram language model<footnote anchor="2"/> trained on training corpus is employed for disambiguation. During the disambiguation procedure, a beam search decoder is used to seek the most possible segmentation. Since the setting in our paper is consistent with the closed test of<page local="3" global="63"/></p><footnote label="2">Language model used in this paper is SLRIM from http://www . speech. sri.com/projects/srilm/</footnote><table caption="Table 1 IV and OOV recall in (Zhang et al., 2006a)" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Corpus</p></td><td class="cell"><p>riv Roov</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>DS       CT      DS CT</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>AS</p></td><td class="cell"><p>0.982   0.967   0.038 0.647</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CityU</p></td><td class="cell"><p>0.989   0.967   0.164 0.736</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MSRA</p></td><td class="cell"><p>0.993    0.972   0.048 0.716</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PKU</p></td><td class="cell"><p>0.981    0.955   0.408 0.754</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Bakeoff, we can only use the information we learn from training corpus though other open resources may be helpful to improve the perfor­mance further. For detail, the decoder reads cha­racters from the input sentence one at a time, and generates candidate segmentations incre­mentally. At each stage, the next incoming cha­racter is combined with an existing candidate in two different ways to generate new candidates: it is either appended to the last word in the candi­date, or taken as the start of a new word. This method guarantees exhaustive generation of possible segmentations for any input sentence. However, the exponential time and space of the length of the input sentence are needed for such a search and it is always intractable in practice. Thus, we use the trigram language model to se­lect top B (B is a constant predefined before search and in our experiment 3 is used) best candidates with highest probability at each stage so that the search algorithm can work in practice. Finally, when the whole sentence has been read, the best candidate with the highest probability will be selected as the segmentation result. Here, the term "dictionary-based" is exactly the method implemented in (Zhang et al., 2006a), it does not mean the generative language model in general.</p></subsection><subsection number="2.2" title="Character-based tagging"><p>Under CT scheme, each character in one sen­tence is labeled as „B' if it is the beginning of a word, „O' tag means the current character is a single-character word, other character is labeled as T. For example, "^r^H (whole China)" is labeled as " ^ (whole)/O ^ (central)/B B (country)/I".</p><p>In (Zhang et al., 2006a), the above CT me­thod is developed as subword-based tagging. First, the most frequent multi-character words and all single characters in training corpus are collected as subwords. During the subword-based tagging, a subword is viewed as an unit instead of several separate characters and given only one tag. For example, in subword-based tagging, "^^H (whole China)" is labeled as " ± (whole)/O +H (China)/O", if the word "+ H (China)" is collected as a subword. As the preprocessing, both training and test corpora are segmented by maximum match with subword set as dictionary. After this preprocessing, every sentence in both training and test corpora be­comes subword sequence. Finally, the tagger is trained by CRFs approach<footnote anchor="3"/> on the training data. Although word information is integrated into this method, it still works in the scheme of "IOB" tagging. Thus, we still call subword-based tagging as a special CT method and in the reminder of this paper "CT" means subword-based tagging in Zhang's paper and "Pure CT" means CT without subword.</p></subsection><subsection number="2.3" title="A detailed evaluation metric"><p>In this paper, data provided by Bakeoff 2005 is used in our experiments in order to compare with the published results in (Zhang et al., 2006a). The statistics of the corpora for Ba­keoff 2005 are listed in Table 2 (Emerson, 2005).</p><p>Evaluation standard is also provided by Ba­keoff, including overall precision, recall, F measure, IV recall and OOV recall (Sproat and Emerson, 2003), (Emerson, 2005). However, some important metrics, such as F measure and precision of both IV and OOV words are omit­ted, which are necessary when the performance of IV or OOV word identification need to be judged. Thus, in order to judge the results of each experiment, a more detailed evaluation with precision and F measure of both IV and OOV words included is used. To calculate the IV and OOV precision and recall, we firstly di­vide words of the segmenter's output and gold data into IV word and OOV word sets respec­tively with the dictionary collected from the training corpus. Then, for IV and OOV word sets respectively, the IV (or OOV) recall is the proportion of the correctly segmented IV (or</p><p>OOV) word tokens to all IV (or OOV) word to­kens in the gold data, and IV (or OOV) precision is the proportion of the correctly segmented IV (or OOV) word tokens to all IV (or OOV) word tokens in the segmenter's output.<page local="4" global="64"/> One thing have to be emphasized is that the single character in test corpus will be defined as OOV if it does not appear in training corpus. We will see later in this section, by this evaluation, some facts cov­ered by the bakeoff evaluation can be illustrated by our new evaluation metric.</p><footnote label="3">CRF tagger in this paper is implemented by CRF++ downloaded from http://crfpp.sourceforge.net/</footnote><table caption="Table 2 Corpora statistics of Bakeoff 2005" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Corpus</p></td><td class="cell"><p>Encoding</p></td><td class="cell"><p>#Training words</p></td><td class="cell"><p>#Test words</p></td><td class="cell"><p>OOV rate</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>AS</p></td><td class="cell"><p>Big5</p></td><td class="cell"><p>5.45M</p></td><td class="cell"><p>122K</p></td><td class="cell"><p>0.043</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CityU</p></td><td class="cell"><p>Big5</p></td><td class="cell"><p>1.46M</p></td><td class="cell"><p>41K</p></td><td class="cell"><p>0.074</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MSRA</p></td><td class="cell"><p>GB</p></td><td class="cell"><p>2.37M</p></td><td class="cell"><p>107K</p></td><td class="cell"><p>0.026</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PKU</p></td><td class="cell"><p>GB</p></td><td class="cell"><p>1.1M</p></td><td class="cell"><p>104K</p></td><td class="cell"><p>0.058</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Here, we repeat two experiments described in (Zhang et al., 2006a), namely dictionary-based approach and subword-based tagging. For CT method, top 2000 most frequent multi-character words and all single characters in training corpus are selected as subwords and the feature tem­plates used for CRF model is listed in Table 3. We present all the segmentation results in Table 6 to see the strength and weakness of each me­thod conveniently.</p><p>Based on IV and OOV recall as we show in Table 1, Zhang argues that the DS performs better on IV word identification while CT performs better on OOV words. But we can see from the results in Table 6 (the lines about DS and CT), the IV precision of DS approach is much lower than that of CT on all the four corpora, which also causes a lower F measure of IV. The reason for low IV precision of DS is that many OOV words are segmented into two IV words by DS. For example, OOV word " ffcPiffi (choral)" is segmented into" fPl(sing) 8<b>E </b>(class)" by DS. These wrongly identified IV words increase the number of all IV words in the segmenter's out­put and cause the low IV precision of the DS result. Since the F measure of IV is a more rea­sonable metric of performance of IV than IV recall only, Table 6 shows that CT method out­performs the DS on IV word segmentation over all four corpora. The comparison also shows that CT outperforms the DS on OOV and overall segmentation as well.</p></subsection></section><section number="3" title="Balance between IV and OOV Per­formance"><p>There are other strategies such as (Goh et al., 2005) trying to seek balance between IV and OOV performance. In (Goh et al, 2005), infor­mation in a dictionary is used in a statistical model. In this way, the dictionary-based ap­proach and the statistical model are combined. We choose the confidence measure to study be­cause it is straight-forward. We show in this sec­tion that there is a representation flaw in the formula of confidence measure in (Zhang et al., 2006a). And we propose an "EIV" tag method to solve this problem. Our experiments show that confidence measure with EIV tag outperforms CT and DS alone.</p><subsection number="3.1" title="Confidence measure"><p>Confidence Measure (CM) means to seek an optimal tradeoff between performance on IV and OOV words. The basic idea of CM comes from the belief that CT performs better on OOV words while DS performs better on IV words.</p><p>When both results of CT and DS are available, the CM can be calculated according to the fol­lowing formula in (Zhang et al., 2006a):</p><doubt alpha="63.6" length="44" tooSmall="False" monospace="0.0">CMftoi|w)= aCMiob(tiob|w)+ (1-a)S(tw,tiob)ng</doubt><p>Here, <i>w </i>is a subword, <i>tiob </i>is "IOB" tag given by CT and <i>tw </i>is "IOB" tag generated by DS. In the first term of the right hand side of the formu­la, <i>CM0ob (tob </i>| w) is the marginal probability of</p><p><i>tiob </i>(we call this marginal probability "MP" for short). And in the second term, <i>S(tw</i>, <i>tiob </i>) is a Kronecker delta function, returning 1 if and only if <i>tw </i>and <i>tiob </i>are identical, else returning 0. But <i>ifS(tw</i>, <i>tibb </i>) <i>ng = </i>1 , there is no requirement of re­placement at all. While if <i>S(tw</i>, <i>tob</i>)<i>ng = </i>0 , when <i>tw</i><i> * toob </i>, CM depends on the first term of its</p><p>right hand side only and <i>a </i>is unnecessary to be set as a weight. Finally, <i>a </i>in the formula is a weight to seek balance between CT tag and DS tag. Another parameter here is a threshold <i>t </i>for the CM. If CM is less than <i>t</i>, <i>tw </i>replaces <i>tiob </i>as the final tag, otherwise <i>tiob </i>will be remained as the final tag.<page local="5" global="65"/> However, two parameters in the CM, namely <i>a </i>and <i>t</i>, are unnecessary, because when MP is greater than or equal to <i>t </i>/a, <i>tiob</i>will be kept, otherwise it will be replaced with <i>tw </i>.   Thus, the CM ultimately is the marginal probability of the "IOB" tag (MP). In the expe­riment of this paper, MP is used as CM because it is equivalent to Zhang's CM but more conve­nient to express.</p><table caption="Table 3 Feature templates used for CRF in our experiments" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Type</p></td><td class="cell"><p>Feature</p></td><td class="cell"><p>Function</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Unigram</p></td><td class="cell"><p>C-2, C-i, Co, Ci, <i>C</i><i>2</i></p></td><td class="cell"><p>Previous two, current and next two subword</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Bigram</p></td><td class="cell"><p><i>C</i><i>-2 </i><i>C</i><i>-i, </i><i>C</i><i>-i </i><i>C</i><i>o, </i><i>C</i><i>o </i><i>C</i><i>i, </i><i>C</i><i>i </i><i>C</i><i>2</i></p></td><td class="cell"><p>Two adjacent subwords</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Jump</p></td><td class="cell"><p><i>C</i><i>-i </i><i>C</i><i>i</i></p></td><td class="cell"><p>Previous character and next subwords</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></subsection><subsection number="3.2" title="Experiments and error analysis about combination"><p>We repeat the experiments about CM in Zhang's paper (Zhang et al., 2006a) and show that there is a representation flaw in the CM formula. Fur­thermore, we propose an EIV tag method to make CM yield a better result.</p><p>In this paper, <i>a </i>= 0.8 and t = 0.7 (Parameters in two papers, Zhang et al. 2006a and Zhang et al. 2006b, are different. And our parameters are consistent with Zhang et al. 2006b which is con­firmed by Dr Zhang through email) are used in CM, namely MP= 0.875 is the threshold. Here, in Table 4, we provide some statistics on the results of CT when MP is less than 0.875. From Table 4 we can see that even with MP less than 0.875, most of the subwords are still tagged cor­rectly by CT and should not be revised by DS result. Besides, lots of the subwords with low MP contained by OOV words in test data, espe­cially for the corpus whose OOV rate is high (i.e. on CityU corpus more than one third sub-words with low MP belong to OOV word) and performance on OOV recognition is the advan­tage of CT rather than that of DS approach. Thus when combining the results of the two methods, it is the <i>tiob </i>should be maintained if the subword is contained by an OOV word. Therefore, the CM formula seems somewhat unreasonable.</p><p>The error analysis about how many original errors are eliminated and how many new errors are introduced by CM is provided in Table 5 (the columns about CM). Table 5 illustrates that, af­ter combining the two results, most original er­rors on IV words are corrected because DS can achieve higher IV recall as described in Zhang's paper. But on OOV part, more new errors are introduced by CM and these new errors decrease the precision of the IV words. For example, the OOV words <i>"WMAm </i>(guard member)" and " iitifl? (design fee)" is recognized correctly by CT but with low CM. In the combining proce­dure, these words are wrongly split as IV errors: "Wi (guard) <i>$km </i>(member)" and "i&amp;if (de­sign) H (fee)". Thus, for two corpora (i.e. CityU and AS), F measure of IV and overall F measure decreases since there are more new er­rors introduced than original ones eliminated and only on the other two corpora (MSRA and PKU), overall F measure of combination method is higher than CT alone, which is shown in Ta­ble 6 by the lines about combination.</p></subsection><subsection number="3.3" title="EIV tag method"><p>Since combining the two results by CM may produce an even worse performance in some case, it is worthy to study how to use this CM to get an enhanced result. Intuitively, if we can change only the CT tags of the subwords which contained in IV word while keep the CT tags of those contained in OOV words unchanged, we will improve the final result according to our error analysis in Table 5. Unfortunately, only from the test data, we can get the information whether a subword contained in an IV word, just as what we do to get Table 4. However, we can get an approximate estimation from DS result. When using subwords to re-segment DS result<footnote anchor="4"/>, all the fractions re-segmented out of multiple-character words, including both multiple-character words and single characters, will be given an "EIV" tag, which means that the cur­rent multiple-character word or single character is contained in an IV word with high probability. For example, "Ajj'Sif! (human resource)" in DS result is a whole word. However, only <i>"rMM </i>(resource)" belongs to the subword set, so dur­ing the re-segmentation "Ajj'Sif! (human re­source)" will be re-segmented as "A (people) (force) <i>rMM </i>(resource)". All these three frac­tions will be labeled with an "EIV" tag respec­tively. It is reasonable because all the multiple-character words in the DS result can match an IV word. After this procedure, when combining #original errors eliminated on IV #original errors eliminated on OOV <u>#original errors eliminated totally </u>#new errors introduced on IV #new errors introduced on OOV <u># new errors introduced totally</u> the two results, only the CT tag with EIV tags and low MP will be replaced by DS tag, other­wise the original CT tag will be maintained.<page local="6" global="66"/> Un­der this condition the errors introduced by OOV will not happen and enhanced results are listed in Table 6 lines about EIV. We can see that on all four corpora the overall F measure of EIV result is higher than that of CT alone, which show that our EIV method works well. Now, let's check what changes happened in the num­ber of error tags after EIV condition added into the CM. We can see from the Table 5 columns about EIV, there are more errors eliminated than the new errors introduced after EIV condition added into CM and most CT tags of subwords contained in OOV words maintained unchanged as we supposed. And then, our results (in Table 6 lines about EIV) are comparable with that in Zhang's paper. Thus, there may be some similar strategies in Zhang's CM too but not presented in Zhang's paper.</p><footnote label="4">For the detail, please refer to (Zhang et al., 2006a).</footnote><doubt alpha="100.0" length="6" tooSmall="False" monospace="0.0">Corpus</doubt><doubt alpha="100.0" length="6" tooSmall="False" monospace="0.0">Method</doubt></subsection></section><section number="4" title="Discussion and Related Works"><p>Although the method such as confidence meas­ure can be helpful at some circumstance, our experiment shows that pure character-based tag­ging (pure CT) can work well with reasonable features and tag set. In (Zhao et al., 2006), an enhanced CRF tag set is proposed to distinguish different positions in the multi-character words when the word length is less than 6. In this me­thod, feature templates are almost the same as shown in Table 3 with a 3-character window and a 6-tag set {B, B2, B3, M, E, O} is used. Here, tag B and E stand for the first and the last posi­tion in a multi-character word, respectively. S stands up a single-character word. B2 and B3 stand for the second and the third position in a multi-character word, whose length is larger than two-character or three-character. M stands for the fourth or more rear position in a multi­character word, whose length is larger than four-character.</p><p>In Table 6, the lines about "pure CT" provide the results generated by pure CT with 6-tag set. We can see from the Table 6 this pure CT ap­proach achieves the state-of-the-art results on all the corpora. On three of the four corpora (AS,</p><p>MSRA and PKU) this pure CT method gets the best result. Even on IV word, this pure CT ap­proach outperforms Zhang's CT method and produces comparable results with combination with EIV tags, which shows that pure CT me­thod can perform well on IV words too. Moreo­ver, this character-based tagging approach is more clear and simple than the confidence measure method.</p><p>Although character-based tagging became mainstream approach in the last two Bakeoffs, it does not mean that word information is valueless in Chinese word segmentation. A word-based perceptron algorithm is proposed recently (Zhang and Clark, 2007), which views Chinese word segmentation task from a new angle in­stead of character-based tagging and gets com­parable results with the best results of Bakeoff.</p><table caption="Table 4 Results of CT when MP is less than 0.875" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Corpus</p></td><td class="cell"><p>AS</p></td><td class="cell"><p>CityU</p></td><td class="cell"><p>MSRA</p></td><td class="cell"><p>PKU</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p># subword tokens belong to IV</p></td><td class="cell"><p>10010</p></td><td class="cell"><p>4404</p></td><td class="cell"><p>9552</p></td><td class="cell"><p>9619</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p># subword tokens belong to IV and tagged correctly by CT</p></td><td class="cell"><p>7452</p></td><td class="cell"><p>3434</p></td><td class="cell"><p>7452</p></td><td class="cell"><p>7213</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p># subword tokens belong to IV and tagged wrongly by CT</p></td><td class="cell"><p>2558</p></td><td class="cell"><p>970</p></td><td class="cell"><p>2100</p></td><td class="cell"><p>2406</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p># subword tokens belong to OOV</p></td><td class="cell"><p>5924</p></td><td class="cell"><p>2524</p></td><td class="cell"><p>2685</p></td><td class="cell"><p>3580</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p># subword tokens belong to OOV and tagged correctly by CT</p></td><td class="cell"><p>3177</p></td><td class="cell"><p>1656</p></td><td class="cell"><p>1725</p></td><td class="cell"><p>2208</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p># subword tokens belong to OOV and tagged wrongly by CT</p></td><td class="cell"><p>2747</p></td><td class="cell"><p>868</p></td><td class="cell"><p>960</p></td><td class="cell"><p>1372</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 5 Error analysis of confidence measure with and without EIV tag" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>AS</p></td><td class="cell"><p>CityU</p></td><td class="cell"><p>MSRA</p></td><td class="cell"><p>PKU</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CM</p></td><td class="cell"><p>EIV</p></td><td class="cell"><p>CM</p></td><td class="cell"><p>EIV</p></td><td class="cell"><p>CM</p></td><td class="cell"><p>EIV</p></td><td class="cell"><p>CM</p></td><td class="cell"><p>EIV</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1905</p></td><td class="cell"><p>1003</p></td><td class="cell"><p>904</p></td><td class="cell"><p>469</p></td><td class="cell"><p>1959</p></td><td class="cell"><p>1077</p></td><td class="cell"><p>1923</p></td><td class="cell"><p>1187</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>755</p></td><td class="cell"><p>75</p></td><td class="cell"><p>155</p></td><td class="cell"><p>80</p></td><td class="cell"><p>104</p></td><td class="cell"><p>30</p></td><td class="cell"><p>230</p></td><td class="cell"><p>76</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>r</b></p></td><td class="cell"><p>2660</p></td><td class="cell"><p><b>T</b></p></td><td class="cell"><p>1078</p></td><td class="cell"><p><b>T</b></p></td><td class="cell"><p>1059</p></td><td class="cell"><p><b>T</b></p></td><td class="cell"><p>549</p></td><td class="cell"><p><b>T</b></p></td><td class="cell"><p>2063</p></td><td class="cell"><p><b>T</b></p></td><td class="cell"><p>1107</p></td><td class="cell"><p><b>T</b></p></td><td class="cell"><p>2153</p></td><td class="cell"><p><b>T</b></p></td><td class="cell"><p>1263</p></td><td class="cell"><p><b>1</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>441</p></td><td class="cell"><p>185</p></td><td class="cell"><p>80</p></td><td class="cell"><p>50</p></td><td class="cell"><p>148</p></td><td class="cell"><p>68</p></td><td class="cell"><p>211</p></td><td class="cell"><p>118</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2487</p></td><td class="cell"><p>77</p></td><td class="cell"><p>1320</p></td><td class="cell"><p>103</p></td><td class="cell"><p>1517</p></td><td class="cell"><p>57</p></td><td class="cell"><p>1681</p></td><td class="cell"><p>58</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>E</b></p></td><td class="cell"><p>2928</p></td><td class="cell"><p><b>I</b></p></td><td class="cell"><p>262</p></td><td class="cell"><p><b>I</b></p></td><td class="cell"><p>1400</p></td><td class="cell"><p><b>I</b></p></td><td class="cell"><p>153</p></td><td class="cell"><p><b>I</b></p></td><td class="cell"><p>1665</p></td><td class="cell"><p><b>I</b></p></td><td class="cell"><p>125</p></td><td class="cell"><p><b>I</b></p></td><td class="cell"><p>1892</p></td><td class="cell"><p><b>I</b></p></td><td class="cell"><p>176</p></td><td class="cell"><p><b>1</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="7" global="67"/><doubt alpha="100.0" length="6" tooSmall="False" monospace="0.0">Corpus</doubt><doubt alpha="100.0" length="5" tooSmall="False" monospace="0.0">CityU</doubt><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">MSRA</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">PKU</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">ÄS</doubt><p>Method PS</p><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">CT_</doubt><p>Combination <u>EIV tag " Pure CT</u></p><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">DS_</doubt><p>Combination <u>EIV tag " </u>Pure CT</p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">R</doubt><doubt alpha="0.0" length="118" tooSmall="False" monospace="0.0">0.943 0.954 0.958 0.960 0.958 0.928 0.947 0.954 0.953 0.947 0.969 0.963 0.977 0.972 0.972 0.948 0.944 0.955 0.9500.946</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">P</doubt><doubt alpha="0.0" length="118" tooSmall="False" monospace="0.0">0.881 0.938 0.929 0.942 0.947 0.848 0.940 0.922 0.949 0.948 0.927 0.964 0.961 0.970 0.975 0.911 0.945 0.942 0.9520.957</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">F</doubt><doubt alpha="0.0" length="118" tooSmall="False" monospace="0.0">0.911 0.946 0.943 0.951 0.953 0.886 0.944 0.938 0.951 0.948 0.947 0.963 0.969 0.971 0.973 0.929 0.945 0.949 0.9510.951</doubt><doubt alpha="0.0" length="118" tooSmall="False" monospace="0.0">0.984 0.967 0.980 0.973 0.971 0.989 0.963 0.984 0.970 0.967 0.994 0.970 0.990 0.980 0.978 0.981 0.955 0.973 0.9610.956</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">Piv</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.892</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.960</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.945</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.962</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.963</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.865</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.964</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.938</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.968</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.973</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.930</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.979</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.970</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.982</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.986</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.920</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.966</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.953</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">Fiv</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.935</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.967</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.923</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.961</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.969</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.975</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.980</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.981</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.950</doubt><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">roov</doubt><doubt alpha="0.0" length="118" tooSmall="False" monospace="0.0">0.044 0.666 0.487 0.667 0.682 0.162 0.739 0.581 0.744 0.692 0.036 0.698 0.511 0.696 0.750 0.403 0.763 0.664 0.7680.672</doubt><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">poov</doubt><doubt alpha="0.0" length="118" tooSmall="False" monospace="0.0">0.217 0.606 0.593 0.624 0.618 0.353 0.717 0.693 0.750 0.660 0.358 0.662 0.653 0.679 0.632 0.711 0.727 0.782 0.7530.580</doubt><p>Table 6 Results of different approach used in our experiments (White background lines the results we repeat Zhang's methods and they have some trivial difference with Table</p><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">foov</doubt><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">"0.076</doubt><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">"0.635</doubt><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">"0.535</doubt><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">"0.645</doubt><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">"0.648</doubt><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">"0.223</doubt><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">"0.728</doubt><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">"0.632</doubt><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">"0.747</doubt><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">"0.676</doubt><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">"0.066</doubt><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">"0.680</doubt><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">"0.574</doubt><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">"0.688</doubt><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">"0.686</doubt><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">"0.515</doubt><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">"0.745</doubt><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">"0.718</doubt><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">"0.760</doubt><doubt alpha="23.1" length="13" tooSmall="False" monospace="0.0">"0.623are 1.)</doubt><p>Therefore, the most important thing worth to pay attention in future study is how to integrate lin­guistic information into the statistical model effec­tively, no matter character or word information.</p><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">5</doubt><p><b>Conclusions and Future Work</b></p><p>In this paper, we first provided a detailed evalua­tion metric, which provides the necessary infor­mation to judge the performance of each method on IV and OOV word identification. Second, by this evaluation metric, we show that character-based tagging outperforms dictionary-based seg­mentation not only on OOV words but also on IV words within Bakeoff closed tests. Furthermore, our experiments show that confidence measure in Zhang's paper has a representation flaw and we propose an EIV tag method to revise the combina­tion. Finally, our experiments show that pure cha­racter-based approach also can achieve good IV word and overall performance. Perhaps, there are two reasons that existing combination results don't outperform the pure CT. One is that most information contained in statistic language model is already captured by the CT feature templates in CRF framework. The other is that confidence measure may not be the effective way to combine the DS and CT results.</p><p>In the fUture work, our research will focus on how to integrate word information into CRF fea­tures rather than using it to modify the results of CRF tagging. In this way, we can capture the word information meanwhile avoid destroying the optimal output of CRF tagging.</p><p><b>Acknowledgement</b></p><p>The authors appreciate Dr. Hai Zhao in City Uni­versity of Hong Kong and Dr. Ruiqiang Zhang in Spoken Language Communications Lab, ATR, Japan providing a lot of help for this paper. Thank those reviewers who gave the valuable comment to improve this paper.</p></section><references><p>Thomas Emerson. 2005. The Second International Chi­nese Word Segmentation Bakeoff. In <i>Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing, </i>pages 123-133, Jeju Island, Korea:</p><p>Chooi-Ling Goh, Masayuku Asahara and Yuji Matsu-moto. 2005. Chinese Word Segmentatin by Classifi­cation of Characters. <i>Computational Linguistics and</i> <i>Chinese Language Processing, </i>Vol.<page local="8" global="68"/> 10(3): pages 381-396.</p><p>Gina-Anne Levow. 2006. The Third International Chi­nese Language Processing Bakeoff: Word Segmen­tation and Named Entity Recognition. In <i>Proceed­ings of the Fifth SIGHAN Workshop on Chinese Language Processing </i>, pages 108-117, Sydney: Ju­ly.</p><p>Jin Kiat Low, Hwee Tou Ng, and Wenyuan Guo. 2005. A Maximum Entropy Approach to Chinese Word Segmentation. In <i>Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing, </i>pages 161­164, Jeju Island, Korea.</p><p>Fuchun Peng, Fangfang Feng, and Andrew McCallum.</p><p>2004. Chinese segmentation and new word detection using conditional random fields. In COLING 2004, pages 562-568. Geneva, Switzerland.</p><p>Richard Sproat and Thomas Emerson. 2003. The First International Chinese Word Segmentation Bakeoff. In <i>Proceedings of the Second SIGHAN Workshop on Chi­nese Language Processing, </i>pages 133-143, Sapporo, Ja­pan: July 11-12,</p><p>Huihsin Tseng, Pichuan Chang et al. 2005. A Conditional</p><p>Random Field Word Segmenter for SIGHAN Bakeoff 2005. In <i>Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing, </i>pages 168-171, Jeju Island, Korea.</p><p>Neinwen Xue and Susan P. Converse. 2002. Combin­ing Classifiers for Chinese Word Segmentation. In <i>Proceedings of the First SIGHAN Workshop on Chinese Language Processing, </i>pages 63-70, Taipei, Taiwan.</p><p>Ruiqiang Zhang, Genichiro Kikui and Eiichiro Sumita. 2006a. Subword-based Tagging by Conditional Random Fields for Chinese Word Segmentation. In <i>Proceedings of the Human Language Technology Conference of the NAACL, Companion volume, </i>pag­es 193-196. New York, USA.</p><p>Ruiqiang Zhang, Genichiro Kikui and Eiichiro Sumita. 2006b. Subword-based Tagging for Confidence-dependent Chinese Word Segmentaion. In <i>Proceed­ings of the COLING/ACL, Main Conference Poster Sessions, </i>pages 961-968. Sydney, Australia.</p><p>Yue Zhang and Stephen Clark. 2007. Chinese Segmenta­tion with a Word-Based Perceptron Algorithm. In <i>Proceedings of the 45th Annual Meeting of the As­sociation for Computational Linguistics, </i>pages 840­847. Prague, Czech Republic.</p><p>Hai Zhao, Changning Huang et al. 2006. Effective Tag Set Selection in Chinese Word Segmentation via Conditional Random Field Modeling. In <i>Proceed-</i> <i>ings of PACLIC-20.</i><i> </i>pages 87-94. Wuhan, China, Novemeber.</p></references></body></article>