<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="82"/><title>A Two-Stage Approach to Chinese Part-of-Speech Tagging</title><pubinfo>Sixth SIGHAN Workshop on Chinese Language Processing</pubinfo><author surname="Chen" givenname="Aitao"><org  name="Yahoo! Inc." country="USA" city="Santa Clara"/></author><author surname="Zhang" givenname="Ya"><org  name="Yahoo! Inc." country="USA" city="Santa Clara"/></author><author surname="Sun" givenname="Gordon"><org  name="Yahoo! Inc." country="USA" city="Santa Clara"/></author></firstpageheader><frontmatter><p><b>A Two-Stage Approach to Chinese Part-of-Speech Tagging</b></p><p><b>Aitao Chen</b></p><p>Yahoo! Inc. 701 First Avenue Sunnyvale, CA 94089</p><p>aitao@yahoo-inc.com</p><p><b>Ya Zhang</b></p><p>yazhang@yahoo-inc.com</p><p><b>Gordon Sun</b></p><p>gzsun@yahoo-inc.com</p></frontmatter><abstract>This paper describes a Chinese part-of-speech tagging system based on the maxi­mum entropy model. It presents a novel two-stage approach to using the part-of-speech tags of the words on both sides of the current word in Chinese part-of-speech tagging. The system is evaluated on four corpora at the Fourth SIGHAN Bakeoff in the close track of the Chinese part-of-speech tagging task. </abstract></header><body><section number="1" title="Introduction"><p>A part-of-speech tagger typically assigns a tag to each word in a sentence sequentially from left to right or in reverse order. When the words are tagged from left to right, the part-of-speech tags assigned to the previous words are available to the tagging of the current word, but not the tags of the following words. And when words are tagged from right to left, only the tags of the words on the right side are available to the tagging of the current word. We expect the use of the tags of the words on both sides of the current word should improve the tagging of the current word. In this paper, we present a novel two-stage approach to using the tags of the words on both sides of the current word in tagging the current word. We train two maxi­mum entropy part-of-speech taggers on the same training data. The difference between the two tag­gers is that the second tagger uses features involv­ing the tags of the words on both sides of the cur­rent word, while the first tagger uses the tags of only the previous words. Both taggers assign tags to words from left to right. In tagging a new sen­tence, the first tagger is applied to the testing data, and then the second tagger is applied to the output of the first tagger to produce the final results.</p><p>We participated in the Chinese part-of-speech tagging task at the Fourth International Chinese Language Processing Bakeoff. Our Chinese part-of-speech taggers were trained only on the training data provided to the participants, and evaluated on four corpora in the close track of the part-of-speech tagging task. The words in both the training and testing data sets are already segmented into words.</p></section><section number="2" title="Maximum Entropy POS Tagger"><p>Maximum entropy model is a machine learning algorithm that has been applied to a range of natu­ral language processing tasks, including part-of-speech tagging (Ratnaparkhi, 1996). Our Chinese part-of-speech taggers are based on the maximum entropy model.</p><subsection number="2.1" title="Maximum Entropy Model"><p>The conditional maximum entropy model (Berger, et. al., 1996) has the form <i>p</i>(y<i> </i>Ix)<i>=ik</i><i> </i>exp(Z <i>Ak</i><i>fk </i>(x<i> </i><i>y))</i> where Z(x) = <i>p(y </i>| x)is a normalization fac­tor, and <i>Ak</i><i> </i>is a weight parameter associated with feature <i>fk </i>(x, <i>y).</i><i></i><i> </i>In the context of part-of-speech tagging, y is the POS tag assigned to a word, and x represents the contextual information regarding the word in consideration, such as the surrounding words. A feature is a real-valued, typically binary, function. For example, we may define a binary fea­ture which takes the value 1 if the current word of X is 'story' and its POS tag is 'NNS'; and 0 other­wise. Given a set of training examples, the log likelihood of the model with Gaussian prior (Chen and Rosenfeld, 1999) has the form<page local="2" global="83"/></p><doubt alpha="100.0" length="1" tooSmall="True" monospace="0.0">k</doubt><doubt alpha="50.0" length="34" tooSmall="False" monospace="0.0">L(A)= ^ logp( y(0| x})-YJri:+const</doubt><p>Malouf (2002) compared iterative procedures such as <i>Generalized Iterative Scaling (GIS) </i>and <i>Im­proved Iterative Scaling (IIS) </i>with numerical opti­mization techniques like limited-memory BFGS (L-BFGS) for estimating the maximum entropy model parameters and found that L-BFGS outper­forms the other methods. The use of L-BFGS re­quires the computation of the gradient of the log likelihood function. The first derivative with re­spect to parameter <i>Xk</i><i> </i>is given by where the first term <i>E</i><i>p</i><i>fk</i><i> </i>is the feature expecta­tion with the empirical model, and the second term <i>Epfk</i><i> </i>is the feature expectation with respect to the model. In our model training, we used L-BFGS to estimate the model parameters by maximizing <i>L(A)</i><i> </i>on the training data.</p><doubt alpha="60.0" length="5" tooSmall="False" monospace="0.0">dL(l)</doubt></subsection><subsection number="2.2" title="Features"><p>The feature templates used in our part-of-speech taggers are presented in Table 1 and Table 2.</p><p>The features are grouped into four categories. The first category contains features involving word to­kens only; the second category consists of features involving tags only; the third category has features involving both word tokens and tags. And the last category has four special features. In the feature templates, wi denotes the current word, wi-2 the second word to the left, wi-1 the previous word, wi+1 the next word, wi+2 the second word to the right of the current word, and ti denotes the part-of-speech tag assigned to the word wi. The <i>FirstChar </i>refers to the initial character of a word, and the <i>LastChar </i>the final character of a word. The <i>Length </i>denotes the length of a word in terms of byte. And the feature <i>ForeignWord </i>indicates whether or not a word is a foreign word. Table 2 shows additional feature templates involving the part-of-speech tags of the following one or two words. The features involving the tags of the words in the right con­texts are used only in the second maximum entropy POS tagger. Features are generated from the train­ing data according to the feature templates pre­sented in Table 1 and Table 2.</p></subsection><subsection number="2.3" title="Training Models"><p>The four training corpora we received for the Chi­nese part-of-speech tagging task include the Aca-demia Sinica corpus <b>(CKIP), </b>the City University of Hong Kong corpus <b>(CityU), </b>the National Chi­nese Corpus <b>(NCC), </b>and the Peking University corpus <b>(PKU). </b>The CKIP corpus and the CityU corpus contain texts in traditional Chinese, while the NCC corpus and the PKU corpus contain texts in simplified Chinese. The texts in all four training corpora are segmented into words according to different word segmentation guidelines. And the words in all training corpora are labeled with part-of-speech tags using different tag sets.</p><p>Two maximum entropy POS taggers were trained on each of the four corpora using our own implementation of the maximum entropy model. The first-stage POS tagger was trained with only the feature templates presented in Table 1, while the second-stage POS tagger with the feature tem­plates presented in both Table 1 and Table 2.</p><p>All the first-stage POS taggers, one for each corpus, were trained with the same feature tem­plates shown in Table 1, and all the second-stage POS taggers were trained with the same feature templates shown in Table 1 and Table 2. The fea­ture templates are not necessarily optimal for each individual corpus. For simplicity, we chose to ap­ply the same feature templates to all four corpora.</p><doubt alpha="50.0" length="2" tooSmall="False" monospace="0.0">S3</doubt><doubt alpha="64.0" length="25" tooSmall="False" monospace="0.0">Eppfkix,y)-Epfk(x,y)--k­o</doubt><table caption="Table 2: Additional feature templates used in the second stage POS tagger." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Word</p></td><td class="cell"><p><i>Wi-2,           Wi , Wi+2</i></p><p><i>Wi-2Wi-l, Wi-lWi, WiWi+l, </i>Wi+lWi+2,</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Tag</p></td><td class="cell"><p><i>ti-l, ti-2ti-l</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Word/Tag</p></td><td class="cell"><p><i>h</i><i>-lWr , ti-2Wi</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Special</p></td><td class="cell"><p><i>FirstChar, LastChar, Length, ForeighWord</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Table 1: Feature templates used in the first stage POS tagger.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Tag</p></td><td class="cell"><p><i>ti+l, </i>ti+lti+2 ' <i>ti-lti+l</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Word/Tag</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="3" global="84"/><p>The same parameter settings were applied in the training of all eight POS taggers. More specifi­cally, no feature selection was performed. All fea­tures, including features occurring just once in the training data, were retained. The sigma square</p><p><b><i>a</i></b><footnote anchor="2"/>was set to 5.0. And the training process was terminated when the ratio of the likelihood differ­ence between the current iteration and the previous iteration over the likelihood of the current iteration is below the pre-defined threshold or the maximum number of iterations, which was set to 400, is reached. Both the first-stage POS tagger and the second-stage POS tagger were trained on the same corpus.</p></subsection><subsection number="2.4" title="Testing the Models"><p>The POS tagger assigns a part-of-speech tag to each word in a new sentence such that the tag se­quence maximizes the probability <i>p(Y\X), </i>where X is the input sentence, and Y the POS tags assigned to X. The decoder implements the beam search procedure described in (Ratnaparkhi, 1996). At each word position, the decoder keeps the top n best tag sequences up to that position. The decoder also uses a word/tag dictionary, consisting of the words in the training data and the tags assigned to each word in the training data. During the decod­ing phase, if a word in the new sentence is found in the training data, only the tags that are assigned to that word in the training corpus are considered. Otherwise, all the tags in the tag set are considered for a new word. So the tagger will not assign to a word, found in the training data, a tag that is never assigned to that word in the training data, even if that word should be assigned a new tag that was never assigned to the word in the training data. A word/tag dictionary is automatically built by col­lecting all the words in the training corpus and the tags assigned to every word in the training corpus.</p><p>The final output is produced in two steps. The first-stage POS tagger is applied on the testing data, and then the second-stage POS tagger is applied on the output of the first POS tagger. The second-stage tagger uses features involving POS tags of the following one or two words. The features in­volving the tags of following one or two words may be erroneous, since the tags assigned to the following one or two words by the first-stage tag­ger may be incorrect.</p><p><b>S    Evaluation Results</b></p><p>Five corpora are provided for the Chinese part-of-speech tagging task at the forth SIGHAN bakeoff. We selected four corpora, two in simplified Chi­nese and two in traditional Chinese.</p><p>Table 3 shows the training corpus size, the tagset size, and the average number of tags per token type. The NCC tagset has 60 tags, but nine of the tags occurred only once in the training corpus. In all four corpora, most of the unique tokens have only a single tag. The percentage of token types having single tag is 83.29% in CityU corpus; 91.09 in CKIP corpus; 94.67 in NCC corpus; and 90.27% in PKU corpus. The proportion of token types having single tag in CityU corpus is much lower than in NCC corpus. In the NCC corpus, the organization names, location names, and a sequence of English words are all treated as single token, and these long single tokens are not ambiguous and are assigned to a single part-of-speech tag in the corpus.</p><table caption="Table 4: The testing data size and the baseline per­formance."></table><p>The baseline performance is computed by assign­ing the most likely tag to each word in the testing data. When a word in the testing data is found in the training corpus, it is assigned the tag that is most frequently assigned to that word in the train­ing corpus. A new word in the testing data is as­signed the most frequent tag found in the training corpus, which is the common noun in all four cor­pora. The baseline performances of the four testing data sets are presented in Table 4, which also shows the percentage of new token/tag in the test­ing data sets.<page local="4" global="85"/></p><table caption="Table 3: Training corpus size." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Corpus</p></td><td class="cell"><p>Training size</p></td><td class="cell"><p>Tagset</p></td><td class="cell"><p>No. of tags</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>(tokens)</p></td><td class="cell"><p>size</p></td><td class="cell"><p>per token</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>type</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CityU</p></td><td class="cell"><p>i,D92,687</p></td><td class="cell"><p>44</p></td><td class="cell"><p>1.2587</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CKIP</p></td><td class="cell"><p>72i,55i</p></td><td class="cell"><p>6D</p></td><td class="cell"><p>1.1086</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NCC</p></td><td class="cell"><p>535,023</p></td><td class="cell"><p>6D</p></td><td class="cell"><p>1.0658</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PKU</p></td><td class="cell"><p>1,116,754</p></td><td class="cell"><p>103</p></td><td class="cell"><p>i.ii94</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 4: The testing data size and the baseline performance." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>corpus</p></td><td class="cell"><p>Baseline</p></td><td class="cell"><p>Testing size</p></td><td class="cell"><p>Token/tag OOV-R</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CityU</p></td><td class="cell"><p>0.8433</p></td><td class="cell"><p>184,314</p></td><td class="cell"><p>0.0921</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CKIP</p></td><td class="cell"><p>0.8865</p></td><td class="cell"><p>9i,07i</p></td><td class="cell"><p>0.0897</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NCC</p></td><td class="cell"><p>0.9159</p></td><td class="cell"><p>102,344</p></td><td class="cell"><p>0.0527</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PKU</p></td><td class="cell"><p>0.8805</p></td><td class="cell"><p>156,407</p></td><td class="cell"><p>0.0594</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Our POS taggers are evaluated on four testing data sets, one corresponding to each training cor­pus. We trained eight POS taggers, two on each training corpus, and submitted eight runs in total on the Chinese part-of-speech tagging task, two runs on each testing data set. The first run, labeled 'a' in Table 5, is produced using the first-stage tagger, and the second run, labeled 'b' in Table 5, is the output of the second-stage tagger, which is applied to the output of the first tagger. For all of our runs, only the provided training data are used. Table 5 shows the official evaluation results of the eight runs we submitted in the close track. The third column, labeled 'Total-A', shows the accu­racy of the eight runs. The accuracy is the propor­tion of correctly tagged words in a testing data set. Only one tag is assigned to every word in the test­ing data set. The remaining three labels, 'IV-R', 'OOV-R', and 'MT-R', may be defined in The Fourth SIGHAN Bakeoff overview paper.</p><p>Table 5: Official evaluation results of eight runs in the close track of the Chinese part-of-speech tag­ging task.</p></subsection></section><section number="4" title="Discussions"><p>A Chinese verb can function as a noun, and vice versa, without suffix change. In PKU corpus, a verb is labeled with the tag 'v', and a verb that functions as a noun is labeled with the tag 'vn'. In the PKU-b run, almost half of the incorrectly tagged verbs (v) were tagged as verbal noun (vn), and slightly more than half of the incorrectly tagged verbal nouns (vn) were tagged as verb (v).</p><p>The accuracy of our best runs on all four corpora is much higher than the baseline performance. On the PKU corpus, the accuracy is increased from the baseline performance of 0.8805 to 0.9450, an im­provement of 7.33% over the baseline. The sec­ond-stage tagging increased the accuracy on all four corpora. On the PKU corpus, the accuracy is increased by about 0.32% over the first-stage tag­ging. The improvement may not seem to be large; however, it corresponds to an error reduction by 5.4%.</p><p>That the accuracy on the CityU corpus is the lowest among all four corpora is not surprising, given that the CityU testing data set has the highest out-of-vocabulary rate, and the CityU training cor­pus has the highest average number of tags as­signed to each token type. Furthermore, the CityU training corpus has the lowest percentage of tokens with only one tag. The POS tagging task on CityU corpus seems to be most challenging among the four corpora.</p></section><section number="5" title="Conclusions"><p>We have described a Chinese part-of-speech tagger with maximum entropy modeling. The tagger with rich lexical and morphological features signifi­cantly outperforms the baseline system which as­signs to a word the most likely tag assigned to that word in the training corpus. The use of features involving the part-of-speech tags of the following words further improves the performance of the tagger. <b>References</b></p><p>Adam L. Berger, Stephen A. Della Pietra, and Vincent J. Della Pietra. 1996. A Maximum Entropy Approach to Natural Language Processing. <i>Computational Lin­guistics, </i>22(1):39-71.</p><p>Stanley F. Chen, and Ronald Rosenfeld. 1999. <i>A Gaus-sion Prior for Smoothing Maximum Entropy Models, </i>Technical Report CMU-CS-99-108, Carnegie Mellon University.</p><p>Rober Malouf. 2002. <i>A Comparison of Algorithms for Maximum Entropy Parameter Estimation, </i>Proceed­ings of CoNLL-2002.</p><p>Adwait Ratnaparkhi. 1996. <i>A Maximum Entropy Model for Part-of-Speech Tagging, </i>Proceedings of the Con­ference on Empirical Methods in Natural Language Processing, pp 133-142.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Corpus</p></td><td class="cell"><p>Run ID</p></td><td class="cell"><p>Total-A</p></td><td class="cell"><p>IV-R</p></td><td class="cell"><p>OOV-R</p></td><td class="cell"><p>MT-R</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CityU</p></td><td class="cell"><p>a</p></td><td class="cell"><p>0.8929</p></td><td class="cell"><p>0.9367</p></td><td class="cell"><p>0.4608</p></td><td class="cell"><p>0.8705</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CityU</p></td><td class="cell"><p>b</p></td><td class="cell"><p>0.8951</p></td><td class="cell"><p>0.9389</p></td><td class="cell"><p>0.4637</p></td><td class="cell"><p>0.8745</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CKIP</p></td><td class="cell"><p>a</p></td><td class="cell"><p>0.9286</p></td><td class="cell"><p>0.9618</p></td><td class="cell"><p>0.5875</p></td><td class="cell"><p>0.9099</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CKIP</p></td><td class="cell"><p>b</p></td><td class="cell"><p>0.9295</p></td><td class="cell"><p>0.9629</p></td><td class="cell"><p>0.5869</p></td><td class="cell"><p>0.9123</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NCC</p></td><td class="cell"><p>a</p></td><td class="cell"><p>0.9525</p></td><td class="cell"><p>0.9717</p></td><td class="cell"><p>0.6059</p></td><td class="cell"><p>0.9135</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NCC</p></td><td class="cell"><p>b</p></td><td class="cell"><p>0.9541</p></td><td class="cell"><p>0.9738</p></td><td class="cell"><p>0.5998</p></td><td class="cell"><p>0.9195</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PKU</p></td><td class="cell"><p>a</p></td><td class="cell"><p>0.9420</p></td><td class="cell"><p>0.9648</p></td><td class="cell"><p>0.5813</p></td><td class="cell"><p>0.9148</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PKU</p></td><td class="cell"><p>b</p></td><td class="cell"><p>0.9450</p></td><td class="cell"><p>0.9679</p></td><td class="cell"><p>0.5818</p></td><td class="cell"><p>0.9252</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section></body></article>