<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="41"/><title>Fast Domain Adaptation for Part of Speech Tagging for Dialogues</title><pubinfo>Proceedings of Recent Advances in Natural Language Processing,pages 41^-8, Hissar, Bulgaria, 12-14 September 2011.</pubinfo><author surname="Kübler" givenname="Sandra"><org  name="Indiana University" country="USA" city="Bloomington"/></author><author surname="Baucom" givenname="Eric"><org  name="Anna University" country="India" city="Chennai"/></author></firstpageheader><frontmatter><p><b>Fast Domain Adaptation for Part of Speech Tagging for Dialogues</b></p><p><b>Sandra Kubler, Eric Baucom</b></p><p>Indiana University <b>{skuebler,eabaucom}@indiana.edu</b></p></frontmatter><abstract>Part of speech tagging accuracy deterio­rates severely when a tagger is used out of domain. We investigate a fast method for domain adaptation, which provides ad­ditional in-domain training data from an unannotated data set by applying POS tag­gers with different biases to the unanno-tated data set and then choosing the set of sentences on which the taggers agree. We show that we improve the accuracy of a trigram tagger, TnT, from 85.77% to 86.10%. In order to improve perfor­mance on unknown words, we investigate using active learning for learning ambi­guity classes of domain specific words, yielding an accuracy of 89.15% for TnT. </abstract></header><body><section number="1" title="Introduction"><p>Part of speech (POS) tagging for English is often considered a solved problem. There are well es­tablished approaches such as Markov model tri-gram taggers (Brants, 2000), maximum entropy taggers (Ratnaparkhi, 1996), or Support Vector Machine based taggers (Gimenez and Marquez, 2004), and accuracy reaches approximately 97%.</p><p>However, most experiments in POS tagging for English have concentrated on data from the Penn Treebank (Marcus et al., 1993). If POS taggers trained on the Penn Treebank are used to tag data from other domains, accuracy deteriorates signif­icantly. Blitzer et al. (2006) apply structural cor­respondence learning for learning pivot features to increase accuracy in the target domain. However, their approach is restricted to discriminative ap­proaches to POS tagging.</p><p>In this paper, we investigate a simple and fast method for domain adaptation that is usable with any POS tagger: selecting reliably tagged in­domain data to add to the training set. This method has been successful for domain adaptation for de­pendency parsing (Chen et al., 2008). We use a corpus of dialogues collected in a collaborative task as target domain, thus introducing the chal­lenges of processing spontaneous speech: hesita­tions, corrections, false starts, and contractions. We assume that this domain is more challenging than a target domain of biomedical texts, which is often used for domain adaptation research. Spon­taneous speech dialogues do not only differ in ter­minology, but also in the types of sentences. Di­alogues, for example, contain a higher percentage of questions and imperatives than formal written language, such as news or scientific writings.</p><p>Our domain adaptation experiments concentrate on adding in-domain training data based on an en­semble of POS taggers. The experiments show that extending the training set generally improves POS tagging accuracy. However, it cannot provide information on the ambiguity classes for words that do not appear in the source domain. For this reason, we integrate an active learning strategy for adding ambiguity classes for words that are iden­tified automatically as unlikely to be tagged cor­rectly.</p><p>The remainder of the paper is structured as fol­lows: In section 2, we provide an overview of do­main adaptation techniques in POS tagging and parsing. Section 3 describes our approach to do­main adaptation, and section 4 describes the exper­imental setup. In section 5, we discuss our find­ings for domain adaptation, and in section 6, we describe the active learning extension.</p></section><section number="2" title="Related Work"><p>Domain adaptation is a task that has received much attention in recent years, with different re­sults, ranging from evaluations that it is "frustrat-ingly easy" (Daume III, 2007) to "frustratingly hard" (Dredze et al., 2007). The main differen­tiating factor seems to be whether a small portion of annotated in-domain training data is available or only a large-size, unannotated data set.<page local="2" global="42"/> In our work, we concentrate on the second, more diffi­cult, scenario.</p><p>Most work on domain adaptation has focused on parsing rather than on POS tagging (e.g. (Mc-Closky et al., 2006; Yoshida et al., 2007; Chen et al., 2008; Rimell and Clark, 2008). Chen et al. (2008)) perform domain adaptation for a de­pendency parser. They show their best results are reached by adding only a selection of the informa­tion provided by a parser trained on out-of-domain data. Since short dependencies are more reliable than long ones, they select only the short, and thus reliable, ones and gain an increase in accuracy. Rimell and Clark (2008) adapt the Penn Treebank to parse grammatical relations in the biomedical domain. They report that the domains are similar structurally and that the lexicon is the main differ­ence between the domains. Yoshida et al. (2007) investigate the influence of an external POS tag­ger on parsing accuracy in an HPSG parser. They show that the quality of the POS tagger has a sig­nificant influence even in-domain. The situation can be improved by allowing the POS tagger to output multiple, weighted POS tags from which the parser can choose. They show that allowing the tagger to output multiple POS tags improves parsing results both in-domain and out-of-domain.</p><doubt alpha="65.2" length="46" tooSmall="False" monospace="0.0">Clark et al. (2003) use the results of one POS</doubt><p>tagger on unannotated data to inform the training of another tagger in a semi-supervised setting us­ing a co-training routine with a Markov model tag­ger and a maximum entropy tagger. The authors test both agreement-based co-training, where the sentences are added to training only if the taggers both agree, and naive co-training, where all sen­tences from one tagger are added to the training of the other, with no filter. For small sets of seed sentences, both types of co-training improve ac­curacy, with the higher quality, smaller training set from agreement-based co-training performing slightly better. The authors also report results for using naive co-training after the taggers were al­ready trained on large amounts of manually anno­tated data. Naive co-training did not improve the taggers when trained in such a way (the authors leave agreement based co-training to future work).</p><p>Blitzer et al. (2006) investigate domain adapta­tion for POS tagging using the method of struc­tural correspondence learning (SCL). SCL provides an informative feature-space for modeling the similarities between source and target domain by identifying <i>pivot </i>features. Pivot features be­have similarly across domains, and if non-pivot features in the different domains correspond to many of the same pivot features, they are as­sumed to correlate. The machine learning algo­rithm is trained with the feature-space model from SCL on the source domain, with the idea that the trained model will now be informative for the unlabeled target domain as well.   Blitzer et al.</p><p>(2006) evaluate the SCL transfer of a POS tagger from the Penn Treebank to a corpus of biomedi­cal abstracts (MEDLINE), reporting an improve­ment from 87.9% to 88.9%. The authors report that vocabulary is the main difference between the domains. However, SCL can only be applied to feature-based discriminative learning methods.</p></section><section number="3" title="Domain Adaptation by Tagger Combination"><p>For our experiments, we use the Wall Street Jour­nal part of the Penn Treebank as source domain and dialogues in a collaborative task as target do­main. In the target domain, we have access to a large unannotated corpus and a small annotated corpus, which we use for evaluation purposes. In order to adapt a POS tagger to the target domain, we extend the training set by sentences from the large unannotated corpus. Our hypothesis is that these sentences will provide the POS tagger with relevant information from the target domain. For assigning POS tags to the additional sentences from the target domain, we use three different POS taggers trained on the Penn Treebank. Then we select those sentences for which a majority of tag­gers agree on the POS tags. The method of using agreement between taggers was originally used by van Halteren et al. (2001) to improve tagger per­formance. We investigate the following questions: 1) How does the number of agreeing POS tag­gers influence the accuracy of the final tagger? 2) Should we select only complete sentences or add all trigrams on which the taggers agree? Lifting the restriction that the taggers agree on complete sentences will increase the size of the training set. 3) Do we need the full Penn Treebank training set, or does this large training set dominate the smaller training set from the target domain?</p><page local="3" global="43"/></section><section number="4" title="Experimental Setup 4.1   Data Sets"><p>We use three corpora: the Penn Treebank for the source domain; the HCRC Map Task Corpus (Thompson et al., 1996) for additional training in the cooperative dialogue domain; and the CReST corpus (Eberhard et al., 2010) for evaluation in the target domain.</p><p><b>The HCRC Map Task Corpus </b>(Thompson et al., 1996) is a multi-modal corpus composed of 18 hours of digital audio and 150 000 words of tran­scription, representing 128 two-person conversa­tions. The conversations were obtained from a co­operative problem solving task, in which two par­ticipants were asked to help one another fill in a route on a map. HCRC is annotated for speaker and dialogue turn information, as well as for POS tags. However, we use only the actual transcrip­tions. This corpus serves as our unannotated, in­domain training corpus.</p><p><b>The CReST Corpus </b>(Eberhard et al., 2010) is a multi-modal corpus consisting of 7 dialogues, comprising 11 317 words in 1 977 sentences. Sim­ilar in domain to the HCRC corpus, it represents cooperative dialogues, but is based on a slightly different task: one of the participants is located in a search environment, while the other is outside but has access to a map of the environment. The participants need to collaborate to fulfill their tasks (locating objects in the environment and placing objects on the map).</p><p>CReST is annotated for POS, syntactic depen­dency and constituency, disfluency, and dialogue structure. The POS tagset is a superset of the tagset for the Penn Treebank, with the additional tags representing features unique to natural dia­logue.</p><p><b>Data Preparation. </b>Due to differences between the transcriptions of HCRC and CReST, we made small, systematic changes to HCRC to make it more consistent with CReST. For instance, HCRC had various permutations of <b>mmhmm </b>which we changed to the standard <b>mhm </b>transcription in CReST. Since the Penn Treebank does not con­tain all tags used in CReST, we translated the ad­ditional CReST tags into tags of the original tagset for our experiments. E.g. the POS tag <b>VBI </b>(imper­ative verb) is translated into <b>VB </b>(verb in the base form).</p><subsection number="4.2" title="POS Taggers"><doubt alpha="64.4" length="45" tooSmall="False" monospace="0.0">We use three POS taggers: TnT (Brants, 2000),</doubt><p>MElt (Denis and Sagot, 2009), and SVMTool (Gimenez and Marquez, 2004). These taggers were chosen because they represent the state of the art for single-direction taggers and also because they use different approaches to POS tagging and thus have different biases. Our assumption is that the different biases will result in different types of POS tagging mistakes.</p><p><b>TnT </b>(Brants, 2000) is a trigram Markov model POS tagger with state-of-the-art treatment of un­known words. TnT generates files containing lexi­cal and transition frequencies and thus provides us with the option of including new trigrams directly into the trained model.</p><p><b>The   Maximum-Entropy Lexicon-Enriched</b> conditional sequence maximum entropy POS tagger that uses a set of lexical and context features, which are a superset of the features used by Ratnaparkhi (1996) and Toutanova and</p><doubt alpha="63.4" length="41" tooSmall="False" monospace="0.0">Tagger (MElt)(Denis and Sagot, 2009) is a</doubt><p>Manning (2000).</p><p><b>SVMTool </b>(Gimenez and Marquez, 2004) is a discriminative POS tagger based on support vec­tor machines. The features and specifications used in training were taken from the SVMTool model for English, based on the Penn Treebank.</p></subsection></section><section number="5" title="Experiments"><p>We perform six experiments: The first experiment establishes a baseline by training the POS taggers out of domain on the Penn Treebank and then us­ing them without adaptation on the target domain. In the second experiment, the training set is ex­tended by those HCRC sentences on which all three taggers agree. In the third experiment, we investigate whether the accuracy of the adapted tagger deteriorates if we choose all sentences on which only two taggers agree. In the fourth exper­iment, we investigate the effect of adding trigram information on which all taggers agree to the TnT trained model. In the fifth experiment, we also add lexical information to the TnT model. In the final experiment, we investigate whether the large size of the Penn Treebank neutralizes effects from the additional training data, based on the experiment with sentences on which all three taggers agree.</p><page local="4" global="44"/><p>Table 1: The results of the baseline and of select­ing all sentences on which all taggers agree. Dags indicate a significant improvement over the base­line.</p><subsection number="5.1" title="Agreement Among All POS Taggers"><p>This experiment uses all three POS taggers, trained on the Penn Treebank, to tag all sentences from the HCRC corpus. Then all sentences are selected on which the taggers agree. These sen­tences are added to the Penn Treebank training set, and the taggers are retrained and evaluated on the CReST corpus. The results of the baseline and this experiment are shown in table 1.</p><p>The results show that both discriminative POS taggers, MElt and SVMTool, improve signifi­cantly over the baseline (McNemar, <i>p &lt; </i>0.001). TnT, in contrast, suffers a non-significant decrease in performance. However, TnT's baseline results are significantly higher than the two other tag­gers'. This can be explained by the state-of-the art module for guessing unknown words in TnT, which is based on suffix tries extracted from hapax legomena in the training data set. For the baseline, TnT reaches an accuracy of 16.64% on unknown words, MElt 11.65%, and SVMTool 10.32%.</p><p>In order to determine whether our initial low performance was due to within-domain tagging is­sues, such as "fuzzy" linguistic boundaries (Man­ning, 2011), or simply to the level of difference between our source and target domains, we con­ducted a brief analysis of the errors from this ex­periment. We found that the top three discrep­ancies in the <b>all3 </b>condition for TnT, compris­ing 55.32% of the incorrect tags, were the result of mistakenly labeling a gold-tagged interjection <b>(UH) </b>with an adjective <b>(JJ),</b><b> </b>noun <b>(NN),</b><b> </b>or an ad­verb <b>(RB) </b>tag. The next most common mistake was labeling a gold-tagged <b>SYM </b>(incomplete or non-word) with <b>JJ </b>(5.32% of discrepancies). <b>SYM </b>and <b>UH </b>are much more common in a corpus of spo­ken dialogue transcriptions than in closely edited financial news. Thus, these top four mistakes rep­resent errors arising from the dissimilarity of the domains (as opposed to the fifth mistake, mistak­ing <b>IN </b>(preposition) for <b>RB, </b>which is a more traditional within-domain tagging error, with "fuzzy" linguistic boundaries partially to blame).</p></subsection><subsection number="5.2" title="Agreement Between Two Taggers"><p>The reason for requiring all three POS taggers to agree on full sentences is that the selected sen­tences will be reliable. However, the method also has the drawback that only a rather small num­ber of sentences fulfill this criterion. The first 2 rows in table 2 show the number of words in the training data for the baseline experiment with only Penn Treebank data and for the <b>all3 </b>experiment. They show that only a very small number of words is added: The number of words increases from ap­proximately 1.34 million words to 1.39 million, i.e. only 50 000 words are added out of the 150 000 words in the HCRC corpus, an insignificant num­ber when compared to the source domain data.</p><p>Thus, in order to provide more in-domain train­ing data, we relax the constraint on the selection of sentences from the HCRC corpus and select all sentences for which two specific taggers agree. The results are shown in table 3. The last column in this table repeats the results from the previous experiment.</p><p>These results show that the additional data (cf. table 2) improves performance over the experi­ment requiring agreement between all three tag­gers. It is worth noting that MElt and TnT per­form best with training where the common sen­tences are from the two <i>other </i>taggers. For SVM-Tool, including TnT improves accuracy, but there is no significant difference between the combina­tion of MElt with TnT and the one with SVMTool and TnT.<page local="5" global="45"/> We assume that TnT has reached satura­tion on the Penn Treebank and cannot learn new information from additional data tagged with its own bias. Sentences from the other taggers, how­ever, do present new information.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Tagger</p></td><td class="cell"><p>baseline</p></td><td class="cell"><p>all3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MElt</p></td><td class="cell"><p>83.91</p></td><td class="cell"><p>84.32t</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SVMTool</p></td><td class="cell"><p>84.60</p></td><td class="cell"><p>85.15+</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>TnT</p></td><td class="cell"><p><b>85.77</b></p></td><td class="cell"><p>85.70</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 2: Number of words in the training set." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Training</p></td><td class="cell"><p># of words</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>baseline</p></td><td class="cell"><p>1 342 561</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>all3</p></td><td class="cell"><p>1 391 238</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>me/svm</p></td><td class="cell"><p>1 413 106</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>me/tnt</p></td><td class="cell"><p>1 418 957</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>svm/tnt</p></td><td class="cell"><p>1 412 917</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 3: Results of adding all sentences for which two taggers agree." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Tagger</p></td><td class="cell"><p>me/svm</p></td><td class="cell"><p>me/tnt</p></td><td class="cell"><p>svm/tnt</p></td><td class="cell"><p>all3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MElt</p></td><td class="cell"><p>84.37*</p></td><td class="cell"><p>84.28</p></td><td class="cell"><p>84.59*</p></td><td class="cell"><p>84.32*</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SVM</p></td><td class="cell"><p>84.98</p></td><td class="cell"><p>85.30t</p></td><td class="cell"><p>85.47t</p></td><td class="cell"><p>85.15t</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>TnT</p></td><td class="cell"><p><b>85.94</b></p></td><td class="cell"><p>85.84</p></td><td class="cell"><p>85.70</p></td><td class="cell"><p>85.70</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>We had a closer look at the sentences that were added to TnT when MElt and SVMTool (me/svm) agree and when MElt and TnT (me/tnt) agree and found considerable differences in the distri­bution of POS tags. These differences can also be found in the test set tagged with TnT, based on the two training sets. In all data sets, the com­bination me/tnt seems to keep the lexical bias of the Penn Treebank more strongly than the combi­nation me/svm. For example, the word left is consistently tagged as a noun when TnT uses the me/svm combination. In most instances, this is the correct decision. The me/tnt combination, in con­trast, prefers a verb reading. For the word back, the me/svm combination selects the correct adverb reading over the verbal particle reading preferred by the me/tnt reading. Since the combination of SVMTool and TnT also keeps the bias, the inno­vation in the me/svm combination cannot be at­tributed to having SVMTool in the combination.</p><p>We also investigated whether using a union of sentences from different pairs of taggers would in­crease overall accuracy. This adds approximately 70 000 words to the training set. However, the results of this experiment proved to be not signifi­cantly different from those based on tagger pairs.</p></subsection><subsection number="5.3" title="From Complete Sentences to n -grams"><p>The results from the previous experiment show that adding more training data, even if it is less certain, improves the accuracy of the final tagger. One possibility to provide more training material consists in relaxing the constraint that the taggers need to agree on complete sentences. Instead, we extract either all longest matching n-grams or all trigrams on which the taggers agree. The n-grams are processed and added to the TnT model from the Penn Treebank. This is only possible because TnT stores its trained model in an accessible for­mat. The discriminative POS taggers could not be used for this experiment since adding incomplete sentences as training data would have influenced their trained models negatively.</p><p>As before, all evaluations are performed on CReST. The results of this experiment are shown in table 4.  The first 3 columns contain the results of merging n-grams or trigrams from 2 dif­ferent taggers; the last column shows the results for merging all 3 taggers. The first row repeats the results from previous experiments using complete sentences that taggers agree upon. We restrict our­selves to adding only transition information here and merely use the lexicon from the Penn Tree-bank baseline. We will investigate adding both transition and lexical information in the next ex­periment. The results show that adding trigrams instead of complete sentences, based on MElt and SVMTool, results in approximately 25 000 addi­tional trigram counts, and it improves the accu­racy of the final tagger from 85.94% to 86.10%. Adding all n-grams, in contrast, adds around 33 000 trigram counts and results in slightly lower accuracies, demonstrating that in some cases, the sheer amount of data may be counteracted by sub­standard quality. Again, TnT profits most from in­domain sentences provided by a combination of</p><p>MElt and SVMTool.</p></subsection><subsection number="5.4" title="Adding Lexical Information"><p>A look at the words that are mistagged with the highest frequency in the previous experiment, in which we added trigram information, shows that they fall into two different categories: words such as okay, um, gonna that are typical for dialogues but do not occur frequently in the Penn Treebank; and words that have a different POS preference in the target domain. An example for this category is the word left, which tends to be a verb in the Penn Treebank and an adverb in CReST.</p><p>For this reason, we decided to add the lexical information from the trigrams to TnT's lexicon. The results of this experiment are shown in table 5. They show that adding lexical information re­sults in lower accuracies: they decrease minimally from 86.10% (adding only trigram transition in­formation) to 86.00% when adding both transition and lexical information. When adding n-grams and lexical information, the results improve over adding only n-grams, but they do not reach the<page local="6" global="46"/></p><table caption="Table 4: Results of adding n-grams or trigrams to TnT's model." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>me/svm</p></td><td class="cell"><p>me/tnt</p></td><td class="cell"><p>svm/tnt</p></td><td class="cell"><p>all3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>full</p></td><td class="cell"><p>85.94</p></td><td class="cell"><p>85.84</p></td><td class="cell"><p>85.70</p></td><td class="cell"><p>85.70</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>n</i></p></td><td class="cell"><p>85.88</p></td><td class="cell"><p>85.55</p></td><td class="cell"><p>85.93</p></td><td class="cell"><p>85.76</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>tri.</p></td><td class="cell"><p><b>86.10</b></p></td><td class="cell"><p>85.77</p></td><td class="cell"><p>85.93</p></td><td class="cell"><p>85.98</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Tagger baseline red. base. red.+all3 best trigram result.</p><doubt alpha="18.2" length="22" tooSmall="False" monospace="0.0">MElt 83.91 79.38 83.86</doubt><doubt alpha="28.0" length="25" tooSmall="False" monospace="0.0">SVMTool 84.60 78.79 83.90</doubt><doubt alpha="14.3" length="21" tooSmall="False" monospace="0.0">TnT 85.77 79.86 84.11</doubt><p>Since this result did not meet our expectation, we analyzed the changes to the lexicon file and the tagging errors. The extended lexicon con­tains 326 additional words, but only 8 of them also occur in CReST <b>(ah, fifteen, forty-five, furthest, hmm, mhm, um, yeah). yeah </b>is by far the most common word in the test data. The small number of added words that actually occur in the test set severely restricts possible improve­ments on in-domain POS tagging.</p><p>A comparison of TnT's performance with and without the extended lexicon shows that there are 101 discrepancies (in 11 317 words) in which the POS tagger without additional lexical information makes the correct decision. Out of these discrep­ancies, the word <b>yeah </b>accounts for 45 errors. Here, the extended lexicon lists the tag <b>NN </b>in­stead of the tag <b>UH. </b>The reason lies in the fact that <b>yeah </b>does not occur in the Penn Treebank, and the three taggers trained on this treebank all (wrongly) tag <b>yeah </b>with the most frequent tag for unknown words.</p><p>From the error analysis, we can conclude that the added words do not correspond to the words that are needed in the test domain, which means that the HCRC map task corpus data are not sim­ilar enough to the CReST data. However, we can also conclude that even if there were a larger over­lap, there is a high chance that those words would be mistagged by the ensemble of taggers so that adding the new words would result in a deteriora­tion of the performance.</p></subsection><subsection number="5.5" title="Decreasing Out-Of-Domain Training Data"><p>In a final experiment, we investigate whether the difference in amounts of training data between source domain and target domain neutralizes the positive influence of adding in-domain informa­tion. Table 2 shows that the number of words added by our methods ranges between one third and half of the original data set. It is therefore possible that the added information does not change the transition probabilities enough to improve the behavior of the final tagger. For this reason, we restrict the size of the Penn Treebank training set to the number of words in the in-domain set, thus reducing the influence of the out-of-domain data. For this experiment, the in-domain training set is taken from the combination of all 3 taggers, as re­ported in table 1. The results of this experiment are shown in table 6.</p><table caption="Table 6: The results of restricting the size of the out-of-domain training set."></table><doubt alpha="21.2" length="99" tooSmall="True" monospace="0.0">1/10    2/10    3/10    4/10    5/10    6/10    7/10    8/10    9/10 10/10 Fraction of WEU Training</doubt><figure caption="Figure 1: Accuracy as a function of amount of WSJ training"></figure><p>The results show that training the POS tagger on only the reduced Penn Treebank containing 46 680 words results in a severe loss in accuracy, from ap­proximately 84% to approximately 79%. Adding more training data from the Penn Treebank con­sistently increases the results, as shown in Figure 1, thus demonstrating that more data is more im­portant than in-domain knowledge.</p><p>These experiments shows that even a fairly "easy" problem such as POS tagging requires a large training set. In the first experiment, combin­ing the reduced Penn Treebank with the in-domain data set increases the accuracies of all POS taggers over the reduced baseline, but they do not reach the baseline based on the whole Penn Treebank. This experiment shows that the sheer size of the training set is more important than access to in­domain training data, at least when the quality of<page local="7" global="47"/></p><table caption="Table 5: Results of adding lexical information to TnT's model." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>me/svm</p></td><td class="cell"><p>me/tnt</p></td><td class="cell"><p>svm/tnt</p></td><td class="cell"><p>all3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>n</i></p></td><td class="cell"><p>85.88</p></td><td class="cell"><p>85.55</p></td><td class="cell"><p>85.70</p></td><td class="cell"><p>85.70</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>n+lex.</p></td><td class="cell"><p>86.00</p></td><td class="cell"><p>85.86</p></td><td class="cell"><p>85.81</p></td><td class="cell"><p>85.88</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>tri.</p></td><td class="cell"><p><b>86.10</b></p></td><td class="cell"><p>85.77</p></td><td class="cell"><p>85.93</p></td><td class="cell"><p>85.98</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>tri.+lex.</p></td><td class="cell"><p>86.00</p></td><td class="cell"><p>85.42</p></td><td class="cell"><p>85.78</p></td><td class="cell"><p>85.86</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Table 7: Results of adding an active learning lexicon to the training for TnT. All differences between the two experiments are significant.</p><p>this additional training set is not guaranteed.</p></subsection></section><section number="6" title="Extending the Lexicon with Active Learning"><p>The results of the previous sections show that adding information on which taggers trained out-of-domain agree is useful for moderately improv­ing tagging accuracy and especially for reestimat-ing transition probabilites. However, the method is unsuitable for finding the correct ambiguity sets for words that do not occur in the out-of-domain training set. Such words must be treated in the tag­ger's module for handling unknown words, which is often based on suffix information extracted from infrequent words in the training set. However, many of the unknown words in the CReST cor­pus are colloquial words and thus do not show the same morphological characteristics as words in the training set. The words yeah and mhm are good examples: it is rather unlikely that the tag­ger can guess their ambiguity set based on their bigram suffixes ah and hm. This problem is not unique to the domain of spontaneous speech. Biomedical terms, for example, also display atypi­cal suffixes, which make them difficult to classify.</p><p>Since the training corpus cannot provide the re­quired information, we decided to acquire mini­mal information from the target domain via active learning. This goal here is to automatically iden­tify words that TnT was likely to tag incorrectly. These words are then presented to the user, who is asked to provide the ambiguity sets for the words. In our experiment, we simulated the user by look­ing up the words identified by our program in the CReST gold standard.</p><p>In order to determine which words would be dif­ficult for TnT, we built a suffix trie similar to TnT's model for unknown words. For the sake of sim­plicity, we restricted the trie to a maximum suf­fix length of three letters. Then, each word in our CReST test corpus that did not occur in the Penn Treebank training lexicon was matched against the suffix trie. If the word's suffix was not present in the trie, the word was presented to the user and added to TnT's lexicon. The extended lexicon was used in combination with the extended transitions based on trigrams from section 5.3. In total, 74 ambiguity classes were added in the active learn­ing lexicon.</p><p>The results in table 7 show that adding the active learning lexicon to the Penn Treebank baseline im­proves tagging accuracy to 89.04%, outperform­ing our best previous results (cf. table 4). The best results of 89.15% are based on combinations of the active learning lexicon and transition informa­tion from where just two taggers agree on HCRC trigrams. This illustrates that adding new words to the lexicon results in a higher improvement than adding new transition information. However, the best results are gained by a combination of the two methods. All active learning results are sig­nificantly higher than the previous best result of 86.10%.</p><p>For the Penn Treebank baseline, there were 176 word types that were wrongly tagged. In the ac­tive learning experiment, 71 types (40.34%) were added with their ambiguity classes, among them the prevalent word yeah. All of these words were unambiguous in the target domain.</p></section><section number="7" title="Conclusion and Future Work"><p>We investigated a generally applicable method of domain adaptation for POS tagging, which uses the consent of three POS taggers with different bi­ases to add in-domain sentences to the training set. We show that we reach a slight but significant in­crease in accuracy from 85.77% to 86.10% when using all trigrams on which the POS taggers agree. Reducing the size of the out-of-domain training set has a detrimental effect on the quality of the POS tagger. The improvement from adding in-domain trigrams is due to more accurate transition prob­abilities. In contrast, the lexical additions from the in-domain data were detrimental. The active learning strategy of adding user-defined lexical in­formation for difficult unknown words improves this accuracy to 89.15%. However, this accuracy is still far below an in-domain accuracy, which reaches 95.<page local="8" global="48"/>66%.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>base</p></td><td class="cell"><p>all3</p></td><td class="cell"><p>me/svm</p></td><td class="cell"><p>me/tnt</p></td><td class="cell"><p>svm/tnt</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>trans.</p></td><td class="cell"><p>85.77</p></td><td class="cell"><p>86.10</p></td><td class="cell"><p>85.77</p></td><td class="cell"><p>85.93</p></td><td class="cell"><p>85.98</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>active</p></td><td class="cell"><p>89.04</p></td><td class="cell"><p>89.13</p></td><td class="cell"><p><b>89.15</b></p></td><td class="cell"><p>89.03</p></td><td class="cell"><p><b>89.15</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>TnT's better performance on this task may be due to its superior handling of unknown words, but may also be a result of the fact that the fea­ture sets used with MElt and SVMTool were de­signed specifically for the Penn Treebank. We may be able to improve results for those two taggers if we optimize the feature set for the target domain. However, this means modifying the implementa­tion of the taggers since the feature extraction is not modular. For the future, we are planning to in­vestigate whether structural correspondence learn­ing (Blitzer et al., 2006) will reach higher accura­cies, even though it cannot be used with our best performing POS tagger, TnT. We will also repeat these experiments with a biomedical target domain to see if our results transcend domains.</p><p><b>Acknowledgment</b></p><p>This work is based on research supported by the US Office of Naval Research (ONR) Grant #N00014-10-1-0140.</p></section><references><p>John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. Domain adaptation with structural correspon­dence learning. In <i>Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), </i>Sydney, Australia.</p><p>Thorsten Brants. 2000. TnT-a statistical part-of-speech tagger. In <i>Proceedings ofthe ANLP-NAACL,</i></p><p>Seattle, WA.</p><p>Wenliang Chen, Youzheng Wu, and Hitoshi Isahara. 2008. Learning reliable information for dependency parsing adaptation. In <i>Proceedings of the 22nd In­ternational Conference on Computational Linguis­tics (COLING), </i>Manchester, UK.</p><p>Stephen Clark, James Curran, and Miles Osborne. 2003. Bootstrapping POS-taggers using unlabelled data. In <i>Proceedings ofthe Seventh Conference on Natural Language Learning (CoNLL), </i>Edmonton, Canada.</p><p>Hal Daume III. 2007. Frustratingly easy domain adap­tation. In <i>Proceedings of the 45th Annual Meet­ing ofthe Association ofComputational Linguistics, </i>Prague, Czech Republic.</p><p>Pascal Denis and Benoit Sagot. 2009. Coupling an annotated corpus and a morphosyntactic lexicon for state-of-the-art POS tagging with less human effort.</p><p>In <i>Proceedings of the Pacific Asia Conference on Language, Information and Computation (PACLIC 23), </i>Hong Kong, China.</p><p>Mark Dredze, John Blitzer, Partha Pratim Taluk-dar, Kuzman Ganchev, Joao Graca, and Fernando Pereira. 2007. Frustratingly hard domain adap­tation for dependency parsing.  In <i>Proceedings of</i></p><p><i>the CoNLL Shared Task Session ofEMNLP-CoNLL</i> <i>2007, </i>Prague, Czech Republic.</p><p>Kathleen Eberhard, Hannele Nicholson, Sandra Kubler, Susan Gunderson, and Matthias Scheutz. 2010. The Indiana "Cooperative Remote Search Task" (CReST) Corpus. In <i>Proceedings of the Seventh International Conference on Language Re­sources and Evaluation (LREC), </i>Valetta, Malta.</p><p>Jesus Gimenez and Lluis Marquez. 2004. SVMTool: A general POS tagger generator based on Support Vector Machines. In <i>Proceedings ofLREC, </i>Lisbon, Portugal.</p><p>Christopher Manning. 2011. Part-of-speech tagging from 97% to 100%: Is it time for some linguistics?</p><p>In <i>Proceedings ofCICLing, </i>Tokyo, Japan.</p><p>Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. <i>Computa­tional Linguistics, </i>19(2):313-330.</p><p>David McClosky, Eugene Charniak, and Mark John­son. 2006. Reranking and self-training for parser adaptation. In <i>Proceedings ofCOLING-ACL, </i>Syd­ney, Australia.</p><p>Adwait Ratnaparkhi. 1996. A maximum entropy model for part-of-speech tagging. In <i>Proceedings</i></p><p><i>ofEMNLP, </i>Philadelphia, PA.</p><p>Laura Rimell and Stephen Clark. 2008. Adapting a lexicalized-grammar parser to contrasting domains. In <i>Proceedings ofEMNLP, </i>Honolulu, Hawaii.</p><p>Henry Thompson, Anne Anderson, Ellen Gurman Bard, Gwyneth Doherty-Sneddon, Alison New-lands, and Cathy Sotillo. 1996. The HCRC Map Task Corpus: Natural dialogue for speech recogni­tion. In <i>Proceedings ofthe ARPA Human Language Technology Workshop, </i>Plainsboro, NJ.</p><p>Kristina Toutanova and Christopher D. Manning. 2000. Enriching the knowledge sources used in a maximum entropy part-of-speech tagger. In <i>Pro­ceedings ofEMNLP-VLC, </i>Hong Kong.</p><p>Hans van Halteren, Walter Daelemans, and Jakub Za-vrel. 2001. Improving accuracy in word class tagging through the combination of machine learn­ing systems. <i>Computational Linguistics, </i>27(2):199229.</p><p>Kazuhiro Yoshida, Yoshimasa Tsuruoka, Yusuke Miyao, and Jun'ichi Tsujii. 2007. Ambiguous part-of-speech tagging for improving accuracy and do­main portability of syntactic parsers. In <i>IJCAI'07: Proceedings of the 20th International Joint Confer­ence on Artifical intelligence</i>, Hyderabad, India.</p></references></body></article>