<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="1051"/><title>Part-of-Speech Tagging for English-Spanish Code-Switched Text</title><author surname="Solorio" givenname="Thamar"><org  name="University of Texas at Dallas" country="USA" city="Richardson"/></author><author surname="Liu" givenname="Yang"><org  name="University of Texas at Dallas" country="USA" city="Richardson"/></author></firstpageheader><frontmatter><p><b>Part-of-Speech Tagging for English-Spanish Code-Switched Text</b></p><p><b>Thamar Solorio </b>and <b>Yang Liu</b></p><p>Human Language Technology Research Institute The University of Texas at Dallas Richardson, TX 75080, USA tsolorio,yangl@hlt.utdallas.edu</p></frontmatter><abstract>Code-switching is an interesting linguistic phenomenon commonly observed in highly bilingual communities. It consists of mixing languages in the same conversational event. This paper presents results on Part-of-Speech tagging Spanish-English code-switched dis­course. We explore different approaches to exploit existing resources for both languages that range from simple heuristics, to language identification, to machine learning. The best results are achieved by training a machine learning algorithm with features that combine the output of an English and a Spanish Part-of-Speech tagger. </abstract></header><body><section number="1" title="Introduction"><p>Worldwide the percentage of bilingual speakers is fairly large, and it keeps increasing at a high rate. In the U.S., 18% of the total population speaks a language other than English at home, the major­ity of which speaks Spanish (U.S. Census Bureau, 2003). A significant percentage of this Spanish-English bilingual population code-switch between the two languages in what is often referred as Span-glish, the mix of Spanish and English. Spanish and English are not the only occurrence oflanguage mixtures. Examples of other popular combinations include Arabic dialects, French and German, Span­ish and Catalan, Maltese and English, and English and French. Typically when there are linguistic bor­ders, or when the country has more than one official language, we can find instances of code-switching.</p><p>Despite the wide use of code-switched discourse among bilinguals, this linguistic phenomenon has received little attention in the fields of Natural Lan­guage Processing and Computational Linguistics. Part-of-Speech (POS) tagging is a well studied prob­lem in these fields. For languages such as English, German, Spanish, and Chinese there are several dif­ferent POS taggers that reach high accuracies, espe­cially in news text corpora. However, to our knowl­edge, there is no previous work on developing a POS tagger for text with mixes of languages.</p><p>In this paper we present results on the problem of POS tagging English-Spanish code-switched dis­course by taking advantage of existing taggers for both languages. This rationale follows the evi­dence from studies of code-switching on different language pairs, which have shown code-switching to be grammatical according to both languages be­ing switched. We use different heuristics to combine POS tag information from existing monolingual tag­gers. We also explore the use of different language identification methods to select POS tags from the appropriate monolingual tagger. However, the best results are achieved by a machine learning approach using features generated by the monolingual POS taggers.</p><p>The next section presents the facts about code-switching, including some previous work done mainly in linguistics. Then in Section 3 we dis­cuss previous work related to the automated pro­cessing of code-switched discourse. In Section 4 we describe the English-Spanish code-switched data set gathered for the experimental evaluation. Sec­tion 5 presents the heuristics-based approaches for POS tagging that we explored. In Section 6 we describe our machine learning approach and show</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1051</doubt><p><i>Proceed</i><i>i</i><i>ngs of the 2008 Conference on Emp</i><i>i</i><i>r</i><i>i</i><i>ca</i><i>l </i><i>Methods </i><i>i</i><i>n Natura</i><i>l </i><i>Language Process</i><i>i</i><i>ng, </i>pages 1051-1060, Honolulu, October 2008. ©2008 Association for Computational Linguistics<page local="2" global="1052"/></p><p>results on POS tagging code-switched text. An in depth analysis of results is presented in Section 7, and we conclude this paper with a summary of the findings and directions for future work in Section 8.</p></section><section number="2" title="Rules of Code-switching"><p>In the linguistic, sociolinguistic, psychology, and psycholinguistic literature, bilingualism and the in­herent phenomena it exhibits have been studied for nearly a century (Espinosa, 1917; Ervin and Osgood, 1954; Gumperz, 1964; Gumperz and Hernandez-Chavez, 1971; Gumperz, 1971; Sankoff, 1968; Lipski, 1978). Despite the numerous previ­ous studies of linguistic characteristics of bilingual-ism, there is no clear consensus on the terminol­ogy related to language alternation patterns in bilin­gual speakers. The alternation of languages within a sentence is known as code-mixing, but it has also been referred as intrasentential code-switching, and intrasentential alternation (Poplack, 1980; Gros­jean, 1982; Ardila, 2005). Alternation across sen­tence boundaries is known as intersentential code-switching, or just code-switching. In the rest of this paper we will refer to the mixing of languages as code-switching. When necessary, we will differen­tiate the type of code-switching by referring to al­ternations within sentences as intrasentential code-switching and alternations across sentence bound­aries as intersentential code-switching.</p><p>Linguistic phenomena in bilingual speakers have been analyzed on different language pairs, includ­ing English-French, English-Dutch, Finish-English, Arabic-French, and Spanish-English, to name a few. There is a general agreement that code-switched pat­terns are not generated randomly; according to these studies, they follow specific grammatical rules. Fur­thermore, some studies suggest that, if these rules are violated, the resulting discourse will sound un­natural (Toribio, 2001b; Toribio, 2001a). The fol­lowing shows the rules governing code-switching discourse described in several studies (Poplack, 1980; Poplack, 1981; Sankoff, 1981; Sankoff, 1998a).</p><p>• Switches can take place only between full word boundaries. This is also known as the free mor­pheme constraint.</p><p>• Monolingual constructs within the sentence</p><p>will follow the grammatical rules of the mono­lingual fragment. • Permissible switch points are those that do not violate the order of adjacent constituents on both sides of the switch point of either of the languages. This is called the equivalence con­straint.</p><p>Although these rules are somewhat controversial, and most of the studies on this area have been con­ducted on small samples, we cannot ignore the fact that patterns bearing the above rules have emerged in different bilingual communities with different back­grounds.</p></section><section number="3" title="Automated Processing of Code-Switched Discourse"><p>A previous work related to the processing of code-switched text deals with language identification on English-Maltese code-switched SMS messages (Rosner and Farrugia, 2007). In addition to deal­ing with intrasentential code-switching, they have to deal with text where misspellings and ad hoc word contractions abound. What Rosner and Far-rigua have found to work best for language identi­fication in this noisy domain is a combination of a bigram Hidden Markov Model, trained on language transitions, and a trigram character Markov Model for handling unknown words. In another related work, Franco and Solorio present preliminary results on training a language model for Spanish-English code-switched text (Franco and Solorio, 2007). To evaluate their language model, they asked a human subject to judge sentences generated by a PCFG in­duced from training data and the language model. However, they only used one human judge.</p><p>Regarding the automated POS tagging and pars­ing of code-mixed utterances there is little prior work. To the best of our knowledge, there is no parser, nor POS tagger, currently available for the syntactic analysis of this type of discourse. There are theoretical approaches that propose formalisms to represent the structure of code-switched utter­ances and describe a framework for parsing and gen­erating mixed sentences, for example for Marathi and English (Joshi, 1982), or Hindi and English (Goyal <i>et al., </i>2003). Sankoff proposed a production model of bilingual discourse that accounts for the equivalence constraint and the unpredictability of code-switching (Sankoff, 1998a; Sankoff, 1998b).<page local="3" global="1053"/> His real-time production model draws on the alter­nation of fragments from two virtual monolingual sentences. It also accounts for other types of code-switching such as repetition-translation and inser-tional code-switching. But no statistical assessment has been conducted on real corpora. Our goal is to develop a POS tagger for code-switched utterances, which is the first step of the syntactic analysis of any language. Among the chal­lenges we face is the lack of a representative sam­ple of code-mixed discourse. Most POS taggers are built using large collections, usually at least a mil­lion words, such as the Brown corpus (Kucera and Francis, 1967), the Wall Street Journal corpus (Paul and Baker, 1992), or the Switchboard corpus (God­frey <i>et al., </i>1992). Currently, there is no annotation of code-switched text of comparable size. But in contrast to the lack of linguistic resources available for Spanish-English code-mixed discourse, English and Spanish have sufficient resources, especially En­glish. Thus, rather than starting from scratch, we will draw on existing taggers for both languages, which will reduce the amount of code-switched data needed. Some examples of POS taggers that per­form reasonably well on monolingual text of each language can be found in (Brants, 2000; Brill, 1992; Carreras and Padro, 2002; Charniak, 1993; Ratna-parkhi, 1996; Schmid, 1994). However, these tools are designed to work on monolingual text, therefore if applied as they are to code-switched text, their ac­curacy will decrease by a large margin. In the fol­lowing sections we will explore different methods for combining monolingual taggers.</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1052</doubt></section><section number="4" title="Data Set"><p>Data collections that have instances of Spanish-English code-switching, Spanglish for short, are not easily found since code-switching is primarily used in spoken form. To gather data we recorded a con­versation among three staff members of a southwest university in the U.S. The three speakers come from a highly bilingual background and code-switch reg­ularly when speaking among themselves, or other bilingual speakers.</p><p>This recording session has around 39 minutes of</p><p>Table 1 : Excerpts taken from the Spanglish data set. <b>Spanglish English Translation</b></p><p><i>(a) </i><i>Entonces le dio el virus y no se lo atendio and the virus spread through his body.</i></p><p><i>(b) </i><i>Cuando yo lo vi he looked pretty bad.</i></p><p>(c) I <i>think she was taller than he was.</i></p><p><i>Y un  caracter muy bonito también ella. Very easy going.</i></p><p>(a) Then he got the virus and he didn't re­ceive treatment and the virus spread through his body.</p><p>(b) When I saw him he looked pretty bad.</p><p>(c) I think she was taller than he was.</p><p>And a very nice char­acter she as well. Very easy going.</p><p>continuous speech (922 sentences, about 8k words) and was transcribed and annotated with POS tags by a human annotator. The annotations were later re­vised by a different annotator but no inter-annotator agreement was measured. The POS tag set used in the annotation is the combination ofthe tag sets from the English and the Spanish Tree Taggers (see Sec­tion 5). The vocabulary of the transcription has a to­tal of 1,516 different word forms<footnote anchor="1"/>. In the conversa­tion a total of 239 switches were identified manually, out of which 129 are intrasentential code-switches, and the rest are intersentential. English is the pre­dominant language used, with a total of 6,020 tokens and 576 monolingual sentences. In contrast, the transcription has close to 2k tokens in Spanish. Ta­ble 1 shows examples of code-switching taken from the recorded conversation; (a) and (b) are instances of intrasentential code-switching, and (c) shows in-tersentential code-switching.</p></section><section number="5" title="Rule-based Methods for Exploiting Existing Resources"><p>In this section we present several heuristics-based methods for POS tagging code-switched text. First, we describe the monolingual taggers used in this work. Then we present the different approaches ex­plored and contrast their performance.</p><footnote label="1">This transcription and the audio file are freely available for research purposes by contacting the first author.</footnote><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1053</doubt><page local="4" global="1054"/><subsection number="5.1" title="Monolingual Taggers"><p>We used the Tree Tagger (Schmid, 1994) for this work because ofthe following considerations:</p><p>1. It has both English and Spanish versions. The English tagger uses a slightly modified version of the Penn Treebank tag set and was trained and eval­uated on different portions of the Penn Treebank, reaching a POS tagging accuracy of 96.36%. The Spanish one uses a different tagset with 75 different POS tags<footnote anchor="2"/> and was trained on the Spanish CRATER corpus.</p><p>2. The transition probabilities are estimated using a modified version of the ID3 decision tree algorithm (Quinlan, 1986), which provides more freedom to learn contextual cues than n-grams.</p><p>3. Both taggers include a special tag for foreign words, PE for Spanish and <i>FW </i>for English. We do not expect this tag to identify correctly all foreign words, but when available this information will be exploited.</p><p>4. The Tree tagger generates probability estimates on the tags that can be used as features.</p><p>5. Finally, when the tagger fails to lemmatize a word it outputs the special token <i>(unknown). </i>This information can be used as a hint of words that do not belong to that particular language.</p></subsection><subsection number="5.2" title="Heuristic-based Systems"><p>For all heuristics the complete Spanglish data set was given to both taggers as a single text, then the final tag for each word was selected from the output of the taggers according to the different heuristics. Table 2 shows the tagging accuracies of the different heuristics we explored, which are explained below. <i>1. Using the monolingual tagger. </i>Here we simply give the Spanglish text to the Spanish and the English tree tagger. We expect from both taggers a performance degradation due to the inclusion offor-eign words in code-switching, as compared against their accuracy on monolingual texts. Another complicating factor to keep in mind is that we are dealing with spoken language. Hesitations, fillers, disfluencies, and interruption points, such as Umm, <i>Mmmhmm, </i>and <i>Uh-huh, </i>are frequently observed in</p><footnote label="2">The authors were unable to identify the source ofthe Span­ish tagset.</footnote><p>Table 2: Accuracy on POS tagging Spanglish text using simple heuristics for combining the output ofthe English and Spanish tagger.</p><p>speech and it is well known that they complicate the POS tagging task. The tagging accuracy from using the individual taggers is rather low, 26% for the Spanish tagger and 54% for the English one. The large difference between the two taggers can be attributed to the fact that the majority of the words in the corpus are in English.</p><p><i>2. Using confidence thresholds. </i>The Tree Tagger can output probabilities for each tag, showing the confidence of the tagger on each particular tag. To use this information we choose for each word the tag from the tagger with the highest confidence. When there is a tie we use either the English or the Spanish tag. Table 2 shows the results for the two cases. The "Highest prob tag or English" heuristic gives an ac­curacy of 51%, which is almost as accurate as using only the English tagger. The "Highest prob tag or Spanish" achieves an accuracy of 49%, which is an improvement over using only the monolingual Span­ish tagger, but it is still below the accuracy of the En­glish monolingual tagger. This is also possibly due to the task being easier for the English tagger.</p><p><i>3. Combining confidence thresholds with knowl­edge from special tags and lemmas. </i>This heuristic uses confidence thresholds combined with decisions based on the special tags, described in Section 5.1, and the unknown lemmas found. Let <i>POSe</i><i>(w</i><i>i) </i>and POSs(wi) be the POS tags assigned to word <i>w</i><i>i </i>by the English and Spanish tagger respectively; and let <i>ProbE</i><i>(wi)</i><i> </i>and <i>Prob</i><i>s</i>(wi) be the confi­dence scores of POS tags for word <i>w</i><i>i </i>computed by the English and Spanish tree taggers, respectively. For each word wi in the text, the final POS tag,<page local="5" global="1055"/></p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1054</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>Heuristic</b></p></td><td class="cell"><p><b>Accuracy (%)</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p>Spanish Tree Tagger</p></td><td class="cell"><p>25.99</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>English Tree Tagger</p></td><td class="cell"><p>54.59</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Highest prob tag or English</p></td><td class="cell"><p>51.51</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Highest prob tag or Spanish</p></td><td class="cell"><p>49.16</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Prob + special tags + lemmas</p></td><td class="cell"><p>64.27</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Dictionary-based Language Id</p></td><td class="cell"><p><b>86.03</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Character 5-grams Language Id</p></td><td class="cell"><p>81.46</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Human Language Id</p></td><td class="cell"><p>89.72</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>POSf(wi), will be assigned as follows:</p><doubt alpha="62.5" length="40" tooSmall="False" monospace="0.0">1.IfPOSe(wi)=   FW,thenPOSf(w,)—POSs(wi)</doubt><doubt alpha="65.9" length="44" tooSmall="False" monospace="0.0">2.Else ifPOSs(wi) = PE,thenPOSf(wi)—POSe(w,)</doubt><doubt alpha="57.9" length="57" tooSmall="False" monospace="0.0">3.Else   ifPOSe(wi)     =(unknown),thenPOSf(w,)&lt;-POSs(w,)</doubt><doubt alpha="61.1" length="54" tooSmall="False" monospace="0.0">4.Else   ifPOSs(wi)     =(unknown),thenPOSf(w,)-POSeK)</doubt><doubt alpha="60.0" length="40" tooSmall="False" monospace="0.0">5.Else   ifProbE(wi)&gt;     ProbS(wi),then</doubt><doubt alpha="66.7" length="18" tooSmall="False" monospace="0.0">posf(wi) —pose(wi)</doubt><doubt alpha="62.5" length="24" tooSmall="False" monospace="0.0">6.ElsePOSf(w,) —POSs(wi)</doubt><p>This heuristic performs betterthanthe othermeth-ods explored so far, yielding an accuracy of64.27%. It seems that knowledge of the taggers can be used to improve results. However, POS tagging accuracy is still poor.</p><p><i>4. Selecting POS tags based on automated language identification. </i>We used two different strategies for automatically identifying the language at the word level. One is based on dictionary look-up and the other is character-based language models. For the first approach, every word in the text is searched in the English and Spanish dictionaries. If a word is found in the English dictionary, then we identify that word as belonging to English and the POS tags from the English tagger are used for that word and the following ones, until a word is found in the Span­ish dictionary. Similarly, for a word not found in the English dictionary, but found in the Spanish dic­tionary, we use the Spanish tags until an English word is found. Note that this simple heuristic will always label words that belong to both languages as English, which is also the case for words not found in either dictionary. This dictionary-based method has a language identification accuracy of 94% on the Spanglish corpus.</p><p>The character language models were trained on the Agence France Presse (AFP) portions of the Gi-gaword for English and Spanish, respectively. For each of the words in the Spanglish corpus, we first decide its language by choosing the one with the lowest perplexity, calculated using character n-gram language models, then we use the corresponding</p><p>POS tag. We experimented with different language model orders, with <i>n </i>ranging from 2 to 6, and found that we achieve the highest accuracy, 81.46%, on POS tagging using a 5-gram language model. This 5-gram method reached a language identification ac­curacy of 85% for the Spanglish corpus. However, the language identification method using dictionary look-up achieved the best POS tagging result so far: 86.03%. The Spanglish conversation is dominated by every-day language that is easily found in dic­tionaries, while the text used to train the charac­ter based n-gram language models includes vocab­ulary that is not commonly used in conversations. This can explain why the simple dictionary look­up approach yielded better results for our corpus. Performing manual identification of the language and sending to the appropriate tagger just the corre­sponding fragments yields a very high POS tagging accuracy, 89.72%. This shows that it is important to deal with the language switches for boosting ac­curacy. However relying on human annotated lan­guage tags would be expensive and for some tasks unfeasible.</p></subsection></section><section number="6" title="Machine Learning for POS Tagging Code-Switched Discourse"><p>From Table 2 we can see that, with the exception of the language identification heuristic, accuracies are low for the previous experiments. However, we be­lieve that we can improve results further by using Machine Learning (ML) algorithms trained specif­ically for this task. In this section we describe the ML setting and present a comparison of the differ­ent algorithms we tested.</p><subsection number="6.1" title="Approach"><p>The key point is that the features selected for de­scribing the learning instances are the output from the English and the Spanish taggers. This scheme is similar to a stacked classifier approach (Wolpert, 1992), where the final classifier takes as input the predictions made by the different learners on the first pass and is trained to select the right tag from them, or a different one if the right answer is not available.</p><p>The gold-standard POS tags are used as the class label, and instances in this learning task are described by the following attributes:</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1055</doubt><page local="6" global="1056"/></subsection></section><section number="1." title="The word (word)"></section><section number="2." title="English POS tag (E t )"></section><section number="3." title="English POS tagger lemma (Ei )"></section><section number="4." title="English POS tagger confidence (E p )"></section><section number="5." title="Spanish POS tag (St)"></section><section number="6." title="Spanish POS tagger lemma ( Si )"></section><section number="7." title="Spanish POS tagger confidence (S p )"><p>Feature 1 is just the lexical word form as it ap­pears in the transcript. Features 2 to 4 are generated by the English Tree tagger, while features 5 to 7 are generated by the Spanish Tree tagger. Thus all fea­tures are automatically extracted.</p><subsection number="6.2" title="Results"><p>We evaluated experimentally the idea of using ML with different learning algorithms in WEKA (Wit-ten and Frank, 1999). We selected some of the most widely known algorithms, including Support Vector Machines (SVM) with a polynomial kernel of ex­ponent one (Scholkopf and Smola, 2002), Weka's modified version of Quinlan's C4.5 (J48) (Quinlan, 1986), Additive Logistic Regression with Decision Stumps (Logit Boost) (Friedman <i>et al., </i>1998) and Naive Bayes. The only parameter we modified was for J48 -we enabled the option for reducing error pruning.</p><p>Table 3: POS tagging accuracy of Spanglish text with different Machine Learning algorithms. Oracle shows the accuracy achieved when always selecting the right POS tag from the output of both Tree Taggers. Language Id shows accuracy of identifying the language and then choosing the output of the corresponding tagger.</p><p>Table 3 shows the average accuracy of 10-fold cross-validation for each classifier together with the variance. SVM and Logit Boost performed the best and the difference between the two algorithms is not significant according to the paired t-test (P-value = 0.1). For comparison, we show the accuracy of the</p><doubt alpha="0.0" length="26" tooSmall="False" monospace="0.0">10 20 30 40 50 60 70 80 90</doubt><p>Percentage of training data used</p><figure caption="Figure 1: Effect of different amounts of training data on accuracy"></figure><p>language identification approach together with the oracle accuracy. The oracle is the accuracy achieved when always selecting the right POS tag, when it is available, from the output of both Tree Taggers. We did not expect the oracle's accuracy to be an up­per bound on the accuracy for the ML learning al­gorithm. Our intuition is that the ML algorithm can be trained to identify when the taggers have made a mistake and what the right answer should be. As the results show, the ML approach can indeed out­perform the oracle, and the language identification method.</p><p>In Figure 1 we show the effect of the amount of training data on the accuracy using Logit Boost. We selected Logit Boost for this and the follow­ing experiments since its accuracy is comparable to SVMs but it is computationally less expensive. We randomly partitioned the transcription into 10 sub­groups. Then we used one subgroup as the test set and the rest for training. Starting with one subgroup in the training set, we incrementally added one sub­group to the training set and evaluate the tagging performance of the test set. We repeated this pro­cess several times, choosing randomly a new test set each time. The percentages shown are the average over all the experiments. With only 10% of the sen­tences for training we are reaching very good accu­racy already, as high as that from the strategy based on language identification. The curve flattens after<page local="7" global="1057"/></p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1056</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ML Algorithms</b></p></td><td class="cell"><p><b>Mean Accuracy (%)</b></p></td><td class="cell"><p><b>Variance</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Naive Bayes</p></td><td class="cell"><p>88.50</p></td><td class="cell"><p>1.9280</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SVM</p></td><td class="cell"><p><b>93.48</b></p></td><td class="cell"><p>1.2784</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Logit Boost</p></td><td class="cell"><p><b>93.19</b></p></td><td class="cell"><p>1.4437</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>J48</p></td><td class="cell"><p>91.11</p></td><td class="cell"><p>2.1527</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Oracle</p></td><td class="cell"><p>90.31</p></td><td class="cell"><p>-</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Language Id</p></td><td class="cell"><p>85.80</p></td><td class="cell"><p>-</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Table 4: Accuracy of Logit Boost with different subsets of attributes. 'X' marks attributes included. Et, E;, <i>Ep, </i>and St, S;, and <i>Sp </i>are the POS tag, lemma and confi­dence output by the English and the Spanish POS tagger, respectively.</p><p>60% of the training data is used. We do not gain much by adding more training data after this.</p><p>Results shown in Table 3 demonstrate that POS tagging can be learned effectively based on the at­tributes described in Subsection 6.1, even if we are not explicitly adding contextual information. To de­termine the extent to which each attribute is con­tributing to the learning task, we performed another set of experiments where we selected different sub­sets of the attributes. Table 4 shows the results with Logit Boost. Overall, the attributes taken from the English POS tagger are more valuable for this learn­ing task. If we only take the word form and the fea­tures from the English Tree tagger (first row in Ta­ble 4) we are reaching an accuracy that outperforms all heuristics. Still, there is some valuable infor­mation provided by the Spanish POS tagger output since the highest accuracy is achieved by including the Spanish-based attributes in combination with the English-based ones. Surprisingly, we can manage to outperform the oracle by using only three attributes:</p><p>the lexical word form and the POS tags from the English and Spanish tagger (see row 7 in table), or the POS tags from the monolingual taggers together with the lemma from the Spanish tagger (see row 4 from bottom to top). We also experimented adding as an attribute the output of the language identifica­tion method, but found no significant changes in the accuracy.</p></subsection></section><section number="7" title="Discussion"><p>We analyzed the different results gathered through the experiments and we present here the most rele­vant insights.</p><p>The first discovery, is that a lot of the errors made by the oracle, and the other methods as well, are due to the difficulties inherent in dealing with sponta­neous speech where fillers, interruption points, hes­itations, and the like abound. About as much as 20% of the errors made by the oracle are due to these features. Another roughly 20% is due to un­known tokens in the transcription, such as mum­bling, slang words such as "gonna" and "wanna", or other sounds unintelligible for the human tran­scriber. For the rest of the analysis we decided to ignore these types of mistakes for all methods and focus only on the remaining mistakes. In the case of the oracle we are left with 445 erroneously POS tagged words. From those, about 50%, or 233 to be exact, are errors in sentences with code-switches. We consider this to be a strong indication of the complexity that intrasentential switches add to the task of POS tagging. For the taggers, these sentences are incomplete, or ill-formed, since they have frag­ments with foreign words and thus, they fail to iden­tify them. The rest of the oracle mistakes can not be attributable to a single cause. Some are fragmented sentences, and some are due to errors inherent of the tagger, but nothing is particulary salient about them.</p><p>The language identification methods share, of course, the same mistakes made by the oracle, plus 342 more, for a total of 787 (in the case of the dictionary-based language identification). The chal­lenge of POS tagging code-switched text is more ev­ident for this method. Out of the mistakes made by the language identification method, 540 lie in sen­tences with code-switching, that is, nearly 70% of the mistakes. For 307 of these mistakes the right<page local="8" global="1058"/></p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1057</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>word</b></p></td><td class="cell"><p><i>E</i><i>t</i></p></td><td class="cell"><p><i>E</i><i>i</i></p></td><td class="cell"><p><i>Ep</i></p></td><td class="cell"><p><i>S</i><i>t</i></p></td><td class="cell"><p><i>S</i><i>i</i></p></td><td class="cell"><p></p></td><td class="cell"><p><b>Accuracy</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>88.80</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>-</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>86.22</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>78.59</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>65.28</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p><b>92.95</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>X</p></td><td class="cell"><p>92.53</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>91.22</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>89.76</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>77.08</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>74.18</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>85.76</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>71.17</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>X</p></td><td class="cell"><p>24.96</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>92.55</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>X</p></td><td class="cell"><p>88.89</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>78.74</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>89.62</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>-</p></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>90.76</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>75.94</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>80.24</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>X</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>X</p></td><td class="cell"><p>79.13</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>POS tag was available from one of the taggers. Some typical examples ofthese errors are words that belong to both languages, such as "a", "no", "me" and "con".</p><p>The ML approach outperformed both the lan­guage identification method and the oracle. Analyz­ing the predictions made by SVM we verified that out of the 445 errors made by the oracle, SVM cor­rectly tagged 223, the majority of which are words in sentences with code-switching (142 words). When compared against the errors from the method based on language identification, SVM correctly tagged 481 words out of the 787, 374 of which are words in sentences with code-switches. In summary, the ML approach is more robust to code-switched sen­tences. Note that we did find some errors made by the ML approach that are not shared by the oracle or the language identification method, a total of 105. Some of these mistakes are due to inconsistencies on the human-annotated tags. For instance, in most cases slang words such as "gonna" and "wanna" are labeled as unknown words, but we found that these words were labeled as verbs in a few cases. Not sur­prisingly this caused the ML algorithm to fail, since these class labels were misleading. The majority of the mistakes, however, seem to be due to systematic mistakes by the POS taggers.</p><p>One last remark is regarding our decision to find a method for successfully exploiting the existing tag­gers for POS tagging Spanglish text. Our origi­nal motivation came from the lack of linguistic re­sources to process Spanglish text. However, we did train from scratch a sequential model for POS tag­ging Spanglish, namely Conditional Random Fields (CRFs) (Lafferty <i>et al., </i>2001). We used MALLET (McCallum, 2002) for this experiment and the same training/testing partitions used in the experiment re­ported in Table 3. The CRF POS tagger was trained using capitalization information and the previous to­ken as context. The average accuracy of this CRF was 81%, which is lower than the language identifi­cation heuristic. We believe that this low accuracy is due to the lack of a representative sample of anno­tated Spanglish. It will be interesting to see if when more data becomes available the ML algorithms still yield the best results.</p></section><section number="8" title="Conclusions"><p>Code-switching is a fresh and exciting research area that has received little attention in the language pro­cessing community. Research on this topic has many interesting applications, including automatic speech recognition, machine translation, and computer as­sisted language learning. In this paper we present preliminary work towards developing a POS tagger for English-Spanish code-switched text that, to the best of our knowledge, is the first effort towards this end.</p><p>We explored different heuristics for taking advan­tage of existing linguistic resources for English and Spanish with unimpressive results. A simple word­level language identification strategy outperformed all heuristics tested. But the best results, even bet­ter than the oracle, were achieved by using machine learning using the output of monolingual POS tag­gers as input features.</p><p>In the error analysis we showed that most of the mistakes made by the language identification method, and the oracle itself, occur in sentences with intrasentential code-switching, showing the diffi­culty of the task. In contrast, our machine learning approach was less sensitive to the complexity of this alternation pattern.</p><p>There is still a lot of work to do in this area. Our ongoing efforts include gathering a larger corpus, with different speakers and conversational styles, as well as written forms of code-switching from blogs and Internet forums. In addition, we are exploring the use of context information. The features we are currently using to represent each word do not take into account the context surrounding the word. We want to test if by using contextual features we can further improve our results.</p><p>In this study we focused on code-switching, but borrowing is another complex language alternation pattern that we want the POS tagger to handle. We are working on developing a special method for identification and morphological analysis of borrow­ings. This method will help increase the accuracy of the POS tagger.</p><p>Spanish-English is not the only popular combi­nation of languages. An interesting line of future work would be to explore if the method presented here can be adapted to different language combinations.<page local="9" global="1059"/> Moreover, multilingual communities will code-switch among more than two codes and this poses fascinating research challenges as well.</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1058</doubt></section><section title="Acknowledgements"><p>We are grateful to Ray Mooney, Melissa Sher­man and the four anonymous reviewers for insight­ful comments and suggestions. Special thanks to Brenda Medina and Nicolle Whitman for helping with some experiments. This research is supported by the National Science Foundation under grant 0812134.</p></section><references><p>Ardila, A. (2005) Spanglish: an anglicized dialect. <i>His­panic Journal of Behavioral Sciences, </i><b>27(1): </b>60-81.</p><p>Brants, T. (2000) TnT - a statistical part-of-speech tagger. In <i>Sixth Applied Natural Language Processing Confer­ence ANLP-2000 </i>Seatle, WA.</p><p>Brill, E. (1990) A simple rule-based part-of-speech tag­ger. In <i>3rd Conference on Applied Natural Language Processing, </i>pp. 152-155. Trento, Italy.</p><p>Carreras, X. and Padro, L. (2002) A flexible distributed architecture for natural language analyzers. In <i>Third International Conference on Language Resources and Evaluation, LREC-02, </i>pp. 1813-1817. Las Palmas de Gran Canaria, Spain.</p><p>Charniak, E. (1993) Equations for part-of-speech tag­ging. In <i>11th National Conference on AI, </i>pp. 784-789.</p><p>Ervin, S. and Osgood, C. (1954) Second language learn­ing and bilingualism. <i>Journal of abnormal and social phsychology, </i>Supplement 49, pp. 139-146.</p><p>Espinosa, A. (1917) Speech mixture in New Mexico: the influence of English language on New Mexican Span­ish. In H. Stevens and H. Bolton, (eds.), <i>The Pacific Ocean in History, </i>pp. 408-428.</p><p>Franco, J.C. and Solorio T. (2007) Baby-steps towards building a Spanglish language model. In <i>8th Interna­tional Conference on Computational Linguistics and Intelligent Text Processing, CICLing-2007, </i>pp. 75-84. Mexico City, Mexico.</p><p>Friedman, J., Hastie, T., and Tibshirani, R. (1998) Addi­tive logistic regression: a statistical view of boosting. Technical Report, Stanford University.</p><p>Godfrey, J., Holliman, E. and McDaniel, J. (1992) Switchboard: Telephone speech corpus for research development. In <i>ICASSP, </i>pp. 517-520, San Francisco,</p><p>CA, USA.</p><doubt alpha="59.0" length="105" tooSmall="False" monospace="0.0">Goyal, P., Mital, Manav R., Mukerjee, A., Raina, Achla M., Sharma, D., Shukla, P. and Vikram, K. (2003) A</doubt><p>bilingual parser for Hindi, English and code-switching structures. In <i>Computational Linguistics for South Asian Languages -Expanding Synergies with Europe, EACL-2003 Workshop, </i>Budapest, Hungary. Grosjean, F. (1982) <i>Life with Two Languages: An Intro­duction to Bilingualism. </i>Harvard University Press. Gumperz, J. J. and Hernandez-Chavez, E. (1971) <i>Cogni­tive aspects ofbilingual communication. </i>Oxford Uni­versity Press, London. Gumperz, J. J. (1964) Linguistic and social interaction in two communities. In John J. Gumperz (ed.), <i>Language in social groups, </i>pp. 151-176, Stanford. Stanford Uni­versity Press.</p><p>Gumperz, J. J. (1971) Bilingualism, bidialectism and classroom interaction. In <i>Language in social groups, </i>pp. 311-339, Stanford. Stanford University Press. Joshi, A. K. (1982) Processing of sentences with in-trasentential code-switching. In Jan Horecky (ed.), <i>COLING-82, </i>pp. 145-150, Prague. Kucera, H. and Francis, W. N. (1967) <i>Computational analysis ofpresent-dayAmerican English. </i>Brown Uni­versity Press.</p><p>Lafferty, J. and McCallum, A. and Pereira F. (2001) Con­ditional random fields: Probabilistic models for seg­menting and labeling sequence data. In <i>18th ICML,</i> pp. 282-289. MA, USA.</p><p>Lipski, J. M. (1978) Code-switching and the problem of bilingual competence. In M. Paradis (ed.), <i>Aspects of bilingualism, </i>pp. 250-264, Columbia, SC. Hornbeam.</p><p>McCallum, A. (2002) MALLET: A Machine Learning for Language Toolkit. Retrieved January 7, 2008 from http://mallet.cs.umass.edu</p><doubt alpha="56.6" length="53" tooSmall="False" monospace="0.0">Paul, D. B. and Baker, J. M. (1992) The design of the</doubt><p>Wall Street Journal-based CSR corpus. In <i>HLT'91: workshop on speech and Natural Language </i>pp. 357­362, Morristown, NJ, USA.</p><p>Poplack, S., Sankoff, D. and Miller, C. (1988) The social correlates and linguistic processes oflexical borrowing and assimilation. <i>Linguistics, </i><b>26(1): </b>47-104. Poplack, S. (1980) Sometimes I'll start a sentence in Spanish y termino en espanol: toward a typology of code-switching. <i>Linguistics, </i><b>18(7/8): </b>581-618. Poplack, S. (1981) Syntactic structure and social function of code-switching. In R. Duran (ed.), <i>Latino discourse and communicative behavior, </i>pp. 169-184, Norwood,</p><p>NJ. Ablex.</p><p>Quinlan, J. R. (1986) Induction of decision trees. <i>Ma­chine Learning, </i><b>1: </b>81-106. Ratnaparkhi, A. (1996) A maximum entropy model for part-of-speech tagging. In <i>EMNLP, </i>pp. 133-142, Philadelphia, PA, May. Rosner, M. and Farrugia, P. (2007) A tagging algorithm for mixed language identification in a noisy domain.</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1059</doubt><page local="10" global="1060"/><p>In <i>INTERSPEECH2007, </i>pp. 190-193, Antwerp, Bel­gium.</p><p>Sankoff, D. (1968) Social aspects of multilingualism in New Guinea. Ph.D. thesis, McGill University.</p><p>Sankoff, D. (1981) Aformal grammar for codeswitching. <i>Papers in Linguistics: International Journal ofHuman communications, </i><b>14(1): </b>3-46.</p><p>Sankoff, D. (1998a) A formal production-based explana­tion ofthe facts ofcode-switching. <i>Bilingualism, Lan­guage and Cognition, </i><b>1: </b>39-50. Cambridge University Press.</p><p>Sankoff, D. (1998b) The production of code-mixed dis­course. In <i>36th ACL, </i>volume I, pp. 8-21, Montreal, Quebec, Canada.</p><p>Schmid, H. (1994) Probabilistic part-of-speech tagging using decision trees. In <i>International Conference on New Methods in Language Processing, Manchester, UK.</i></p><p>Scholkopf, B. and Smola, A. J. (2002) <i>Learning with Ker­nels: Support Vector Machines, Regularization, Opti­mization and Beyond. </i>MIT Press.</p><p>Toribio, A. J. (2001a) Accessing Spanish-English code-switching competence. <i>International Journal ofBilin-gualism, </i><b>5(4</b>):403-436.</p><p>Toribio, A. J. (2001b) On the emergence of bilingual code-switching competence. <i>Bilingual Language and Cognition, </i><b>4(3</b>):203-231.</p><p>U.S. Census Bureau. (2003) Language use and En­glish speaking ability: 2000. Retrieved October 30, 2006 from http://www.census.gov/prod/ 2003pubs/c2kbr-29.pdf.</p><p>Witten, I. H. and Frank, E. (1999) <i>Data Mining, Practi­cal Machine Learning Tools and Techniques with Java Implementations. </i>Morgan Kaufmann.</p><p>Wolpert, D. H. (1992) Stacked Generalization. <i>Neural Networks, </i><b>5(2</b>):241-259.</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1060</doubt></references></body></article>