<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="725"/><title>A New Approach to Lexical Disambiguation of Arabic Text</title><pubinfo>Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing,pages 725-735, MIT, Massachusetts, USA, 9-11 October 2010. ©2010 Association for Computational Linguistics</pubinfo><author surname="Shah" givenname="Rushin"><org  name="University of Pennsylvania" country="USA" city="Philadelphia"/></author><author surname="Dhillon" givenname="Paramveer S."><org  name="University of Pennsylvania" country="USA" city="Philadelphia"/></author><author surname="Liberman" givenname="Mark"><org  name="University of Pennsylvania" country="USA" city="Philadelphia"/></author><author surname="Foster" givenname="Dean"><org  name="University of Pennsylvania" country="USA" city="Philadelphia"/></author><author surname="Maamouri" givenname="Mohamed"><org  name="University of Pennsylvania" country="USA" city="Philadelphia"/></author><author surname="Ungar" givenname="Lyle"><org  name="University of Pennsylvania" country="USA" city="Philadelphia"/></author></firstpageheader><frontmatter><p><b>A New Approach to Lexical Disambiguation of Arabic Text</b></p><p><b>Rushin Shah Paramveer S. Dhillon, Mark Liberman,</b></p><p>Carnegie Mellon University <b>Dean Foster, Mohamed Maamouri</b></p><p>5000 Forbes Avenue <b>and Lyle Ungar</b></p><p>Pittsburgh, PA 15213, USA University of Pennsylvania</p><p>rnshah@cs.cmu.edu 3451 Walnut Street</p><p>Philadelphia, PA 19104, USA</p><p>{dhillon|myl|ungar}@cis.upenn.edu, foster@wharton.upenn.edu, maamouri@ldc.upenn.edu</p></frontmatter><abstract>We describe a model for the lexical analy­sis of Arabic text, using the lists of alterna­tives supplied by a broad-coverage morpho­logical analyzer, SAMA, which include sta­ble lemma IDs that correspond to combina­tions of broad word sense categories and POS tags. We break down each of the hundreds of thousands of possible lexical labels into its constituent elements, including lemma ID and part-of-speech. Features are computed for each lexical token based on its local and document-level context and used in a novel, simple, and highly efficient two-stage super­vised machine learning algorithm that over­comes the extreme sparsity of label distribu­tion in the training data. The resulting system achieves accuracy of 90.6% for its first choice, and 96.2% for its top two choices, in selecting among the alternatives provided by the SAMA lexical analyzer. We have successfully used this system in applications such as an online reading helper for intermediate learners of the Arabic language, and a tool for improving the productivity of Arabic Treebank annotators. </abstract></header><body><section number="1" title="Background and Motivation"><p>This paper presents a methodology for generating high quality lexical analysis of highly inflected lan­guages, and demonstrates excellent performance ap­plying our approach to Arabic. Lexical analysis of the written form of a language involves resolving, explicitly or implicitly, several different kinds of am­biguities. Unfortunately, the usual ways of talking about this process are also ambiguous, and our gen­eral approach to the problem, though not unprece­dented, has uncommon aspects. Therefore, in order to avoid confusion, we begin by describing how we define the problem.</p><p>In an inflected language with an alphabetic writ­ing system, a central issue is how to interpret strings of characters as forms of words. For example, the English letter-string 'winds' will normally be in­terpreted in one of four different ways, all four of which involve the sequence of two formatives <b><i>wind+s. </i></b>The stem 'wind' might be analyzed as (1) a noun meaning something like "air in motion", pro­nounced [wInd] , which we can associate with an ar­bitrary but stable identifier like wind.nl<b><i> </i></b>; (2) a verb <b><i>wind-vl </i></b>derived from that noun, and pronounced the same way; (3) a verb <b><i>wind-v2 </i></b>meaning something like "(cause to) twist", pronounced [waInd]; or (4) a noun <b><i>windji2 </i></b>derived from that verb, and pro­nounced the same way. Each of these "lemmas", or dictionary entries, will have several distinguishable senses, which we may also wish to associate with stable identifiers. The affix '-s' might be analyzed as the plural inflection, if the stem is a noun; or as the third-person singular inflection, if the stem is a verb.</p><p>We see this analysis as conceptually divided into four parts: 1) <i>Morphological analysis, </i>which rec­ognizes that the letter-string 'winds' might be (per­haps among other things) <b><i>wind/N + s/PLURAL </i></b>or <b><i>wind/V + s/3SING; </i></b>2) <i>Morphological disambigua­tion, </i>which involves deciding, for example, that in the phrase "the four winds", 'winds' is probably a plural noun, i.e. <b><i>wind/N + s/PLURAL; </i></b>3) <i>Lemma analysis, </i>which involves recognizing that the stem wind in 'winds' might be any of the four lem­mas listed above - perhaps with a further listing of senses or other sub-entries for each of them; and 4) <i>Lemma disambiguation, </i>deciding, for example, that the phrase "the four winds" probably involves the lemma <b><i>windjil.</i></b><page local="2" global="726"/><b><i></i></b></p><p>Confusingly, the standard word-analysis tasks in computational linguistics involve various combina­tions of pieces of these logically-distinguished op­erations. Thus, "part of speech (POS) tagging" is mainly what we've called "morphological disam­biguation", except that it doesn't necessarily require identifying the specific stems and affixes involved. In some cases, it also may require a small amount of "lemma disambiguation", for example to distinguish a proper noun from a common noun. "Sense disam­biguation" is basically a form of what we've called "lemma disambiguation", except that the sense dis­ambiguation task may assume that the part of speech is known, and may break down lexical identity more finely than our system happens to do. "Lemmatiza-tion" generally refers to a radically simplified form of "lemma analysis" and "lemma disambiguation", where the goal is simply to collapse different in­flected forms of any similarly-spelled stems, so that the strings 'wind', 'winds', 'winded', 'winding' will all be treated as instances of the same thing, without in fact making any attempt to determine the identity of "lemmas" in the traditional sense of dictionary entries.</p><p>Linguists use the term <i>morphology </i>to include all aspects of lexical analysis under discussion here. But in most computational applications, "morpho­logical analysis" does not include the disambigua­tion of lemmas, because most morphological ana­lyzers do not reference a set of stable lemma IDs. So for the purposes of this paper, we will continue to discuss lemma analysis and disambiguation as con­ceptually distinct from morphological analysis and disambiguation, although, in fact, our system dis-ambiguates both of these aspects of lexical analysis at the same time.</p><p>The lexical analysis of textual character-strings is a more complex and consequential problem in Arabic than it is in English, for several reasons. First, Arabic inflectional morphology is more com­plex than English inflectional morphology is. Where an English verb has five basic forms, for example, an Arabic verb in principle may have dozens. Sec­ond, the Arabic orthographic system writes elements such as prepositions, articles, and possessive pro­nouns without setting them off by spaces, roughly as if the English phrase "in a way" were written "in-away". This leads to an enormous increase in the number of distinct "orthographic words", and a sub­stantial increase in ambiguity. Third, short vowels are normally omitted in Arabic text, roughly as if English "in a way" were written "nway".</p><p>As a result, a whitespace/punctuation-delimited letter-string in Arabic text typically has many more alternative analyses than a comparable English letter-string does, and these analyses have many more parts, drawn from a much larger vocabulary of form-classes. While an English "tagger" can spec­ify the morphosyntactic status of a word by choos­ing from a few dozen tags, an equivalent level of detail in Arabic would require thousands of alterna­tives. Similarly, the number of lemmas that might play a role in a given letter-sequence is generally much larger in Arabic than in English.</p><p>We start our labeling of Arabic text with the alter­native analyses provided by SAMA v. 3.1, the Stan­dard Arabic Morphological Analyzer (Maamouri et al., 2009). SAMA is an updated version of the ear­lier Buckwalter analyzers (Buckwalter, 2004), with a number of significant differences in analysis to make it compatible with the LDC Arabic Treebank 3-v3.2 (Maamouri et al., 2004). The input to SAMA is an Arabic orthographic word (a string of letters delimited by whitespace or punctuation), and the output of SAMA is a set of alternative analyses, as shown in Table 1. For a typical word, SAMA pro­duces approximately a dozen alternative analyses, but for certain highly ambiguous words it can pro­duce hundreds of alternatives.</p><p>The SAMA analyzer has good coverage; for typ­ical texts, the correct analysis of an orthographic word can be found somewhere in SAMA's list of alternatives about <b>95% </b>of the time. However, this broad coverage comes at a cost; the list of analytic alternatives must include a long Zipfian tail of rare or contextually-implausible analyses, which collec­tively are correct often enough to make a large con­tribution to the coverage statistics. Furthermore, SAMA's long lists of alternative analyses are not evaluated or ordered in terms of overall or contex­tual plausibility. This makes the results less useful in most practical applications.</p><p>Our goal is to rank these alternative analyses so that the correct answer is as near to the top of the list<page local="3" global="727"/></p><p>Table 1: Partial output of SAMA for <b><i>yHlm </i></b>and qbl. On average, every token produces more than 10 such analyses as possible. Despite some risk of confusion, we'll refer to SAMA's list of alternative analyses for an orthographic word as potential <i>labels </i>for that word. And despite a greater risk of confusion, we'll refer to the assignment of probabilities to the set of SAMA labels for a particular Arabic word in a particular textual context as <i>tagging, </i>by analogy to the oper­ation of a stochastic part-of-speech tagger, which similarly assigns probabilities to the set of labels available for a word in textual context.</p><p>Although our algorithms have been developed for the particular case of Arabic and the particular set of lexical-analysis labels produced by SAMA, they should be applicable without modification to the sets of labels produced by any broad-coverage lexical analyzer for the orthographic words of any highly-inflected language.</p><p>In choosing our approach, we have been moti­vated by two specific applications. One applica­tion aims to help learners of Arabic in reading text, by offering a choice of English glosses with asso­ciated Arabic morphological analyses and vocaliza­tions. SAMA's excellent coverage is an important basis for this help; but SAMA's long, unranked list of alternative analyses for a particular letter-string, where many analyses may involve rare words or al­ternatives that are completely implausible in the con­text, will be confusing at best for a learner. It is much more helpful for the list to be ranked so that the correct answer is almost always near the top, and is usually one of the top two or three alternatives.</p><p>In our second application, this same sort of rank­ing is also helpful for the linguistically expert native speakers who do Arabic Treebank analysis. These annotators understand the text without difficulty, but find it time-consuming and fatiguing to scan a long list of rare or contextually-implausible alternatives for the correct SAMA output. Their work is faster and more accurate if they start with a list that is ranked accurately in order of contextual plausibility.</p><p>Other applications are also possible, such as vo­calization of Arabic text for text-to-speech synthe­sis, or lexical analysis for Arabic parsing. However, our initial goals have been to rank the list of SAMA outputs for human users.</p><p>We note in passing that the existence of set of sta­ble "lemma IDs" is an unusual feature of SAMA, which in our opinion ought to be emulated by ap­proaches to lexical analysis in other languages. The lack of such stable lemma IDs has helped to disguise the fact that without lemma analysis and disam­biguation, morphological analyses and disambigua­tion is only a partial solution to the problem of lexi­cal analysis.</p><p>In principle, it is obvious that lemma disambigua­tion and morphological disambiguation are mutually beneficial. If we know the answer to one of the ques­tions, the other one is easier to answer. However, these two tasks require rather different sets of con­textual features. Lemma disambiguation is similar to the problem of word-sense disambiguation - on some definitions, they are identical - and as a re­sult, it benefits from paragraph-level and document-level bag-of-words attributes that help to character­ize what the text is "about" and therefore which lem­mas are more likely to play a role in it. In contrast, morphological disambiguation mainly depends on features of nearby words, which help to characterize how inflected forms of these lemmas might fit into local phrasal structures.<page local="4" global="728"/></p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Token</b></p></td><td class="cell"><p><b>Lemma</b></p></td><td class="cell"><p><b>Vocalization</b></p></td><td class="cell"><p><b>Segmentation</b></p></td><td class="cell"><p><b>Morphology</b></p></td><td class="cell"><p><b>Gloss</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>yHlm</p></td><td class="cell"><p>Halam-u_1</p></td><td class="cell"><p>yaHolumu</p></td><td class="cell"><p>ya + Holum + u</p></td><td class="cell"><p>IV3MS + IV + IV-SUFF_MOOD:I</p></td><td class="cell"><p>he / it + dream + [ind.]</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>yHlm</p></td><td class="cell"><p>Halam-u_1</p></td><td class="cell"><p>yaHoluma</p></td><td class="cell"><p>ya + Holum + a</p></td><td class="cell"><p>IV3MS + IV + IV-SUFF_MOOD:S</p></td><td class="cell"><p>he / it + dream + [sub.]</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>yHlm</p></td><td class="cell"><p>Halum-u_1</p></td><td class="cell"><p>yaHolumo</p></td><td class="cell"><p>ya + Holum +</p><p>o</p></td><td class="cell"><p>IV3MS + IV + IV-SUFF_MOOD:J</p></td><td class="cell"><p>he / it + be gentle + [jus.]</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>qbl</p></td><td class="cell"><p>qabil-a_1</p></td><td class="cell"><p>qabila</p></td><td class="cell"><p>qabil + a</p></td><td class="cell"><p>PV       + PV-SUFF_SUBJ:3MS</p></td><td class="cell"><p>accept/receive/approve + he/it [verb]</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>qbl</p></td><td class="cell"><p>qaboLl</p></td><td class="cell"><p>qabol</p></td><td class="cell"><p>qabol</p></td><td class="cell"><p>NOUN</p></td><td class="cell"><p>Before</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><section number="2" title="Problem and Methodology"><p>Consider a collection of tokens (observations), <i>ti, </i>re­ferred to by index i G {1,..., n}, where each token is associated with a set of p features, <i>x</i><i>ij</i>, for the <i>jth</i>feature, and a label, <i>li, </i>which is a combination of a lemma and a morphological analysis. We use in­dicator functions <i>yik </i>to indicate whether or not the <i>kth </i>label for the <i>ith </i>token is present. We represent the complete set of features and labels for the en­tire training data using matrix notation as X and Y, respectively. Our goal is to predict the label l (or equivalently, the vector y for a given feature vector X.</p><p>A standard linear regression model of this prob­lem would be</p><doubt alpha="33.3" length="12" tooSmall="False" monospace="0.0">y=xß + e (1)</doubt><p>The standard linear regression estimate of ß (ig­noring, for simplicity the fact that the ys are 0/1) is:</p><p>ß = <i>(XtrainXtrain)     XtrainYtrain </i>(2)</p><p>where <i>Ytrain </i>is an n <i>x </i>h matrix containing 0s and 1s indicating whether or not each of the h possible labels is the correct label for each of the n tokens <i>ti, Xtrain </i>is an n x p matrix of context features for each of the n tokens, the coefficients ß are p x h.</p><p>However, this is a large, sparse, multiple label problem, and the above formulation is neither statis­tically nor computationally efficient. Each observa­tion <b>(x, y) </b>consists of thousands of features associ­ated with thousands of potential labels, almost all of which are zero. Worse, the matrix of coefficients ß, to be estimated is large (p x h) and one should thus use some sort of transfer learning to share strength across the different labels.</p><p>We present a novel principled and highly compu­tationally efficient method of estimating this multi-label model. We use a two stage procedure, first using a subset <i>(Xtrain </i>, <i>Ytrain </i>) of training datato give a fast approximate estimate of ß; we then use a second smaller subset of the training data <i>(X</i><b><i>train2</i></b><i>,Y</i><b><i>train2,) </i></b>to "correct" these estimates in a way that we will show can be viewed as a spe­cialized shrinkage. Our first stage estimation ap­proximates ß, but avoids the expensive computation of <i>(X</i><i>f</i><i>rainXtrain)~l</i>. Our second stage corrects (shrinks) these initial estimates in a manner special­ized to this problem. The second stage takes ad­vantage of the fact that we only need to consider those candidate labels produced by SAMA. Thus, only dozens of the thousands of possible labels are considered for each token.</p><p>We now present our algorithm. We start with a corpus D of documents d of labeled Arabic text. As described above, each token, <i>ti </i>is associated with a set of features characterizing its context, computed from the other words in the same document, and a la­bel, <i>li = </i>(lemmai, morphology^, which is a combi­nation of a lemma and a morphological analysis. As described below, we introduce a novel factorization of the morphology into 15 different components.</p><p>Our estimation algorithm, shown in Algorithm 1, has two stages. We partition the training corpus into two subsets, one of which <i>(X</i><i>train </i>) is used to es­timate the coefficients ßs and the other of which (Xtrain2) is used to optimally "shrink" these coeffi­cient estimates to reduce variance and prevent over-fitting due to data sparsity.</p><p>For the first stage of our estimation procedure, we simplify the estimate of the (ß) matrix (Equation 2) to avoid the inversion of the very high dimensional ( p x p) matrix (X X) by approximating (X X) by its diagonal, Var(X), the inverse of which is trivial to compute; i.e. we estimate ß using <i>ß = </i><i>Var(Xtrain</i><b><i>l)</i></b><b><i> </i></b><b><i><footnote anchor="1"/></i></b><i>Xtrain</i><b><i>1</i></b><i>Ytrain</i><b><i>1</i></b><b><i> </i></b>(3)</p><p>For the second stage, we assume that the coeffi­cients for each feature can be shrunk differently, but that coefficients for each feature should be shrunk the same regardless of what label they are predict­ing. Thus, for a given observation we predict:</p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">p</doubt><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">j=1</doubt><p>where the weights <i>W</i><i>j </i>indicate how much to shrink each of the <i>p </i>features.</p><p>In practice, we fold the variance of each of the <i>j </i>features into the weight, giving a slightly modified equation:</p><doubt alpha="50.0" length="2" tooSmall="False" monospace="0.0">j=</doubt><page local="5" global="729"/><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">p</doubt><p>where <i>ß* = X</i><i>t</i><i>[,ainiYtraini </i>is just a matrix of the counts of how often each context feature shows up with each label in the first training set. The vec­tor a, which we will estimate by regression, is just the shrinkage weights <i>w </i>rescaled by the feature vari­ance.</p><p>Note that the formation here is different from the first stage. Instead of having each observation be a token, we now let each observation be a (token, label) pair, but only include those labels that were output by SAMA. For a given token <i>ti </i>and poten­tial label <i>I</i><i>k</i>, our goal is to approximate the indica­tor function <i>g(i,</i><i> </i>k), which is 1 if the <i>kth </i>label of token <i>ti </i>is present, and 0 otherwise. We find candi­date labels using a morphological analyzer (namely SAMA), which returns a set of possible candidate labels, say <i>C</i><i> </i><i>(t)</i>, for each Arabic token t. Our pre­dicted label for ti is then argmaxkeC(t.k).</p><p>The regression model for learning the weights <i>a</i><i>j </i>in the second stage thus has a row for each label <i>g(i,k)</i><i> </i>associated with a SAMA candidate for each token i = <i>ntrain\+\ ... ntrain2 </i>in the second train­ing set. The value of <i>g(i,</i><i> </i><i>k)</i><i> </i>is predicted as a func­tion of the feature vector zijk = ß*k<i>x</i><i>ij</i>.</p><p>The shrinkage coefficients, <i>a</i><i>j</i>, could be estimated from theory, using a version of James-Stein shrink­age (James and Stein, 1961), but in practice, superior results are obtained by estimating them empirically. Since there are only <i>p </i>of them (unlike the <i>p * h </i>ßs), a relatively small training set is sufficient. We found that regression-SVMs work slightly better than lin­ear regression and significantly better than standard classification SVMs for this problem.</p><p>Prediction is then done in the obvious way by tak­ing the tokens in a test corpus <i>Dtest, </i>generating con­text features and candidate SAMA labels for each token <i>t</i><i>i, </i>and selected the candidate label with the highest score <i>g(i,k)</i><i> </i>that we set out to learn. More formally, The model parameters <i>ß* </i>and <i>a </i>produced by the algorithm allow one to estimate the most likely label for a new token <i>t</i><i>i </i>out of a set of can­didate labels <i>C</i>(ti) using</p><p><i>kpred </i><i>= </i>argmaxkec(ti^ <i>ajßjkxij </i>(6)</p><p>The most expensive part of the procedure is es­timating <i>ß </i>, which requires for each token in cor<b>Algorithm 1 </b>Training algorithm.</p><p><b>Input: </b>A training corpus <i>Dtrain </i>of <i>n </i>observations</p><p><i>(Xtrain, Ytrain)</i></p><p>Partition <i>Dtrain </i>into two sets, D<b>i </b>and D2, of sizes <i>n</i><i>train</i><b><i>i </i></b>and <i>n</i><i>train</i><b><i>2 </i></b><i>= n - n</i><i>train</i><b><i>i </i></b>observations // Using Di, estimate <i>ß*</i> <i>ß**k = </i>En=Tinl <i>xijyik </i>for the <i>jth </i>feature and <i>kth</i> // Using <i>D</i><b>2, </b>estimate <i>a</i><i>j</i> // Generate new "features" <i>Z </i>and the true labels <i>g(i,</i><i> </i><i>k)</i><i> </i>for each of the SAMA candidate labels for each of the tokens in <i>D</i><b>2</b> <i>Z</i><i>ijk </i><i>= ß*k </i><i>X</i><i>ij </i>for <i>i</i><i> </i>in <i>i</i><i> = </i><i>n</i><i>train</i><b><i>i </i></b><i>+ 1 </i><i>.</i><i>..</i><i> </i><i>n</i><i>train</i><b><i>2</i></b></p><doubt alpha="100.0" length="5" tooSmall="False" monospace="0.0">label</doubt><p>Estimate <i>a</i><i>j </i>for the above (feature,label) pairs <i>(z</i><i>ijk</i><i>,</i><i> </i><i>g(i,k))</i><i> </i>using Regression SVMs <b>Output: </b><i>a </i>and <i>ß*</i> pus Di, (a subset of D), finding the co-occurrence frequencies of each label element (a lemma, or a part of the morphological segmentation) with the target token and jointly with the token and with other tokens or characters in the context of the to­ken of interest. For example, given an Arabic to­ken, <b><i>"yHlm", </i></b>we count what fraction of the time it is associated with each lemma (e.g. <b><i>Halam-</i></b><i>«_/</i>), <i>count(lemma<b>=Halam-U-l, </b>token<b>=yHlm) </b></i>and each segment (e.g. "ya"), <i>count(segment<b>=ya, </b>to-ken<b>=yHlm). </b></i>(Of course, most tokens never show up with most lemmas or segments; this is not a prob­lem.) We also find the base rates of the components of the labels (e.g., <i>count(lemma<b>=Halam-u</b></i><i>A),</i><i> </i>and what fraction of the time the label shows up in vari­ous contexts, e.g. <i>count(lemma<b>=Halam-u</b></i><i>J,</i><i> previ­ous token = <b>yHlm). </b></i>We describe these features in more detail below.</p></section><section number="3" title="Features and Labels used for Training"><p>Our approach to tagging Arabic differs from conven­tional approaches in the two-part shrinkage-based method used, and in the choice of both features and labels used in our model. For features, we study both local context variables, as described above, and document-level word frequencies. For the labels, the key question is what labels are included and how they are factored. Standard "taggers" work by doing an n-way classification of all the alternatives, which is not feasible here due to the thousands of possible labels.<page local="6" global="730"/> Standard approaches such as Conditional Random Fields (CRFs) are intractable with so many labels. Moreover, few if any taggers do any lemma disambiguation; that is partly because one must start with some standard inventory of lemmas, which are not available for most languages, perhaps because the importance of lemma disambiguation has been underestimated.</p><p>We make a couple of innovations to deal with these issues. First, we perform lemma disambigua­tion in addition to "tagging". As mentioned above, lemmas and morphological information are not in­dependent; the choice of lemma often influences morphology and vice versa. For example, Table 1 contains two analyses for the word <b>qbl. </b>For the first analysis, where the lemma is <b><i>qabil-aA </i></b>and the gloss is <i>accept/receive/approve + he/it [verb], </i>the word is a verb. However, for the second analysis, where the lemma is <b><i>qabol A </i></b>and the gloss is <i>before, </i>the word is a noun.</p><p>Simultaneous lemma disambiguation and tagging introduces additional complexity: An analysis of ATB and SAMA shows that there are approximately 2,200 possible morphological analyses ("tags") and 40,000 possible lemmas; even accounting for the fact that most combinations of lemmas and morpho­logical analyses don't occur, the size of the label space is still in the order of tens of thousands. To deal with data sparsity, our second innovation is to factor the labels. We factor each label <i>I </i>into a set of 16 label elements <b>(LE</b>s). These include lemmas, as well as morphological elements such as basic part-of-speech, suffix, gender, number, mood, etc. These are explained in detail below. Thus, since each la­bel <i>I </i>is a set of 15 categorical variables, each <b>y </b>in the first learning stage is actually a vector with 16 nonzero components and thousands of zeros. Since we do simultaneous estimation of the entire set of label elements, the value <i>g(i,k)</i><i> </i>being predicted in the second learning phase is 1 if the entire label set is correct, and zero otherwise. We do <i>not </i>learn sep­arate models for each label.</p><subsection number="3.1" title="Label Elements (LEs)"><p>The fact that there are tens of thousands of possible labels presents the problem of extreme sparsity of label distribution in the training data. We find that a model that estimates coefficients <i>ß* </i>to predict a sin-</p><p>Table 2: Label Elements <b>(LE</b>s). Examples of additional data on basic POS include whether a noun is proper or common, whether a verb is transitive or not, etc. Both the basic POS and its suffix may have person, gender and number data.</p><p>gle label (a label being in the Cartesian product of the set of label elements) yields poor performance. Therefore, as just mentioned, we factor each label <i>I </i>into a set of label elements <b>(LE</b>s), and learn the correlations <i>ß </i>between features and label elements, rather than features and entire label sets. This re­duces, but does not come close to eliminating, the problem sparsity. A complete list of these <b>LE</b>s and their possible values is detailed in Table 2.</p></subsection><subsection number="3.2" title="Features"><subsubsection number="3.2.1" title="Local Context Features"><p>We take <i>(t,</i><i> </i><i>I)</i><i> </i>pairs from D2, and for each such pair generate features <i>Z </i>based on co-occurrence statistics <i>ß* </i>in Di, as mentioned in Algorithm 2. These statistics include unigram co-occurrence fre­quencies of each label with the target token and bi­gram co-occurrence of the label with the token and with other tokens or characters in the context of the target token. We define them formally in Table 3. Let <i>Zbaseline </i>denote the set of all such basic features based on the local context statistics of the target to­ken, namely the words and letters preceding and fol­lowing it. We will use this set to create a baseline model.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>LE</b></p></td><td class="cell"><p><b>Description</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>lemma</i></p></td><td class="cell"><p>Lemma</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>pre1</i></p></td><td class="cell"><p>Closer prefix</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>pre2</i></p></td><td class="cell"><p>Farther prefix</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>det</i></p></td><td class="cell"><p>Determiner</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>pos</i></p></td><td class="cell"><p>Basic POS</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>dpos</i></p></td><td class="cell"><p>Additional data on basic pos</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>suf</i></p></td><td class="cell"><p>Suffix</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>perpos</i></p></td><td class="cell"><p>Person (basic pos)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>numpos</i></p></td><td class="cell"><p>Number (basic pos)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>genpos</i></p></td><td class="cell"><p>Gender (basic pos)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>persuf</i></p></td><td class="cell"><p>Person (suffix)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>numsuf</i></p></td><td class="cell"><p>Number (suffix)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>gensuf</i></p></td><td class="cell"><p>Gender (suffix)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>mood</i></p></td><td class="cell"><p>Mood of verb</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>pron</i></p></td><td class="cell"><p>Pronoun suffix</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="7" global="731"/><p>For each label element <b>(LE) </b>e, we define a set of features <i>Ze </i>similar to <i>Z</i><i>baseiine; </i>these features are based on co-occurrence frequencies ofthe particular <b>LE </b><i>e,</i><i> </i>not the entire label <i>l.</i></p><p>Finally, we define an aggregate feature set <i>Zaggr</i>as follows:</p><p><i>Zaggr — Zbaseline </i>{Ze} (7)</p><p>where <i>e </i>G <i>{lemma, prel, pre2, det, pos, dpos, suf, perpos, numpos, genpos, persuf, numsuf, gensuf, mood, pron}.</i></p></subsubsection><subsubsection number="3.2.2" title="Document Level Features"><p>When trying to predict the lemma, it is useful to include not just the words and characters immedi­ately adjacent to the target token, but also the all the words in the document. These words capture the "topic" of the document, and help to disambiguate different lemmas, which tend to be used or not used based on the topic being discussed, similarly to the way that word sense disambiguation systems in En­glish sometimes use the "bag of words" the docu­ment to disambiguate, for example a "bank" for de­positing money from a "bank" of a river. More pre­cisely, we augment the features for each target token with the counts of each word in the document (the "term frequency" f) in which the token occurs with a given label.</p><p><i>Zfull — Zaggr     </i><i>[J     </i><i>Ztf </i>(8)</p><p>This set <i>Z</i><i>full </i>is our final feature set. We use <i>Z</i><i>full</i>to train an SVM model <i>M</i><i>full </i>; this is our final pre­dictive model.</p></subsubsection></subsection><subsection number="3.3" title="Corpora used for Training and Testing"><p>We use three modules of the Penn Arabic Tree-bank (ATB) (Maamouri et al., 2004), namely ATB1, ATB2 and ATB3 as our corpus of labeled Ara­bic text, D. Each ATB module is a collection of newswire data from a particular agency. ATB1 uses the Associated Press as a source, ATB2 uses Ummah, and ATB3 uses Annahar. <i>D </i>contains a total of 1,835 documents, accounting for approximately 350,000 words. We construct the training and test­ing sets <i>Dtrain </i>and <i>Dtest </i>from <i>D </i>using 10-fold cross validation, and we construct <i>D1 </i>and <i>D2 </i>from <i>D</i><i>train</i>by randomly performing a 9:1 split.</p><p>As mentioned earlier, we use the SAMA mor­phological analyzer to obtain candidate labels <i>C</i><i> </i><i>(t)</i><i> </i>for each token <i>t </i>while training and testing an SVM model on <i>D2 </i>and <i>Dtest </i>respectively. A sample out­put of SAMA is shown in Table 1. To improve cov­erage, we also add to <i>C(t)</i><i> </i>all the labels <i>l</i><i> </i>seen for <i>t</i><i> </i>in <i>D1</i>. We find that doing so improves coverage to <b>98%. </b>This is an upper bound on the accuracy of our model.</p><doubt alpha="50.0" length="40" tooSmall="False" monospace="0.0">C(t)—SAMA(t)(J{l\(t,l)G D1} (9)4 Results</doubt><p>We use two metrics of accuracy: <b>A1, </b>which mea­sures the percentage of tokens for which the model assigns the highest score to the correct label or <b>LE </b>value (or <b>E1— </b>100 — A1, the corresponding percent­age error), and <b>A2, </b>which measures the percentage of tokens for which the correct label or <b>LE </b>value is one of the two highest ranked choices returned by the model (or <b>E2 </b>= 100 — A2). We test our model <i>M</i><i>full </i>on <i>Dtest </i>and achieve <b>A1 </b>and <b>A2 </b>scores of 90.6% and 96.2% respectively. The accuracy achieved by our <i>M</i><i>full </i>model is, to the best of our knowledge, higher than prior approaches have been able to achieve so far for the problem of combined morphological and lemma disambiguation. This is all the more impressive considering that the upper bound on accuracy for our model is <b>98% </b>because, as described above, our set of candidate labels is in­complete.</p><p>In order to analyze how well different <b>LE</b>s can be predicted, we train an SVM model <i>Me </i>for each <b>LE </b><i>e </i>using the feature set Ze, and test all such models on <i>Dtest.</i><page local="8" global="732"/><i> </i>The results for all the <b>LE</b>s are reported in the form of error percentages <b>E1 </b>and <b>E2 </b>in Table 4.</p><table caption="Table 3: Co-occurrence statisticsß*.We use these to generate feature sets for our regression SVMs." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Statistic</b></p></td><td class="cell"><p><b>Description</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Freq</p></td><td class="cell"><p><i>count</i><i>d</i><i>1 </i><i>(t,</i><i> </i><i>l)</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PrevWord</p></td><td class="cell"><p><i>count </i><i>d</i><i>1 </i>(t,l,t<b>_i)</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NextWord</p></td><td class="cell"><p><i>counto1 </i><i>(t,</i><i> </i><i>l,</i><i> </i>t<b>+i)</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PreviLetter</p></td><td class="cell"><p><i>count</i><i>d</i><i>1 </i><i>(t,l,</i><i> </i>first letter(t_ <b>1 </b>))</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NextiLetter</p></td><td class="cell"><p><i>count</i><i>d</i><i>1 </i><i>(t,l,</i><i> </i>first letter(t+1)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PrevfLetter</p></td><td class="cell"><p><i>count</i><i>d</i><i>1 </i><i>(t,</i><i> </i><i>l,</i><i> </i>last letter(t_1)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NextfLetter</p></td><td class="cell"><p><i>countDl </i><i>(t,l,</i><i> </i>last letter<i>(t+1 )</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>reported are 10 fold cross validation test accuracies and no parameters have been tuned on them.</p><p>A comparison of the results for <i>M</i><i>full </i>with the results for <i>Mlemma </i>and <i>Mpos </i>is particularly infor­mative. We see that <i>M</i><i>full </i>is able to achieve a sub­stantially lower <b>E1 </b>error score (9.4%) than <i>Mlemma</i>(11.1%) and <i>Mpos </i>(23.4%); in other words, we find that our full model is able to predict lemmas and ba­sic parts-of-speech more accurately than the individ­ual models for each of these elements.</p><p>We examine the effect of varying the size of D2, i.e. the number of SVM training instances, on the performance of <i>M</i><i>full </i>on <i>Dtest, </i>and find that with increasing sizes of D2, <b>E1 </b>reduces only slightly from 9.5% to 9.4%, and shows no improvement thereafter. We also find that the use of document-level features in <i>Mlemma </i>reduces <b>E1 </b>and <b>E2 </b>per­centages for <i>Mlemma </i>by 5.7% and 3.2% respec­tively.</p><p><b>4.1   Comparison to Alternate Approaches 4.1.1   Structured Prediction Models</b></p><p>Preliminary experiments showed that knowing the predicted labels (lemma + morphology) of the sur­rounding words can slightly improve the predic­tive accuracy of our model. To further investi­gate this effect, we tried running experiments us­ing different structured models, namely CRF (Con­ditional Random Fields) (Lafferty et al., 2001), (Structured) MIRA (Margin Infused Relaxation Al­gorithm) (Crammer et al., 2006) and Structured Perceptron (Collins, 2002). We used linear chain</p><p>CRFs as implemented in MALLET Toolbox (Mc-</p><p>Callum, 2001) and for Structured MIRA and Perceptron we used their implementations from EDLIN Toolbox (Ganchev and Georgiev, 2009). However, given the vast label space of our problem, running these methods proved infeasible. The time complex­ity of these methods scales badly with the number of labels; It took a week to train a linear chain CRF for only ~ 50 labels and though MIRA and Per-ceptron are online algorithms, they also become in­tractable beyond a few hundred labels. Since our label space contains combinations of lemmas and morphologies, so even after factoring, the dimension of the label space is in the order of thousands.</p><p>We also tried a naive version (two-pass approxi­mation) of these structured models. In addition to the features in <i>Z</i><i>full, </i>we include the predicted la­bels for the tokens preceding and following the tar­get token as features. This new model is not only slow to train, but also achieves only slightly lower error rates (1.2% lower <b>E1 </b>and 1.0% lower <b>E2) </b>than <i>M</i><i>full. </i>This provides an upper bound on the bene­fit of using the more complex structured models, and suggests that given their computational demands our (unstructured) model <i>M</i><i>full </i>is a better choice.</p><doubt alpha="40.0" length="10" tooSmall="False" monospace="0.0">4.1.2 MADA</doubt><p>(Habash and Rambow, 2005) perform morpho­logical disambiguation using a morphological ana­lyzer. (Roth et al., 2008) augment this with lemma disambiguation; they call their system MADA. Our work differs from theirs in a number of respects. Firstly, they don't use the two step regression proce­dure that we use. Secondly, they use only "unigram" features. Also, they do not learn a single model from a feature set based on labels and <b>LE</b>s; instead, they combine models for individual elements by using weighted agreement. We trained and tested MADA v2.32 using its full feature set on the same <i>D</i><i>train </i>and <i>Dtest. </i>We should point out that this is not an ex­act comparison, since MADA uses the older Buck­walter morphological analyzer.<footnote anchor="1"/></p><subsubsection number="4.1.3" title="Other Alternatives"><p><b>Unfactored Labels: </b>To illustrate the benefit ob­tained by breaking down each label <i>l </i>into <b>LE</b>s, we contrast the performance ofour<i>M</i><i>full </i>model to an SVM model <i>Mbaseline </i>trained us­ing only the feature set <i>Zbaseline, </i>which only contains features based on entire labels, those based on individual <b>LE</b>s.<page local="9" global="733"/></p><footnote label="1">A new version of MADA was released very close to the submission deadline for this conference.</footnote><table caption="Table 4: Results ofMefor eachLEe.Note:The results" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Model</b></p></td><td class="cell"><p><b>E1</b></p></td><td class="cell"><p><b>E2</b></p></td><td class="cell"><p><b>Model</b></p></td><td class="cell"><p><b>E1</b></p></td><td class="cell"><p><b>E2</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Mlemma</i></p></td><td class="cell"><p>11.1</p></td><td class="cell"><p>4.9</p></td><td class="cell"><p><i>Mpre</i><b><i>1</i></b></p></td><td class="cell"><p>1.9</p></td><td class="cell"><p>1.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Mpre</i><b><i>2</i></b></p></td><td class="cell"><p>0.2</p></td><td class="cell"><p>0</p></td><td class="cell"><p><i>M</i><i>det</i></p></td><td class="cell"><p>0.7</p></td><td class="cell"><p>0.1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Mpos</i></p></td><td class="cell"><p>23.4</p></td><td class="cell"><p>4.0</p></td><td class="cell"><p><i>Mdpos</i></p></td><td class="cell"><p>10.3</p></td><td class="cell"><p>1.9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>M</i><i>suf</i></p></td><td class="cell"><p>7.6</p></td><td class="cell"><p>2.5</p></td><td class="cell"><p><i>Mperpos</i></p></td><td class="cell"><p>3.0</p></td><td class="cell"><p>0.1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Mnumpos</i></p></td><td class="cell"><p>3.2</p></td><td class="cell"><p>0.2</p></td><td class="cell"><p><i>Mgenpos</i></p></td><td class="cell"><p>1.8</p></td><td class="cell"><p>0.1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Mpersuf</i></p></td><td class="cell"><p>3.2</p></td><td class="cell"><p>0.1</p></td><td class="cell"><p><i>Mnumsuf</i></p></td><td class="cell"><p>8.2</p></td><td class="cell"><p>0.5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Mgensuf</i></p></td><td class="cell"><p>11.6</p></td><td class="cell"><p>0.4</p></td><td class="cell"><p><i>Mmood</i></p></td><td class="cell"><p>1.6</p></td><td class="cell"><p>1.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Mpron</i></p></td><td class="cell"><p>1.8</p></td><td class="cell"><p>0.6</p></td><td class="cell"><p><i>Mcase</i></p></td><td class="cell"><p>14.7</p></td><td class="cell"><p>5.9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b><i>Mfuii</i></b></p></td><td class="cell"><p><b>9.4</b></p></td><td class="cell"><p><b>3.8</b></p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p><b>Independent lemma and morphology prediction:</b></p><p>Another alternative approach is to pre­dict lemmas and morphological analyses separately. We construct a feature set <i>Z</i><i>lemma' </i><i>— Z</i><i>full </i><i>— Z</i><i>lemma </i>and train an SVM model <i>M</i><b><i>lemmai </i></b>using this feature set. Labels are then predicted by simply combining the results predicted independently by <i>Mlemma</i>and <i>M</i><b><i>lemmai</i></b>. Let <i>Mind </i>denote this approach.</p><p><b>Unigram Features: </b>Finally, we also consider a context-less approach, i.e. using only "uni­gram" features for labels as well as <b>LE</b>s. We call this feature set <i>Z</i><i>uni, </i>and the correspond­ing SVM model <i>M</i><i>uni.</i></p><p>The results of these various models, along with those of <i>M</i><i>full </i>are summarized in Table 5. We see that <i>M</i><i>full </i>has roughly half the error rate of the state-of-the-art MADA system.</p><p>Table 5: Percent error rates of alternative approaches. <b>Note: </b>The results reported are 10 fold cross validation test accuracies and no parameters have been tuned on them. We used same train-test splits for all the datasets.</p></subsubsection></subsection></section><section number="5" title="Related Work"><p>(Hajic, 2000) show that for highly inflectional languages, the use of a morphological analyzer improves accuracy of disambiguation. (Diab et al., 2004) perform tokenization, POS tagging and base phrase chunking using an SVM based learner. (Ahmed and Nürnberger, 2008) perform word-sense disambiguation using a Naive Bayesian model and rely on parallel corpora and match­ing schemes instead of a morphological ana­lyzer. (Kulick, 2010) perform simultaneous tok-enization and part-of-speech tagging for Arabic by separating closed and open-class items and focus­ing on the likelihood of possible stems of open-class words. (Mohamed and Kubler, 2010) present a hybrid method between word-based and segment-based POS tagging for Arabic and report good re­sults. (Toutanova and Cherry, 2009) perform joint lemmatization and part-of-speech tagging for En­glish, Bulgarian, Czech and Slovene, but they do not use the two step estimation-shrinkage model de­scribed in this paper; nor do they factor labels. The idea of joint lemmatization and part-of-speech tag­ging has also been discussed in the context of Hun­garian in (Kornai, 1994).</p><p>A substantial amount of relevant work has been done previously for Hebrew. (Adler and Elhadad, 2006) perform Hebrew morphological disambigua­tion using an unsupervised morpheme-based HMM, but they report lower scores than those achieved by our model. Moreover, their analysis doesn't include lemma IDs, which is a novelty of our model. (Gold­berg et al., 2008) extend the work of (Adler and El-hadad, 2006) by using an EM algorithm, and achieve an accuracy of 88% for full morphological analy­sis, but again, this does not include lemma IDs. To the best of our knowledge, there is no existing re­search for Hebrew that does what we did for Arabic, namely to use simultaneous lemma and morpholog­ical disambiguation to improve both. (Dinur et al., 2009) show that prepositions and function words can be accurately segmented using unsupervised meth­ods. However, by using this method as a preprocess­ing step, we would lose the power of a <i>simultaneous solution </i>for these problems. Our method is closer in style to a CRF, giving much of the accuracy gains of simultaneous solution, while being about 4 orders of magnitude easier to train.</p><p>We believe that our use of factored labels is novel for the problem of simultaneous lemma and mor­phological disambiguation; however, (Smith et al., 2005) and (Hatori et al., 2008) have previously made use of features based on parts of labels in CRF models for morphological disambiguation and word-sense disambiguation respectively. Also, we note that there is a similarity between our two-stage machine learning approach and log-linear models in machine translation that break the data in two parts, estimating log-probabilities of generative models from one part, and discriminatively re-weighting the models using the second part.<page local="10" global="734"/></p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Model</b></p></td><td class="cell"><p><b>E1</b></p></td><td class="cell"><p><b>E2</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Mbaseline</i></p></td><td class="cell"><p>13.6</p></td><td class="cell"><p>9.1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Mind</i></p></td><td class="cell"><p>18.7</p></td><td class="cell"><p>6.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Muni</i></p></td><td class="cell"><p>11.6</p></td><td class="cell"><p>6.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Mcheat</i></p></td><td class="cell"><p>8.2</p></td><td class="cell"><p>2.8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>MADA</b></p></td><td class="cell"><p><b>16.9</b></p></td><td class="cell"><p><b>12.6</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b><i>Mfuii</i></b></p></td><td class="cell"><p><b>9.4</b></p></td><td class="cell"><p><b>3.8</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><section number="6" title="Conclusions"><p>We introduced a new approach to accurately predict labels consisting of both lemmas and morphologi­cal analyses for Arabic text. We obtained an accu­racy of over 90% - substantially higher than current state-of-the-art systems. Key to our success is the factoring of labels into lemma and a large set of mor-phosyntactic elements, and the use of an algorithm that computes a simple initial estimate of the coef­ficient relating each contextual feature to each la­bel element (simply by counting co-occurrence) and then regularizes these features by shrinking each of the coefficients for each feature by an amount deter­mined by supervised learning using only the candi­date label sets produced by SAMA.</p><p>We also showed that using features of word n-grams is preferable to using features of only individ­ual tokens of data. Finally, we showed that a model using a full feature set based on labels as well as factored components of labels, which we call label elements (LEs) works better than a model created by combining individual models for each LE. We believe that the approach we have used to create our model can be successfully applied not just to Arabic but also to other languages such as Turkish, Hungar­ian and Finnish that have highly inflectional mor­phology. The current accuracy of of our model, get­ting the correct answer among the top two choices 96.2% of the time is high enough to be highly use­ful for tasks such as aiding the manual annotation of Arabic text; a more complete automation would require that accuracy for the single top choice.</p></section><section title="Acknowledgments"><p>We woud like to thank everyone at the Linguis­tic Data Consortium, especially Christopher Cieri, David Graff, Seth Kulick, Ann Bies, Wajdi Za-ghouani and Basma Bouziri for their help. We also wish to thank the anonymous reviewers for their comments and suggestions.</p></section><references><p>Meni Adler and Michael Elhadad. 2006. An Unsuper­vised Morpheme-Based HMM for Hebrew Morpho­logical Disambiguation. In <i>Proceedings of the 21st International Conference on Computational Linguis­tics and the 44th annual meeting of the Association for Computational Linguistics. </i>Farag Ahmed and Andreas Nürnberger. 2008. Ara­bic/English Word Translation Disambiguation using Parallel Corpora and Matching Schemes. In <i>Proceed­ings ofEAMT'08, </i>Hamburg, Germany. Tim Buckwalter. 2004. Buckwalter Arabic Morphologi­cal Analyzer version 2.0. Michael Collins. 2002. Discriminative Training Meth­ods for Hidden Markov Models: Theory and Experi­ments with Perceptron Algorithms. In <i>Proceedings of EMNLP'02.</i></p><p>Koby Crammer, Ofer Dekel, Joseph Keshet, Shai Shalev-Shwartz, and Yoram Singer. 2006. Online Passive-Aggressive Algorithms. <i>Journal of Machine Learning Research, </i>7:551-585. Mona Diab, Kadri Hacioglu, and Daniel Jurafsky. 2004. Automatic Tagging of Arabic text: From Raw Text to Base Phrase Chunks. In <i>Proceedings of the 5th Meet­ing </i><i>ofthe</i><i> North American Chapter </i><i>ofthe</i><i> Associa­tion for Computational Linguistics/Human Language Technologies Conference (HLT-NAACL'04). </i>Elad Dinur, Dmitry Davidov, and Ari Rappoport. 2009. Unsupervised Concept Discovery in Hebrew Using Simple Unsupervised Word Prefix Segmentation for Hebrew and Arabic. In <i>Proceedings of the EACL 2009 Workshop on Computational Approaches to Semitic Languages.</i></p><p>Kuzman Ganchev and Georgi Georgiev. 2009. Edlin: An Easy to Read Linear Learning Framework. In <i>Pro­ceedings ofRANLP 09. </i>Yoav Goldberg, Meni Adler, and Michael Elhadad. 2008. EM Can Find Pretty Good HMM POS-Taggers (When</p><p>Given a Good Start)*. In <i>Proceedings ofACL'08. </i>Nizar Habash and Owen Rambow. 2005. Arabic Tok-enization, Part-of-Speech Tagging and Morphological Disambiguation in One Fell Swoop. In <i>Proceedings of</i> <i>ACL 05, </i>Ann Arbor, MI, USA.</p><p>Jan Hajic. 2000. Morphological Tagging: Data vs. Dic­tionaries. In <i>Proceedings of the 1st Meeting of the North American Chapter </i><i>ofthe</i><i> Association for Com­putational Linguistics (NAACL 00). </i>Jun Hatori, Yusuke Miyao, and Jun'ichi Tsujii. 2008. Word Sense Disambiguation for All Words using Tree-Structured Conditional Random Fields. In <i>Proceed­ings of COLing'08. </i>W. James and Charles Stein. 1961. Estimation with Quadratic Loss. In <i>Proceedings of the Fourth Berkeley</i> <i>Symposium on Mathematical Statistics and Probabil­ity, Volume 1.</i><page local="11" global="735"/><i></i></p><p>Andras Kornai. 1994. On Hungarian morphology (Lin­guistic<b>s </b>Series A: Studia et Dissertationes 14). <i>Lin­guistics Institute ofHungarian Academy ofSciences, Budapest.</i></p><p>Seth Kulick. 2010. Simultaneous Tokenization and Part-of-Speech Tagging for Arabic without a Morphologi­cal Analyzer. In <i>Proceedings ofACL 10. </i>John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional Random Fields: Proba­bilistic Models for Segmenting and Labeling Sequence</p><p>Data. In <i>Proceedings ofICML 01, </i>pages 282-289.</p><p>Mohamed Maamouri, Ann Bies, and Tim Buckwalter.</p><p>2004. The Penn Arabic Treebank: Building a Large Scale Annotated Arabic Corpus. In <i>Proceedings of NEMLAR Conference on Arabic Language Resources and Tools.</i></p><p>Mohamed Maamouri, David Graff, Basma Bouziri, Son-dos Krouna, and Seth Kulick. 2009. LDC Standard Arabic Morphological Analyzer (SAMA) v. 3.0. Andrew McCallum, 2001. <i>MALLET: A Machine Learn­ing for Language Toolkit. </i>Software available at http://mallet.cs.umass.edu. Emad Mohamed and Sandra Kubler. 2010. Arabic Part</p><p>of Speech Tagging. In <i>Proceedings ofLREC 10. </i>Ryan Roth, Owen Rambow, Nizar Habash, Mona Diab, and Cynthia Rudin. 2008. Arabic Morphological Tag­ging, Diacritization, and Lemmatization Using Lex­eme Models and Feature Ranking. In <i>Proceedings of ACL'08, </i>Columbus, Ohio, USA. Noah A. Smith, David A. Smith, and Roy W. Tromble.</p><p>2005. Context-Based Morphological Disambiguation with Random Fields*. In <i>Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing</i> <i>(HLT/EMNLP).</i><i></i></p><p>Kristina Toutanova and Colin Cherry. 2009. A Global Model for Joint Lemmatization and Part-of-Speech Prediction. In <i>Proceedings ofthe Joint Conference of the 47th Annual Meeting of the ACL and the 4th Inter­national Joint Conference on Natural Language Pro­cessing, </i>pages 486-494.</p></references></body></article>