<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="539"/><title>Can Alternations Be Learned? A Machine Learning Approach To Romanian Verb Conjugation</title><pubinfo>Proceedings of Recent Advances in Natural Language Processing,pages 539-544, Hissar, Bulgaria, 12-14 September 2011.</pubinfo><author surname="Dinu" givenname="Liviu P."><org  name="University of Bucharest" country="Romania" city="Bucharest"/></author><author surname="Ionescu" givenname="Emil"><org  name="University of Bucharest" country="Romania" city="Bucharest"/></author><author surname="Niculae" givenname="Vlad"><org  name="University of Bucharest" country="Romania" city="Bucharest"/></author><author surname="Sulea" givenname="Octavia-Maria"><org  name="University of Bucharest" country="Romania" city="Bucharest"/></author></firstpageheader><frontmatter><p><b>Can Alternations Be Learned? A Machine Learning Approach </b><b>To</b></p><p><b>Romanian Verb Conjugation</b></p><p><b>Liviu </b><b>P.</b><b> Dinu</b></p><p>Faculty of Mathematics and Computer Science University of Bucharest</p><p>ldinu@ funinf.cs.unibuc.ro</p><p><b>Emil Ionescu</b></p><p>Faculty of Letters University of Bucharest</p><p>emilionescu@ unibuc.ro</p></frontmatter><abstract>In this paper we look at the conjugation of the Romanian verb, in particular, at its ir­regularities, from a machine learning point of view. Our attempt is to predict the pres­ence or absence of any alternation in the stem (apophony), using n-gram represen­tations of the infinitive. We combine for­mal labelling mechanisms with learning methods in order to build a general con-jugational model. </abstract></header><body><section number="1" title="Introduction"><p>The problem that we approached in this paper deals with phonological alternations in the stem of the Romanian irregular verbs during their conju­gation. What we attempted to investigate, using machine learning techniques, was whether there is reason to believe that a pattern can be identi­fied in the conjugation of the Romanian verb and whether that pattern could be learnt through auto­matic means such that, given the infinitive form of a verb, its correct conjugation could be produced.</p><p>Like other Romance languages, Romanian has tra­ditionally received a Latin-inspired classification of verbs into 4 conjugational classes (or some­times 5, where the 4th conjugation is divided be­tween verbs with the infinitive ending in i and i, respectively) based on the ending of their infini­tival form alone (Costanzo, 2011). However, this infinitive-based classification has often been found inadequate due to the many conjugational patterns that have been found in each class and to its inabil­ity to account for the behavior of partially irregular verbs (those whose stem has a smaller number of allomorphs than the completely irregular such as "a fi") during their conjugation. There have been, thus, numerous attempts throughout the history of</p><p><b>Vlad Niculae Octavia-Maria §ulea</b></p><p>Faculty of Mathematics Faculty of Foreign and Computer Science Languages and</p><p>University of Literatures</p><p>Bucharest University of Bucharest</p><p>vlad@vene.ro mary.octavia@</p><footnote>gmail.com</footnote><p>Romanian Linguistics to give other conjugational classifications.</p><p>Lombard (1955) combined the traditional 4 infinitive-based conjugational classes with the in­formation related to the variation in the suffix re­ceived by 1st and 4th conjugational class verbs in the indicative first person singular form and came up with 6 classes. Other classifications based on the way Romanian verbs conjugate include (Ciompec et. al., 1985 in Costanzo, 2011), who proposed 10 conjugations, and Felix (1964) who came up with 12 conjugations by looking at the in­flection of the verbs and the number of allomorphs of the stem. Moisil (1960) proposed 5 regrouped classes of verbs and introduced the method of let­ters with variable values, in his computer scien­tific effort toward a "mechanical grammar". Pa-pastergiou et al. (2007) have developed a classifi­cation from a (second) language acquisition point of view, dividing the 1st and 4th traditional classes into 3 and respectively 5 subclasses, each with a different conjugational pattern, and offering rules for alternations in the stem.</p><p>Finally, Barbu (2007) offered a highly compre­hensive classification of the verb conjugation in Romanian.This classification was based on a cor­pus of more than 7000 verbs, representing verbs of contemporary Romanian, and distinguished 41 conjugational classes which cover the whole cor­pus. The corpus has also been used in the present research.</p><p>As stated before, our focus has been on captur­ing rules of variation in the stem for partially ir­regular verbs like a a§tepta (to wait), which be­comes eu a§tept (I wait) and el a§teapta (he waits) in the "indicativ prezent" tense. It can be seen that the letter e in the stem changes to ea during conjugation, this rich morphology making the lan­guage seem difficult to acquire. Attempts to formalize rules from a computer scientific point of view date back to Moisil in 1960.<page local="2" global="540"/> Such (incom­plete) rules can be formulated as context-sensitive grammars, since the alternations are determined by the (phonologic) context in which certain char­acters (phonemes) appear. This lead us to the idea of analyzing the Romanian verbs from a machine learning point of view: what can one find out by looking at n-gram representation of the infinitives?</p><p>In the following, we give a brief description of the context-sensitive grammar rules that we've de­veloped based on a slight modification to Moisil's concept of letters with variable values, a discus­sion of the implementation of these rules as a parser, and the machine learning techniques ap­plied to the infinitives of a particular class of verbs from a selected corpus.</p></section><section number="2" title="Capturing Verb Alternations in Context-Sensitive Rules"><p>For the moment we limited the discussion to verbs ending in -ta, for which Dinu and Ionescu (2011) gave two rules that, according to our findings, cover 80% of the alternations that appear.</p><p>The first rule is for the variable letter to and can be described as:</p><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">to</doubt><doubt alpha="66.7" length="21" tooSmall="False" monospace="0.0">[t]inthecontext#_[i]#</doubt><doubt alpha="52.5" length="40" tooSmall="False" monospace="0.0">[t]in the context#_[e]V[a]V[a]V[î]V [$]#</doubt><p>The second rule is for the variable letter <i>u0 </i>and amounts to:</p><doubt alpha="63.0" length="92" tooSmall="False" monospace="0.0">{[oa]in the context#_[a]V[e]# [o]in the context#_[i]V[$]#[u]in the context#.stressed vowel]#</doubt><p>For example, the verb "a purta" (to wear) has the lemma <i>pu0rt0. </i>The third person singular "el poarta" (he wears) is matched by the first context for u0 and the second context for t0. The second person singular "tu porti" (you wear) is matched by the second context for u0 and by the first con­text for t0.</p><p>These rules can be formulated as a context sensi­tive grammar G = (VN<i>,VT, </i>E, P) in the follow­ing way:</p><doubt alpha="50.0" length="110" tooSmall="False" monospace="0.0">E—$aT0#|$aU0ßT0# E-$aT0i#|$aU0ßT0i#E—$aT0a#|$aU0ßT0a# E—$aT0am#|$aU0ßT0am#E—$aT0ati#|$aU0ßT0ati#T0#—#tT0i#—#ti</doubt><doubt alpha="46.2" length="13" tooSmall="False" monospace="0.0">i0a#—#ta |&amp;ta</doubt><doubt alpha="58.9" length="112" tooSmall="False" monospace="0.0">P=^T0am#—#tam| !tamT0ati#—#tati|!tatix#—#x,for allxGVTx&amp;—&amp;x,for allxGVTx!—!x,for allxGVTU0!—#uU0#—#oU0&amp;—#oa^$#—A</doubt><p>VN is the set of non-terminals, VT is the set of terminals (the alphabet), E is the starting non­terminal and P is the set of production rules. For the Romanian language, <i>VT</i><i> </i>= {a, a, a, b,...}. <i>a</i><i> </i>and ß are arbitrary strings from Vp. Note that this is the reunion of the two grammars Gi and G2 given in (Dinu and Ionescu, 2011).</p><p>The power of this grammar does not lie in its lan­guage: some of its derivations are general enough to accept any string over the alphabet ending in the letter t, for example. However, derivations in this grammar represent a generative process that can build present indicative forms of verbs.</p><p>The way verbal forms are parsed by this grammar with regard to the arbitrary a and ß is very impor­tant. Take the verb "a certa" (to scold or to quar­rel). At the second person singular form, in the present indicative tense, it becomes "tu certi" (you scold) which is accurately modeled by the T0 al­ternation. However its third person singular form "el cearta" (he scolds) exhibits an alternation in the stem vowel "e" that is not captured by these rules. The form "cearta" is however generated by the grammar, as:</p><p>E — $cearT0a# — $cear#ta — — $#cearta — cearta.</p><page local="3" global="541"/><p>Therefore, we cannot say that the grammar mod­els this alternation. How can we tell? Note that if we would assume that these derivations com­pletely explain the alternations, it would mean that the verb has two allomorphs for the stem, "cert" and "ceart", yet we have no alternating "e" vowel rule to account for that variation. This leads to the natural restriction that for a verb to be consid­ered fully modeled by the grammar we previously described, a and ß need to remain the same dur­ing the derivation of all its forms. This is equiva­lent to saying that the variable letters should be the only alternations in the stem of a partially irregular verb.</p><p>Such grammars are hard to control methods that cannot directly solve the problem of conjugating a verb starting from its infinitive form. We used a simplification of this system to assign labels to verbs depending on how they are conjugated and what alternations they present.</p></section><section number="3" title="Labeling Method"><p>The correct derivations in the grammar presented in the previous section can be formulated as reg­ular expressions. Furthermore, we can associate a particular regular expression for each one of the six possible forms of a verb in this tense. For ex­ample, the regular expressions for the conjugation pattern of the word "a canta" (to sing) at the first person singular is X+)t$, while for the second per­son singular it is X+)ti$, therefore catching the t-{ alternation. Note that the dot accepts any letter in the Romanian alphabet. The restriction is that, for each of the six forms, the value of the capturing groups (the characters captured by the bracketed part of the expressions) remains constant. These groups correspond to all parts of the stem that re­main unchanged and ensure that, given the infini­tive and the regular expressions, one can produce a correct conjugation. When the stem has no alter­nation, the expression will contain only one such capturing group that represents the whole stem.</p><p>We will use the term "rule" to refer to a set of six regular expressions describing the conjugation of a verb. We started incrementally adding rules to cover more of the verbs in the dataset, and ar­rived at a total of 14 rules. We dropped the rules that only covered one or two verbs, and eliminated these verbs from the dataset. We ended up with seven rules covering 616 of the 628 verbs ending in -ta (98.1%).</p><p>An example of one such rule, covering the verb "a tresalta", is:</p><p>It can be observed that the forms of the verb are consistently accepted by the regular expressions of the rule, with the two groups in brackets always having the values "tres" and "l". This rule is the 5th in our line of 7 rules. Below are listed all 7 of them:</p><p>• the 1st rule accepts verbs like "a ajuta" (to help), which has an alternation in the stem of the sort t—sdue to palatalization (determined by the 2nd person singular suffix "i")</p><p>• the 2nd rule accepts verbs like "a exista" (to exist), which has an alternation in the stem of the type s—s, due to palatalization as well</p><p>• the 3rd rule accepts verbs like "a destepta" (to awake/arouse), whose stem has an alternation of the type a—ea</p><p>• the 4th rule accepts verbs like "a desserta" (to empty), with a stem alternation of the type</p><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">e—a</doubt><p>• the 5th rule accepts verbs like "a tresalta" (to start, to take fright), with a stem alternation of the kind a—a</p><p>• the 6th rule accepts verbs like "a desfata" (to delight), with a stem alternation of the type a—a in the 3rd person, and a—e in the 2nd person singular</p><p>• the 7th rule accepts verbs like "a decapita" (to decapitate), which conjugates with "ez" suffixes (-ez, -ezi, -eaza, -am, -ati, -eaza )</p><p>These rules were run against the dataset of conju­gated Romanian verbs (only those ending in -ta), and a label was assigned to each of the distinct in­finitives found, such that the end result consists of a dataset of 616 infinitives, each labeled from 0 to 6 depending on how the verb is inflected during conjugation.<page local="4" global="542"/></p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Person</p></td><td class="cell"><p>Regexp</p></td><td class="cell"><p>Example</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1st singular</p></td><td class="cell"><p>î(.+)a(.+)t$</p></td><td class="cell"><p>tresalt</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2nd singular</p></td><td class="cell"><p>î(.+)a(.+)ti$</p></td><td class="cell"><p>tresalti</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3rd singular</p></td><td class="cell"><p>î(.+)a(.+)ta$</p></td><td class="cell"><p>tresalta</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1st plural</p></td><td class="cell"><p>î(.+)a(.+)tam$</p></td><td class="cell"><p>tresaltam</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2nd plural</p></td><td class="cell"><p>î(.+)a(.+)tati$</p></td><td class="cell"><p>tresaltati</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3rd plural</p></td><td class="cell"><p>î(.+)a(.+)ta$</p></td><td class="cell"><p>tresalta</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><section number="4" title="Posing the Learning Problem"><subsection number="4.1" title="Objectives"><p>The problem that we are aiming to solve is to de­termine how to conjugate a verb, given its infini­tive form. The traditional infinitive-based classi­fication taught in school does not take one all the way. Many variations exist within these 4 classes.</p><p>The rules from the previous section allow us to separate the verbs ending in -ta into more specific classes, knowing their inflected forms. By assign­ing the correct label to an infinitive of a verb not inlcuded in our dataset, one obtains all required information in order to produce a correct conjuga­tion. We will now tackle the problem of fitting a model that is able to predict this labelling.</p><p>The context sensitive nature of the alternations leads to the idea of n-gram representations. A text feature extractor can be tuned to convert a list of verbs into a data matrix. The features of this data matrix are the substrings of length up to n that oc­cur in the data, and the values can be taken either as occurence counts or simply as binary indica­tors of occurence. While occurence counts are useful in, for example, information retrieval, we have found that for such character-level applica­tions, frequencies are less relevant than occurence, and it is not useful to give larger weight to n-grams that appear more often.</p></subsection><subsection number="4.2" title="Approach"><p>In order to get from a list of strings to a data format suitable for machine learning algorithms, we put together a feature extractor that returns a sparse matrix.</p><p>The feature extractor takes two parameters: the maximum n-gram size and whether to binarize the features. It is based on a character n-gram ana­lyzer that takes a Unicode string as input and out­puts a list of n-grams that constitute it. For the input "canta", and for n = 3 it would produce on all of them, then establishes the "vocabulary" of features as the set of distinct n-grams outputted by the analyzer. Afterwards, the vectorizer trans­forms every string in the dataset into a vector of the same size as the vocabulary. If we want to count features, the i-th element of the vector vector will contain the number of times the i-th n-gram ap­pears in the word. If we want binary features, the vector will contain ones and zeros, indicating whether the n-gram appears or not in the word.</p><doubt alpha="65.7" length="213" tooSmall="False" monospace="0.0">the list "c", "aî", "n", "t", "a", "caî", "aîn", "nt", "ta", "caîn", "aînt", "nta". The second component of the feature extractor is the vectorizer. This takes a list of unicode strings as input, runs the analyzer</doubt><p>The average word length in the set of infinitives ending in -ta is 7.48. The larger we choose n, the more features the model will have, and therefore the more complex it will be. Considering both of these aspects, we decided on using the value n = 3.</p><p>The model is built as a pipeline. The list of verbs first passes through the feature extractor and is then fed into the classifier. For classification we evaluated Naive Bayes and linear support vector machines. When using counted features, we used multinomial Naive Bayes, while in the case of bi-narized features we used Bernoulli Naive Bayes. The support vector classifier uses the one-versus-all approach. Due to the limited size of the dataset, all scores are estimated using leave-one-out cross-validation.</p><p>The value of the regularization parameter C for the SVM is decided by a grid search. This consists in defining grid points for fixed parameter values, then fitting and evaluating a model for each grid point using cross-validation.</p><p>The system was put together using the scikits.learn machine learning library for the Python program­ming language (scikits.learn). It provides text fea­ture extraction tools as described above, a linear support vector machine implementation based on the efficient liblinear library, and an automatic grid search framework for tuning the parameters.</p></subsection><subsection number="4.3" title="Results"><p>We first looked at how the Naive Bayes score varies as a function of n, the maximum n-gram length. The results can be seen in figure 1. Con­sidering the fact that the number of features grows exponentially with n, the value of n = 3 seems to offer an acceptable model compexity trade-off versus classification score.</p><page local="5" global="543"/><p><u>Effect of n-gram length</u>_ <u>Optimizing model parameters by grid search</u></p><doubt alpha="0.0" length="30" tooSmall="True" monospace="0.0">12345678910" 10'210"110° 10110</doubt><p>Maximum n-gram length <b>C</b></p><p>Figure 1: Evolution of Naive Bayes scores as we Figure 2: Determining the optimal value for the vary n. model parameters</p><p>The grid search for optimizing the support vector machine's C, detailed in figure 2 parameter shows that binary features perform better, and the value of C that maximizes the success rate within the grid is found at 10-1, with an accurate classifi­cation rate of 82.47%. Precision, recall and <i>F1</i>scores for this optimal classifier are presented in table 1. It can be seen that even the poorly rep­resented classes are accounted for. The last class (with label 6), which contains verbs that conjugate without alternations, is the most clearly separated.</p><p>The results show that indeed, n-gram based fea­tures for classification can give good results for such morphological tasks that are difficult to solve using simple decision rules.</p></subsection></section><section number="5" title="Conclusions and Future Works"><p>Our results show that the labelling system based on the verb conjugation model we developed for verbs ending in -ta can be learned with reasonable accuracy. We are currently working on a more ex­tensive system for labelling the infinitives, based on a near exhaustive conjugational model for the Romanian verb.</p><p>Our future work will revolve around a more ex­haustive classification of the verbs such that, for each class, there is a simple and deterministic way to produce the correct present tense forms, given the infinitive. Following recent works in Romanian linguistics and the study of Romanian as a foreign language such as (Papastergiou et al., 2007), wherein a newer, more comprehensive and in-depth infinitive-based classification of the Ro­manian verb is given, we aim to extend these re­sults to all verbs, not just the ones ending in -ta, and obtain a usable present tense indicative conju-gator for the Romanian language.</p></section><section number="6" title="Acknowledgements"><p>We would like to thank the anonymous reviewers for their insight and constructive criticism which have helped us greatly in polishing this article. We would also like to thank Ana-Maria Barbu for the very useful corpus without which this article wouldn't have existed.</p><table caption="Table 1: Scores estimated by cross validation for the support vector classifier." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>class</p></td><td class="cell"><p>precision</p></td><td class="cell"><p>recall</p></td><td class="cell"><p>Fi</p></td><td class="cell"><p>support</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>0</p></td><td class="cell"><p>0.65</p></td><td class="cell"><p>0.38</p></td><td class="cell"><p>0.48</p></td><td class="cell"><p>106</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p>0.38</p></td><td class="cell"><p>0.23</p></td><td class="cell"><p>0.29</p></td><td class="cell"><p>13</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2</p></td><td class="cell"><p>1.00</p></td><td class="cell"><p>0.60</p></td><td class="cell"><p>0.75</p></td><td class="cell"><p>5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3</p></td><td class="cell"><p>1.00</p></td><td class="cell"><p>0.25</p></td><td class="cell"><p>0.40</p></td><td class="cell"><p>4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>4</p></td><td class="cell"><p>1.00</p></td><td class="cell"><p>0.80</p></td><td class="cell"><p>0.89</p></td><td class="cell"><p>5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>5</p></td><td class="cell"><p>1.00</p></td><td class="cell"><p>0.25</p></td><td class="cell"><p>0.40</p></td><td class="cell"><p>4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>6</p></td><td class="cell"><p>0.85</p></td><td class="cell"><p>0.95</p></td><td class="cell"><p>0.90</p></td><td class="cell"><p>479</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>avg/total</p></td><td class="cell"><p>0.81</p></td><td class="cell"><p>0.82</p></td><td class="cell"><p>0.80</p></td><td class="cell"><p>616</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><references><p>Ana-Maria Barbu. <i>Conjugarea verbelor româ-ne§ti. Dictionar: 7500 de verbe române§ti gru-pate pe clase de conjugare. </i>Bucharest: Coresi, 2007. 4th edition, revised. (In Romanian.) (263 pp.).</p><page local="6" global="544"/><p>Angelo Roth Costanzo. <i>Romance Conjugational Classes: Learning from the Peripheries. </i>PhD thesis, Ohio State University, 2011.</p><p>Liviu Dinu and Emil Ionescu. A context sensi­tive approach to the problem of phonetic alter­nations. submitted, 2011.</p><p>Jiri Felix. <i>Classification des verbes roumains, </i>vol­ume VII. Philosophica Pragensia, 1964.</p><p>Alf Lombard. <i>Le verbe roumain. Etude mor­phologique, </i>volume 1. Lund, C. W. K. Gleerup, 1955.</p><p>Grigore C. Moisil. Probleme puse de traducerea automata. conjugarea verbelor în limba romana. <i>Studii si cercetari lingvistice, </i>XI(1):7-29,1960.</p><p>I. Papastergiou, N. Papastergiou, and L. Man-deki. Verbul romanesc - reguli pentru înlesnirea însusirii indicativului prezent. In <i>Romanian National Symposium "Directions in Romanian Philological Research", 7th Edition, </i>May 2007.</p><p>scikits.learn. scikits.learn, Apr 2011. URL http: //scikit-learn.sourceforge.net.</p></references></body></article>