<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="638"/><title>A Bayesian Mixture Model for PoS Induction Using Multiple Features</title><pubinfo>Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing,pages 638-647, Edinburgh, Scotland, UK, July 27-31, 2011. ©2011 Association for Computational Linguistics</pubinfo><author surname="Christodoulopoulos" givenname="Christos"><org  name="University of Edinburgh" country="United Kingdom" city="Edinburgh"/></author><author surname="Goldwater" givenname="Sharon"><org  name="University of Edinburgh" country="United Kingdom" city="Edinburgh"/></author><author surname="Steedman" givenname="Mark"><org  name="University of Edinburgh" country="United Kingdom" city="Edinburgh"/></author></firstpageheader><frontmatter><p><b>A Bayesian Mixture Model for Part-of-Speech Induction Using Multiple Features</b></p><p><b>Christos Christodoulopoulos Sharon Goldwater               Mark Steedman</b></p><p>School of Informatics School of Informatics            School of Informatics</p><p>University of Edinburgh University of Edinburgh         University of Edinburgh</p><p>christos.c@ed.ac.uk sgwater@inf.ed.ac.uk steedman@inf.ed.ac.uk</p></frontmatter><abstract>In this paper we present a fully unsupervised syntactic class induction system formulated as a Bayesian multinomial mixture model, where each word type is constrained to belong to a single class. By using a mixture model rather than a sequence model (e.g., HMM), we are able to easily add multiple kinds of features, including those at both the type level (mor­phology features) and token level (context and alignment features, the latter from parallel cor­pora). Using only context features, our sys­tem yields results comparable to state-of-the art, far better than a similar model without the one-class-per-type constraint. Using the addi­tional features provides added benefit, and our final system outperforms the best published results on most of the 25 corpora tested. </abstract></header><body><section number="1" title="Introduction"><p>Research on unsupervised learning for NLP has be­come widespread recently, with part-of-speech in­duction, or syntactic class induction, being a partic­ularly popular task.<footnote anchor="1"/> However, despite a recent pro­liferation of syntactic class induction systems (Bie-mann, 2006; Goldwater and Griffiths, 2007; John­son, 2007; Ravi and Knight, 2009; Berg-Kirkpatrick et al., 2010; Lee et al., 2010), careful compari­son indicates that very few systems perform better than some much simpler and quicker methods dating back ten or even twenty years (Christodoulopoulos</p><p>!The task is more commonly referred to as part-of-speech induction, but we prefer the term syntactic class induction since the induced classes may not coincide with part-of-speech tags.</p><p>et al., 2010). This fact suggests that we should con­sider which features of the older systems led to their success, and attempt to combine these features with some of the machine learning methods introduced by the more recent systems. We pursue this strat­egy here, developing a system based on Bayesian methods where the probabilistic model incorporates several insights from previous work.</p><p>Perhaps the most important property of our model is that it is <i>type-based, </i>meaning that all tokens of a given word type are assigned to the same clus­ter. This property is not strictly true of linguistic data, but is a good approximation: as Lee et al. (2010) note, assigning each word type to its most frequent part of speech yields an upper bound ac­curacy of 93% or more for most languages. Since this is much better than the performance of cur­rent unsupervised syntactic class induction systems, constraining the model in this way seems likely to improve performance by reducing the number of parameters in the model and incorporating useful linguistic knowledge. Both of the older systems discussed by Christodoulopoulos et al. (2010), i.e., Clark (2003) and Brown et al. (1992), included this constraint and achieved very good performance rel­ative to token-based systems. More recently, Lee et al. (2010) presented a new type-based model, and also reported very good results.</p><p>A second property of our model, which distin­guishes it from the type-based Bayesian model of Lee et al. (2010), is that the underlying probabilistic model is a <i>clustering model, </i>(specifically, a multino­mial mixture model) rather than a sequence model (HMM). In this sense, our model is more closely related to several non-probabilistic systems that clus­ter context vectors or lower-dimensional represen­tations of them (Redington et al., 1998; Schütze, 1995; Lamar et al., 2010).<page local="2" global="639"/> Sequence models are by far the most common method of süpervised part-of-speech tagging, and have also been widely used for unsupervised part-of-speech tagging both with and without a dictionary (Smith and Eisner, 2005; Haghighi and Klein, 2006; Goldwater and Griffiths, 2007; Johnson, 2007; Ravi and Knight, 2009; Lee et al., 2010). However, systems based on context vec­tors have also performed well in these latter scenar­ios (Schütze, 1995; Lamar et al., 2010; Toutanova and Johnson, 2007) and present a viable alternative to sequence models.</p><p>One advantage of using a clustering model rather than a sequence model is that the features used for clustering need not be restricted to context words. Additional types of features can easily be incorpo­rated into the model and inference procedure using the same general framework as in the basic model that uses only context word features. In particu­lar, we present two extensions to the basic model. The first uses <i>morphological features, </i>which serve as cues to syntactic class and seemed to partly ex­plain the success of two best-performing systems analysed by Christodoulopoulos et al. (2010). The second extension to our model uses <i>alignment fea­tures </i>gathered from parallel corpora. Previous work suggests that using parallel text can improve perfor­mance on various unsupervised NLP tasks (Naseem et al., 2009; Snyder and Barzilay, 2008).</p><p>We evaluate our model on 25 corpora in 20 lan­guages that vary substantially in both syntax and morphology. As in previous work (Lee et al., 2010), we find that the one-class-per-type restriction boosts performance considerably over a comparable token-based model and yields results that are comparable to state-of-the-art even without the use of morphol­ogy or alignment features. Including morphology features yields the best published results on 14 or 15 of our 25 corpora (depending on the measure) and alignment features can improve results further.</p></section><section number="2" title="Models"><p>Our model is a multinomial mixture model with Bayesian priors over the mixing weights 9 and</p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">M</doubt><p>Figure 1: Plate diagram of the basic model with a single feature per token (the observed variable f ). M, Z, and <b><i>nj </i></b>are the number of word types, syntactic classes z, and features (= tokens) per word type, respectively.</p><p>multinomial class output parameters 0. The model is defined so that all observations associated with a single word type are generated from the same mixing component (syntactic class). In the basic model, these observations are token-level features; the morphology model adds type-level features as well. We begin by describing the simplest version of our model, where each word token is associated with a single feature, for example its left context word (the word that occurs to its left in the corpus). We then show how to generalise the model to multiple token-level features and to type-level features.</p><subsection number="2.1" title="Basic model"><p>In the basic model, each word token is represented by a single feature such as its left context word. These features are the observed data; the model ex­plains the data by assuming that it has been gener­ated from some set of latent syntactic classes. The ith class is associated with a multinomial parameter vector 0<b>j </b>that defines the distribution over features generated from that class, and with a mixing weight 9<b>j </b>that defines the prior probability of that class. 9 and 0<b>i </b>are drawn from symmetric Dirichlet distribu­tions with parameters <i>a </i>and ß respectively.</p><p>The generative story goes as follows: First, gen­erate the prior class probabilities <i>9. </i>Next, for each word type j = 1.<page local="3" global="640"/>.. M, choose a class assignment <i>Zj</i><i> </i>from the distribution 9. For each class i = 1... Z, choose an output distribution over features 0j. Fi­nally, for each token <i>k</i><i> </i><i>=</i><i> </i><i>1.</i><i> </i><i>..nj</i><i> </i>of word type j, generate a feature f jk from <i>(f)Zj</i>, the distribution as­sociated with the class that word type <i>j </i>is assigned to. The model is illustrated graphically in Figure 1 and is defined formally as follows:</p><doubt alpha="20.0" length="5" tooSmall="False" monospace="0.0">9|a —</doubt><doubt alpha="40.0" length="5" tooSmall="False" monospace="0.0">Zj|9~</doubt><doubt alpha="60.0" length="5" tooSmall="False" monospace="0.0">fi|ß-</doubt><p><b>fjfc</b><b> </b><footnote anchor="1"/> <b>fZj </b>-</p><p>Dirichlet(a) Multinomial (9) Dirichlet(ß ) Multinomial(f <b>Zj)</b></p><p>In addition to the variables defined above, we will use F to refer to the number of different possible values a feature can take on (so that ff is a Z x F matrix). Thus, one way to think of the model is as a vector-based clustering system, where word type j is associated with a 1 x F vector of feature counts rep­resenting the features of all n<b>j </b>tokens of j, and these vectors are clustered into similar classes. The differ­ence from other vector-based syntactic class induc­tion systems is in the method of clustering. Here, we define a Gibbs sampler that samples from the posterior distribution of the clusters given the ob­served features; other systems have used various standard distance-based vector clustering methods. Some systems also include dimensionality reduction (Schütze, 1995; Lamar et al., 2010) to reduce the size of the context vectors; we simply use the F most common words as context features.</p></subsection><subsection number="2.2" title="Inference"><p>At inference time we want to sample a syntactic class assignment <b>z </b>from the posterior of the model. We use a collapsed Gibbs sampler, integrating out the parameters 9 and f and sampling from the fol­lowing distribution:</p><p>P<b>(z|f</b>,a,ß) ce P<b>(z</b>|a)P<b>(f|z</b>,ß).</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(1)</doubt><p>Rather than sampling the joint class assignment P<b>(z|f</b>, <i>a,</i><i> </i><i>ß)</i><i> </i>directly, the sampler iterates over each word type j, resampling its class assignment Zj given the current assignments <b>z</b>_j of all other word types. The posterior over Zj can be computed as where <b>f</b><b>j </b>are the features associated with word type <i>j </i>(one feature for each token of j). The first (prior) factor is easy to compute due to the conjugacy be­tween the Dirichlet and multinomial distributions, and is equal to where nz is the number of types in class z and n is the total number of word types in all classes. All counts in this and the following equations are com­puted with respect to <b>z</b><b>_j </b>(e.g., n = M — 1).</p><doubt alpha="53.3" length="15" tooSmall="False" monospace="0.0">P(zj|z_j,f,a,ß)</doubt><doubt alpha="53.6" length="28" tooSmall="False" monospace="0.0">P(zj|z_j,a,ß)P(fj|f_j,z,a,ß)</doubt><doubt alpha="53.8" length="13" tooSmall="False" monospace="0.0">P(zj=z|z_j,a)</doubt><doubt alpha="60.0" length="5" tooSmall="False" monospace="0.0">nz+ a</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(3)</doubt><doubt alpha="42.9" length="7" tooSmall="False" monospace="0.0">n. + Za</doubt><p>Computing the second (likelihood) factor is slightly more complex due to the dependencies be­tween the different variables in <b>f</b><b>j </b>that are induced by integrating out the ff parameters. Consider first a simple case where word type j occurs exactly twice in the corpus, so <b>f</b><b>j </b>contains two features. The prob­ability of the first feature f <b>i</b><b> </b>is equal to where n <b>f z </b>is the number of times feature f has been seen in class z, n^z is the total number of feature tokens in the class, and F is the number of different possible features.</p><doubt alpha="66.7" length="6" tooSmall="False" monospace="0.0">nf,z+ß</doubt><doubt alpha="56.5" length="23" tooSmall="False" monospace="0.0">P(fji=f|zj=z,z_j,f_j,ß)</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(4)</doubt><doubt alpha="50.0" length="8" tooSmall="False" monospace="0.0">n.,z+ Fß</doubt><p>The probability of the second feature f<b>j</b><b>2</b><b> </b>can be calculated similarly, except that it is conditioned on f<b>j</b><b>i</b><b> </b>in addition to the other variables, so the counts for previously observed features must include the counts due to fj1 as well as those due to <b>f</b><b>_j</b>. Thus, the probability is where ö is the Kronecker delta function, equal to 1 if its arguments are equal and 0 otherwise.</p><doubt alpha="55.6" length="27" tooSmall="False" monospace="0.0">P(fj2=f|fji,zj=z,z_j,f_j,ß)</doubt><doubt alpha="55.6" length="18" tooSmall="False" monospace="0.0">_nf,z+S(fji,fj2)+ß</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(5)</doubt><doubt alpha="33.3" length="12" tooSmall="False" monospace="0.0">n.,z+ 1 + Fß</doubt><p>Extending this example to the general case, the probability of a sequence of features <b>f</b><b>j </b>is computed using the chain rule, where the counts used in each factor are incremented as necessary for each addi­tional conditioning feature, yielding the following expression:</p><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">oc</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(2)</doubt><doubt alpha="59.4" length="32" tooSmall="False" monospace="0.0">nLinrTWz+i+ß )rr=o_i(n.;z+i+Fß )</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(6)</doubt><page local="4" global="641"/><p>where njk is the number of instances of feature k in word type j.<footnote anchor="2"/></p></subsection><subsection number="2.3" title="Extended models"><p>We can extend the model above in two different ways: by adding more features at the word token level, or by adding features at the type level. To add more token-level features, we simply assume that each word token generates multiple features, one feature from each of several different kinds.<footnote anchor="3"/> For example, the left context word might be one kind of feature and the right context word another. We as­sume conditional independence between the gener­ated features given the syntactic class, so each kind of feature t has its own output parameters ff<b>(t)</b>. A plate diagram of the model with T kinds of features is shown in Figure 2 (a type-level feature is also in­cluded in this diagram, as described below).</p><p>Due to the independence assumption between the different kinds of features, the basic Gibbs sampler is easy to extend to this case by simpling multiplying in extra factors for the additional kinds of features, with the prior (Equation 3) unchanged. The likeli­hood becomes:</p><doubt alpha="43.3" length="30" tooSmall="False" monospace="0.0">P(if,...,ff)|f_i-TU-= z,z_j,ß)</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">T</doubt><doubt alpha="40.0" length="30" tooSmall="False" monospace="0.0">=11P(fj(t)|f_j&gt;zj= z,z_j,ß)(7)</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">t=i</doubt><p>where each factor in the product is computed using Equation 6.</p><p>In addition to monolingual context features, we also explore the use of alignment features for those languages where we have parallel corpora. These features are extracted for language <i>I </i>by word-aligning <i>I </i>to another language <i>£</i>' (details of the alignment procedure are described in Section 3.1). The features used for each token e in <i>I </i>are the left and right context words of the word token that is aligned to e (if there is one). As with the mono­lingual context features, we use only the F most fre­quent words in <i>£</i>' as possible features.</p><footnote label="2">One could approximate this likelihood term by assuming independence between all nj feature tokens of word type j. This is the approach taken by Lee et al. (2010).</footnote><footnote label="3">We use the word kind here to avoid confusion with type, which we reserve for the type-token distinction, which can ap­ply to features as well as words.</footnote><p>Note that this model with multiple context fea­tures is deficient: it can generate data that are in­consistent with any actual corpus, because there is no mechanism to constrain the left context word of token e<b>i </b>to be the same as the right context word of token ei_i (and similarly with alignment features). However, deficient models have proven useful in other unsupervised NLP tasks (Klein and Manning, 2002; Toutanova and Johnson, 2007). In particular, Toutanova and Johnson (2007) demon­strate good performance on unsupervised part-of-speech tagging (using a dictionary) with a Bayesian model similar to our own. If we remove the part of their model that relies on the dictionary (the mor­phological ambiguity classes), their model is equiv­alent to our own, without the restriction of one class per type. We use this token-based version of our model as a baseline in our experiments.</p><p>The final extension to our model introduces type-level features, specifically morphology features. The model is illustrated in Figure 2. We assume conditional independence between the morphology features and other features, so again we can simply multiply another factor into the likelihood during in­ference. There is only one morphological feature per type, so this factor has the form of Equation 4. Since frequent words will have many token-level features contributing to the likelihood and only one morphol­ogy feature, the morphology features will have a greater effect for infrequent words (as appropriate, since there is less evidence from context and align­ments). As with the other kinds of features, we use only a limited number Fm of morphology features, as described below.</p></subsection></section><section number="3" title="Experiments"><subsection number="3.1" title="Experimental setup"><p>We evaluate our models using an increasing level of complexity, starting with a model that uses only monolingual context features. We use the F = 100 most frequent words as features, and consider two versions of this model: one with two kinds of fea­tures (one left and one right context word) and one with four (two context words on each side).</p><p>For the model with morphology features we ran the unsupervised morphological segmentation sys­tem Morfessor (Creutz and Lagus, 2005) to get a<page local="5" global="642"/></p><p>Figure 2: Plate diagram of the extended model with T kinds of token-level features (f(t) variables) and a single kind of type-level feature (morphology, m).</p><p>segmentation for each word type in the corpus. We then extracted the suffix of each word type<footnote anchor="4"/> and used it as a feature type. This process yielded on average Fm = 110 morphological feature types<footnote anchor="5"/>. Each word type generates at most one of these possible features. If there are overlapping possibilities (e.g. -ingly and -y) we take the longest possible match.</p><p>We also explore the idea of extending the mor­phology feature space beyond suffixes, by including features like capitalisation and punctuation. Specif­ically we use the features described in Haghighi and Klein (2006), namely <i>initial-capital, contains-hyphen, contains-digit </i>and we add an extra feature <i>contains-punctuation.</i></p><p>For the model with alignment features, we fol­low (Naseem et al., 2009) in using only bidirectional alignments: using Giza++ (Och and Ney, 2003), we get the word alignments in both directions be­tween all possible language pairs in our parallel cor­pora (i.e., alternating the source and target languages within each pair). We then use only those align­ments that are found in both directions. As discussed above, we use two kinds of alignment features: the left and right context words of the aligned token in the other language. The feature space is set to the F = 100 most frequent words in that language.</p><footnote label="4">Since Morfessor yields multiple affixes for each word we concatenated all the suffixes into a single suffix.</footnote><footnote label="5">There was large variance in the number of feature types for each language ranging from 11 in Chinese to more than 350 in German and Czech.</footnote><p>Instead of fixing the hyperparameters a and ß, we used the Metropolis-Hastings sampler presented by Goldwater and Griffiths (2007) to get updated values based on the likelihood of the data with respect to those hyperparameters<footnote anchor="6"/>. In order to improve conver­gence of the sampler, we used simulated annealing with a sigmoid-shaped cooling schedule from an ini­tial temperature of 2 down to 1. Preliminary experi­ments indicated that we could achieve better results by cooling even further (approximating the MAP so­lution rather than a sample from the posterior), so for all experiments reported here, we ran the sampler for a total of 2000 iterations, with the last 400 of these decreasing the temperature from 1 to 0.66.</p><p>Finally, we investigated two different initialisa­tion techniques: First, we use random class as­signments to word types (referred to as method 1) and second, we assign each of the Z most frequent word types to a separate class and then randomly distribute the rest of the word types to the classes (method 2).</p></subsection><subsection number="3.2" title="Datasets"><p>Although unsupervised systems should in principle be language- and corpus-independent, most part-of-speech induction systems (especially in the early lit­erature) have been developed on English. Whether because English is simply an easier language, or be­cause of bias introduced during development, these systems' performance is considerably worse in other languages (Christodoulopoulos et al., 2010)</p><p>Since we aim to use our system mostly on non-English corpora, and ones that are significantly smaller than the large English treebank corpora, we developed our models using one of the languages of the MULTEXT-East corpus (Erjavec, 2004), namely Bulgarian. The other languages in the corpus were used during development as a source of word align­ments, but otherwise were only used for testing final versions of our models. Since none of the authors speak any of the languages in the MULTEXT collection, we also used the Penn Treebank WSJ cor­pus (Marcus et al., 1993) for development.<page local="6" global="643"/> Fol­lowing Christodoulopoulos et al. (2010) we created a smaller version of the WSJ corpus (referred to as wsj-s) to approximate the size of the corpora in MULTEXT-East. For comparison to other systems, we also used the full WSJ at test time.</p><footnote label="6">For simplicity, we tied the ß parameters for the two or four kinds of context features to the same value, and similarly the ß parameters for the two kinds of alignment features.</footnote><p>For further testing, we used the remaining MUL­TEXT languages, as well as the languages of the CONNL-X (Buchholz and Marsi, 2006) shared task. This dataset contains 13 languages, 4 of which are freely available (Danish, Dutch, Portuguese and Swedish) and 9 that are used with permission from the creators of the corpora ( Arabic<footnote anchor="7"/>, Bul-garian<footnote anchor="8"/>, Czech<footnote anchor="9"/>, German<footnote anchor="10"/>, Chinese<footnote anchor="11"/>, Japanese<footnote anchor="12"/>, Slovene<footnote anchor="13"/>, Spanish<footnote anchor="14"/>, Turkish<footnote anchor="15"/> ). Following Lee et al. (2010) we used only the training sections for each language.</p><p>Finally, to widen the scope of our system, we gen­erated two more corpora in French<footnote anchor="16"/> and Ancient Greek<footnote anchor="17"/>, extracting the gold standard parts of speech from the respective dependency treebanks.</p></subsection><subsection number="3.3" title="Baselines"><p>We chose three baselines for comparison. The first is the basic k-means clustering algorithm, which we applied to the same feature vectors we extracted for our system (context + extended morphology), using a Euclidean distance metric. This provides a very simple vector-based clustering baseline. The second baseline is a more recent vector-based syntactic class induction method, the SVD approach of (Lamar et al., 2010), which extends Schütze (1995)'s original method and, like ours, enforces a one-class-per-tag restriction. As a third baseline we use the system of Clark (2003) since it is a type-level system that models morphology and has produced very good results on multilingual corpora.</p><footnote label="7">Part of the Prague Arabic Treebank (Hajic et al., 2003; Smrz and Pajas, 2004)</footnote><footnote label="8">Part of the BulTreeBank (Simov et al., 2004). 9 Part of the Prague Dep. Treebank (Bohmova et al., 2001) 10 Part of the TIGER Treebank (Brants et al., 2002) "Part of the Sinica Treebank (Keh-Jiann et al., 2003) 12 Part of the Tübingen Treebank of Spoken Japanese (for­merly VERMOBIL Treebank - Kawata and Bartels (2000)). 13 Part of the Slovene Dep. Treebank (Dzeroski et al., 2006) 14 Part of the Cast3LB Treebank (Civit et al., 2006) 15 Part of the METU-Sabanci Treebank (Oflazer et al., 2003). 16 French Treebank (Abeille et al., 2000) 17 Greek Dependency Treebank (Bamman et al., 2009)</footnote></subsection></section><section number="4" title="Results and Analysis"><subsection number="4.1" title="Development results"><p>Tables 1 and 2 present the results from develop­ment runs, which were used to decide which fea­tures to incorporate in the final system. We used V-Measure (Rosenberg and Hirschberg, 2007) as our primary evaluation score, but also present many-to-one matching accuracy (M-1) scores for better com­parison with previously published results. We chose V-Measure (VM) as our evaluation score because it is less sensitive to the number of classes induced by the model (Christodoulopoulos et al., 2010), allow­ing us to develop our models without using the num­ber of classes as a parameter. We fixed the number of classes in all systems to 45 during development; note however that the gold standard tag set for Bul­garian contains only 12 tags, so the results in Ta­ble 1 (especially the M-1 scores) are not comparable to previous results. For results using the number of gold-standard tags refer to Table 4.</p><p>The first conclusion that can be drawn from these results is the large difference between the token-and type-based versions of our system, which con­firms that the one-class-per-type restriction is help­ful for unsupervised syntactic class induction. We also see that for both languages, the performance of the model using 4 context words (±2 on each side) is worse than the 2 context words model. We therefore used only two context words for all of our additional test languages (below).</p><p>We can clearly see that morphological features are helpful in both languages; however the extended features of Haghighi and Klein (2006) seem to help only on the English data. This could be due to the fact that Bulgarian has a much richer morphology and thus the extra features contribute little to the overall performance of the model.</p><p>The contribution of the alignment features on the Bulgarian corpus (aligned with English) is less sig­nificant than that of morphology but when com­bined, the two sets of features yield the best per­formance. This provides evidence in favor of using multiple features.</p><p>Finally, initialisation method 2 does not yield<page local="7" global="644"/></p><p>Table 1: V-measure (VM) and many-to-one (M-1) results on the MULTEXT-Bulgarian corpus for various mod­els using either ±1 or ±2 context words as features. base: context features only; (tokens): token-based model; (init): Initialisation method 2—other results use method 1; (ext): Extended morphological features.</p><p>Table 2: V-measure and many-to-one results on the wsj-s corpus for various models, as described in Table 1.</p><p>consistent improvements over the standard ran­dom initialisation—if anything, it seems to perform worse. We therefore use only method 1 in the re­maining experiments.</p></subsection><subsection number="4.2" title="Overall results"><p>Table 3 presents the results on our parallel corpora. We tested all possible combinations of two lan­guages to align, and present both the average score over all alignments, and the score under the best choice of aligned language.<footnote anchor="18"/> Also shown are the results of adding morphology features to the basic model (context features only) and to the best align­ment model for each language. In accord with our development results, adding morphology to the ba­sic model is generally useful. The alignment results are mixed: on the one hand, choosing the best pos­sible language to align yields improvements, which can be improved further by adding morphological features, resulting in the best scores of all models for most languages. On the other hand, without knowing which language to choose, alignment fea­tures do not help on average. We note, however, that three out of the seven languages have English as their best-aligned pair (perhaps due to its better overall scores), which suggests that in the absence of other knowledge, aligning with English may be a good choice.</p><footnote label="18">The choice of language was based on the same test data, so the 'best-language' results should be viewed as oracle scores.</footnote><p>The low average performance of the alignment features is disappointing, but there are many pos­sible variations on our method for extracting these features that we have not yet tested. For example, we used only bidirectional alignments in an effort to improve alignment precision, but these alignments typically cover less than 40% of tokens. It is pos­sible that a higher-recall set of alignments could be more useful.</p><p>We turn now to our results on all 25 corpora, shown in Table 4 along with corpus statistics, base­line results, and the best published results for each language (when available). Our system, includ­ing morphology features in all cases, is listed as BMMM (Bayesian Multinomial Mixture Model). We do not include alignment features for the MUL­TEXT languages since these features only yielded improvements for the oracle case where we know which aligned language to choose. Nevertheless, our MULTEXT scores mostly outperform all other sys­tems. Overall, we acheive the highest published re­sults on 14 (VM) or 15 (M-1) of the 25 corpora.</p><p>One surprising discovery is the high performance of the k-means clustering system. Despite its sim­plicity, it is competitive with the other systems and in a few cases even achieves the best published re­sults.</p></subsection></section><section number="5" title="Conclusion"><p>We have presented a Bayesian model for syntactic class induction that has two important properties. First, it is type-based, assigning the same class to every token of a word type.  We have shown by<page local="8" global="645"/></p><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>system</b></p></td><td class="cell"><p><b>±1 words</b></p></td><td class="cell"><p><b>±2 words</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>VM/M-1</p></td><td class="cell"><p>VM/M-1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>base</p></td><td class="cell"><p>58.1</p></td><td class="cell"><p>/70.8</p></td><td class="cell"><p>55.4/67.6</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>base(tokens)</p></td><td class="cell"><p>48.3</p></td><td class="cell"><p>/62.5</p></td><td class="cell"><p>37.0/54.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>base(init)</p></td><td class="cell"><p>57.6</p></td><td class="cell"><p>/70.1</p></td><td class="cell"><p>56.1 / 68.6</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>+morph</p></td><td class="cell"><p>58.3</p></td><td class="cell"><p>/ 74.9</p></td><td class="cell"><p>57.4/71.9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>+morph(ext)</p></td><td class="cell"><p>57.8</p></td><td class="cell"><p>/73.7</p></td><td class="cell"><p>57.8 / 70.1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(init)+morph</p></td><td class="cell"><p>57.8</p></td><td class="cell"><p>/ 74.3</p></td><td class="cell"><p>57.3 / 69.5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(init)+morph(ext)</p></td><td class="cell"><p>58.1</p></td><td class="cell"><p>/ 74.3</p></td><td class="cell"><p>57.2/71.3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>+aligns(EN)</p></td><td class="cell"><p>58.1</p></td><td class="cell"><p>/72.6</p></td><td class="cell"><p>56.7 / 71.1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>+aligns(EN)+morph</p></td><td class="cell"><p><b>59.0</b></p></td><td class="cell"><p>/ <b>75.4</b></p></td><td class="cell"><p>57.5 / 69.7</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>system</b></p></td><td class="cell"><p><b>±1 words</b></p><p>VM/M-1</p></td><td class="cell"><p><b>±2 words</b></p><p>VM/M-1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>base</p></td><td class="cell"><p>63.3</p></td><td class="cell"><p>/ 64.3</p></td><td class="cell"><p>62.4</p></td><td class="cell"><p>/63.3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>base(tokens)</p></td><td class="cell"><p>48.6</p></td><td class="cell"><p>/57.8</p></td><td class="cell"><p>49.3</p></td><td class="cell"><p>/38.3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>base(init)</p></td><td class="cell"><p>62.7</p></td><td class="cell"><p>/ 62.9</p></td><td class="cell"><p>62.2</p></td><td class="cell"><p>/62.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>+morph</p></td><td class="cell"><p>66.4</p></td><td class="cell"><p>/66.7</p></td><td class="cell"><p>65.1</p></td><td class="cell"><p>/67.2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>+morph(ext)</p></td><td class="cell"><p><b>67.7</b></p></td><td class="cell"><p>/ <b>72.0</b></p></td><td class="cell"><p>65.6</p></td><td class="cell"><p>/67.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(init)+morph</p></td><td class="cell"><p>64.8</p></td><td class="cell"><p>/66.9</p></td><td class="cell"><p>64.2</p></td><td class="cell"><p>/66.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(init)+morph(ext)</p></td><td class="cell"><p>67.4</p></td><td class="cell"><p>/71.3</p></td><td class="cell"><p>65.7</p></td><td class="cell"><p>/67.1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Table 3: V-measure (VM) and many-to-one (M-1) results on the languages in the MULTEXT-East corpus using the gold standard number of classes shown in Table 4. BASE results use ±1-word context features alone or with morphology. ALIGNMENTS adds alignment features, reporting the average score across all possible choices of paired language and the scores under the best performing paired language (in parens), alone or with morphology features.</p><p>Table 4: Final results on 25 corpora in 20 languages, with the number of induced classes equal to the number of gold standard tags in all cases. <b>k-means </b>and <b>SVD2 </b>models could not produce a clustering in the Czech CoNLL corpus due its size. Best published results are from *Christodoulopoulos et al. (2010), *Berg-Kirkpatrick et al. (2010) and *Lee et al. (2010). The latter two papers do not report VM scores. No best published results are shown for the MULTEXT languages; Christodoulopoulos et al. (2010) report results based on 45 tags suggesting that <b>clark </b>performs best on these corpora.</p><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>BASE</p></td><td class="cell"><p>ALIGNMENTS</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Lang.</b></p></td><td class="cell"><p><b>base</b></p></td><td class="cell"><p><b>+morph</b></p></td><td class="cell"><p><b>Avg.</b></p></td><td class="cell"><p><b>Best</b></p></td><td class="cell"><p><b>+morph</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>VM/M-1</p></td><td class="cell"><p>VM/M-1</p></td><td class="cell"><p>VM/M-1</p></td><td class="cell"><p>VM M1</p></td><td class="cell"><p>VM/M1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Bulgarian</p></td><td class="cell"><p>54.4/61.5</p></td><td class="cell"><p>54.5 / 64.3</p></td><td class="cell"><p>53.1 /60.5</p></td><td class="cell"><p>55.2 / 64.5(EN)</p></td><td class="cell"><p><b>55.7 </b>/ <b>66.0</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Czech</p></td><td class="cell"><p>54.2/58.9</p></td><td class="cell"><p>53.9 / 64.2</p></td><td class="cell"><p>52.6/58.4</p></td><td class="cell"><p>53.8 / 59.7(EN)</p></td><td class="cell"><p><b>55.4 </b>/ <b>66.4</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>English</p></td><td class="cell"><p>62.9 / 72.4</p></td><td class="cell"><p>63.3 / 73.3</p></td><td class="cell"><p>62.5/72.0</p></td><td class="cell"><p>63.2/71.9(HU)</p></td><td class="cell"><p><b>63.5 </b>/ <b>73.7</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Estonian</p></td><td class="cell"><p>52.8/63.5</p></td><td class="cell"><p>53.3/ <b>67.4</b></p></td><td class="cell"><p>52.8/63.9</p></td><td class="cell"><p>53.5 / 65.0(EN)</p></td><td class="cell"><p><b>54.3 </b>/ 66.9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Hungarian</p></td><td class="cell"><p>53.3/60.4</p></td><td class="cell"><p>54.8 / <b>68.2</b></p></td><td class="cell"><p>53.3/60.8</p></td><td class="cell"><p>53.9 / 61.1(RO)</p></td><td class="cell"><p><b>55.9 </b>/ 67.1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Romanian</p></td><td class="cell"><p>53.9/62.4</p></td><td class="cell"><p>52.3 / 61.1</p></td><td class="cell"><p>56.2/63.7</p></td><td class="cell"><p><b>57.5 </b>/ <b>64.6</b>(ES)</p></td><td class="cell"><p>54.5 / 63.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Slovene</p></td><td class="cell"><p><b>57.2 </b>/ 65.9</p></td><td class="cell"><p>56.7 / 67.9</p></td><td class="cell"><p>54.7/64.1</p></td><td class="cell"><p>55.9 / 64.4(HU)</p></td><td class="cell"><p>56.7 / <b>67.9</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Serbian</p></td><td class="cell"><p><b>49.1 </b>/56.6</p></td><td class="cell"><p>49.0 / <b>62.0</b></p></td><td class="cell"><p>47.3/55.6</p></td><td class="cell"><p>48.9 / 59.4(CZ)</p></td><td class="cell"><p>48.3 / 60.8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>Language</b></p></td><td class="cell"><p><b>Types</b></p></td><td class="cell"><p><b>Tags</b></p></td><td class="cell"><p><b>k-means</b></p></td><td class="cell"><p><b>SVD2</b></p></td><td class="cell"><p><b>clark</b></p></td><td class="cell"><p><b>Best Pub.</b></p></td><td class="cell"><p><b>BMMM</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>WSJ</p></td><td class="cell"><p>wsj</p></td><td class="cell"><p>49,190</p></td><td class="cell"><p>45</p></td><td class="cell"><p>59.5/61.6</p></td><td class="cell"><p>58.2/64.0</p></td><td class="cell"><p>65.6/71.2</p></td><td class="cell"><p>68.8/76.1*</p></td><td class="cell"><p>66.1 /72.8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>wsj-s</p></td><td class="cell"><p>16,850</p></td><td class="cell"><p>45</p></td><td class="cell"><p>56.7/60.1</p></td><td class="cell"><p>54.3 / 60.7</p></td><td class="cell"><p>63.8/68.8</p></td><td class="cell"><p>62.3/70.7*</p></td><td class="cell"><p><b>67.7 </b>/ <b>72.0</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Bulgarian</p></td><td class="cell"><p>16,352</p></td><td class="cell"><p>12</p></td><td class="cell"><p>50.3/59.3</p></td><td class="cell"><p>41.7/51.0</p></td><td class="cell"><p><b>55.6 </b>/ <b>66.5</b></p></td><td class="cell"><p>-</p></td><td class="cell"><p>54.5 / 64.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MULTEXT-East</p></td><td class="cell"><p>Czech</p></td><td class="cell"><p>19,115</p></td><td class="cell"><p>12</p></td><td class="cell"><p>48.6/56.7</p></td><td class="cell"><p>35.5 / 50.9</p></td><td class="cell"><p>52.6/64.1</p></td><td class="cell"><p>-</p></td><td class="cell"><p><b>53.9 </b>/ <b>64.2</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>English</p></td><td class="cell"><p>9,773</p></td><td class="cell"><p>12</p></td><td class="cell"><p>56.5/65.4</p></td><td class="cell"><p>52.3 / 65.5</p></td><td class="cell"><p>60.5 / 70.6</p></td><td class="cell"><p>-</p></td><td class="cell"><p><b>63.3 </b>/ <b>73.3</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Estonian</p></td><td class="cell"><p>17,845</p></td><td class="cell"><p>11</p></td><td class="cell"><p>45.3/55.6</p></td><td class="cell"><p>38.7 / 55.3</p></td><td class="cell"><p>44.4 / 58.4</p></td><td class="cell"><p>-</p></td><td class="cell"><p><b>53.3 </b>/ <b>64.4</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Hungarian</p></td><td class="cell"><p>20,321</p></td><td class="cell"><p>12</p></td><td class="cell"><p>46.7/53.9</p></td><td class="cell"><p>39.8 / 49.5</p></td><td class="cell"><p>48.9/61.4</p></td><td class="cell"><p>-</p></td><td class="cell"><p><b>54.8 </b>/ <b>68.2</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Romanian</p></td><td class="cell"><p>15,189</p></td><td class="cell"><p>14</p></td><td class="cell"><p>45.2/55.1</p></td><td class="cell"><p>42.1 / 52.6</p></td><td class="cell"><p>40.9 / 49.9</p></td><td class="cell"><p>-</p></td><td class="cell"><p><b>52.3 </b>/ <b>61.1</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Slovene</p></td><td class="cell"><p>17,871</p></td><td class="cell"><p>12</p></td><td class="cell"><p>46.9/56.2</p></td><td class="cell"><p>39.5 / 54.2</p></td><td class="cell"><p>54.9 / 69.4</p></td><td class="cell"><p>-</p></td><td class="cell"><p><b>56.7 </b>/ <b>67.9</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Serbian</p></td><td class="cell"><p>18,095</p></td><td class="cell"><p>12</p></td><td class="cell"><p>41.4/47.0</p></td><td class="cell"><p>39.1 /54.6</p></td><td class="cell"><p><b>51.0 </b>/ <b>64.1</b></p></td><td class="cell"><p>-</p></td><td class="cell"><p>49.0 / 62.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Arabic</p></td><td class="cell"><p>12,915</p></td><td class="cell"><p>20</p></td><td class="cell"><p><b>43.3 </b>/ 60.7</p></td><td class="cell"><p>27.6/49.0</p></td><td class="cell"><p>40.6/59.8</p></td><td class="cell"><p>-</p></td><td class="cell"><p>42.4 / <b>61.5</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Bulgarian</p></td><td class="cell"><p>32,439</p></td><td class="cell"><p>54</p></td><td class="cell"><p>53.6/65.6</p></td><td class="cell"><p>49.0 / 65.3</p></td><td class="cell"><p><b>59.6 </b>/ <b>70.4</b></p></td><td class="cell"><p>-</p></td><td class="cell"><p>58.8 / 68.9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ce</b></p><p>as</p></td><td class="cell"><p>Chinese</p></td><td class="cell"><p>40,562</p></td><td class="cell"><p>15</p></td><td class="cell"><p>32.6/61.1</p></td><td class="cell"><p>24.5 / 54.6</p></td><td class="cell"><p>31.8/56.7</p></td><td class="cell"><p>-</p></td><td class="cell"><p><b>42.6 </b>/ <b>69.4</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Czech</p></td><td class="cell"><p>130,208</p></td><td class="cell"><p>12</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>47.1 /65.5</p></td><td class="cell"><p>-</p></td><td class="cell"><p><b>48.4 </b>/ <b>65.7</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>) Sharedr.</p></td><td class="cell"><p>Danish</p></td><td class="cell"><p>18,356</p></td><td class="cell"><p>25</p></td><td class="cell"><p>51.7/61.6</p></td><td class="cell"><p>40.8 / 57.6</p></td><td class="cell"><p>52.7/65.3</p></td><td class="cell"><p>- / 66.7t</p></td><td class="cell"><p><b>59.0 </b>/ <b>71.1</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Dutch</p></td><td class="cell"><p>28,393</p></td><td class="cell"><p>13</p></td><td class="cell"><p>45.3/60.5</p></td><td class="cell"><p>36.7 / 52.4</p></td><td class="cell"><p>52.2 / 67.9</p></td><td class="cell"><p>-/67.3*</p></td><td class="cell"><p><b>54.7 </b>/ <b>71.1</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>German</p></td><td class="cell"><p>72,326</p></td><td class="cell"><p>54</p></td><td class="cell"><p>58.7/67.5</p></td><td class="cell"><p>54.1 / 64.2</p></td><td class="cell"><p><b>63.0 </b>/ 73.9</p></td><td class="cell"><p>- / 68.4*</p></td><td class="cell"><p>61.9 / <b>74.4</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>o</p><p>L L</p><p>2;</p><p>o U</p></td><td class="cell"><p>Japanese</p></td><td class="cell"><p>3,231</p></td><td class="cell"><p>80</p></td><td class="cell"><p>76.1 /76.2</p></td><td class="cell"><p>74.4 / 75.5</p></td><td class="cell"><p><b>78.6 </b>/ 77.4</p></td><td class="cell"><p>-</p></td><td class="cell"><p>77.4 / <b>78.5</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Portuguese</p></td><td class="cell"><p>28,931</p></td><td class="cell"><p>22</p></td><td class="cell"><p>51.6/64.4</p></td><td class="cell"><p>45.9 / 63.1</p></td><td class="cell"><p>57.4 / 69.2</p></td><td class="cell"><p>- / 75.3*</p></td><td class="cell"><p><b>63.9 </b>/ <b>76.8</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Slovene</p></td><td class="cell"><p>7,128</p></td><td class="cell"><p>29</p></td><td class="cell"><p>52.6/ <b>64.2</b></p></td><td class="cell"><p>44.0 / 60.3</p></td><td class="cell"><p><b>53.9 </b>/ 63.5</p></td><td class="cell"><p>-</p></td><td class="cell"><p>49.4 / 56.2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Spanish</p></td><td class="cell"><p>16,458</p></td><td class="cell"><p>47</p></td><td class="cell"><p>59.5/69.2</p></td><td class="cell"><p>54.8 / 68.2</p></td><td class="cell"><p>61.6/71.9</p></td><td class="cell"><p>- / <b>73.2*</b></p></td><td class="cell"><p><b>63.2 </b>/ 71.7</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Swedish</p></td><td class="cell"><p>20,057</p></td><td class="cell"><p>41</p></td><td class="cell"><p>53.2/62.2</p></td><td class="cell"><p>47.4 / 59.1</p></td><td class="cell"><p><b>58.9 </b>/ <b>68.7</b></p></td><td class="cell"><p>- / 60.6*</p></td><td class="cell"><p>58.0 / 68.2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Turkish</p></td><td class="cell"><p>17,563</p></td><td class="cell"><p>30</p></td><td class="cell"><p><b>40.8 </b>/ <b>62.8</b></p></td><td class="cell"><p>27.4 / 52.4</p></td><td class="cell"><p>36.8 / 58.1</p></td><td class="cell"><p>-</p></td><td class="cell"><p>40.2 / 58.7</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>French</p></td><td class="cell"><p>49,964</p></td><td class="cell"><p>23</p></td><td class="cell"><p>48.2/68.6</p></td><td class="cell"><p>46.3 /68.5</p></td><td class="cell"><p><b>57.3 </b>/ <b>77.8</b></p></td><td class="cell"><p>-</p></td><td class="cell"><p>55.0 / 76.6</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>A.Greek</p></td><td class="cell"><p>15,194</p></td><td class="cell"><p>15</p></td><td class="cell"><p>38.6/44.8</p></td><td class="cell"><p>24.2/38.5</p></td><td class="cell"><p>33.3 / <b>45.4</b></p></td><td class="cell"><p>-</p></td><td class="cell"><p><b>40.5 </b>/ 45.1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="9" global="646"/><p>comparison with a token-based version of the model that this restriction is very helpful. Second, it is a clustering model rather than a sequence model. This property makes it easy to incorporate multi­ple kinds of features into the model at either the to­ken or the type level. Here, we experimented with token-level context features and alignment features and type-level morphology features, showing that morphology features are helpful in nearly all cases, and alignment features can be helpful if the aligned language is properly chosen. Our results even with­out these extra features are competitive with state-of-the-art; with the additional features we achieve the best published results in the majority of the 25 corpora tested.</p><p>Since it is so easy to add extra features to our model, one direction for future work is to explore other possible features. For example, it could be useful to add dependency features from an unsuper-vised dependency parser. We are also interested in improving our morphology features, either by con­sidering other ways to extract features during pre­processing (for example, including prefixes or not concatenating together all suffixes), or by develop­ing a joint model for inducing both morphology and syntactic classes simultaneously. Finally, our model could be extended by replacing the standard mixture model with an infinite mixture model (Rasmussen, 2000) in order to induce the number of syntactic classes automatically.</p></section><section title="Acknowledgments"><p>The authors would like to thank Emily Thomforde, Ioannis Konstas, Tom Kwiatkowski and the anony­mous reviewers for their comments and suggestions. We would also like to thank Kiril Simov, Toni Marti, Tomaz Erjavec, Jess Lin and Kathrin Beck for pro­viding us with CoNLL data. This work was sup­ported by an EPSRC graduate Fellowship, and by</p><p>ERC Advanced Fellowship 249520 GRAMPLUS.</p></section><references><p>Anne Abeille, Lionel Clement, and Alexandra Kinyon. 2000. Building a treebank for French. In <i>In Proceed­ings of the LREC 2000.</i></p><p>David Bamman, Francesco Mambrini, and Gregory Crane. 2009. An ownership model of annotation: The</p><p>Ancient Greek dependency treebank. In <i>TLT 2009-Eighth International Workshop on Treebanks and Lin­guistic Theories.</i></p><p>Taylor Berg-Kirkpatrick, Alexandre B. Cote, John DeN­ero, and Dan Klein. 2010. Painless unsupervised learning with features. In <i>Proceedings of NAACL 2010, </i>pages 582-590, Los Angeles, California, June.</p><p>Chris Biemann. 2006. Unsupervised part-of-speech tag­ging employing efficient graph clustering. In <i>Proceed­ings of COLING ACL 2006, </i>pages 7-12, Morristown,</p><p>NJ, USA.</p><p>Alena Bohmova, Jan Hajic, Eva Hajicova, and Barbora Hladka. 2001. The Prague dependency treebank: Three-level annotation scenario. In Anne Abeille, ed­itor, <i>Treebanks: Building and Using Syntactically An­notated Corpora, </i>pages 103 - 126. Kluwer Academic Publishers.</p><p>Sabine Brants, Stefanie Dipper, Silvia Hansen, Wolfgang</p><p>Lezius, and George Smith. 2002. The TIGER treebank. In <i>Proceedings </i><i>ofthe</i><i> Workshop on Treebanks and Linguistic Theories, </i>Sozopol.</p><p>Peter F. Brown, Vincent J. Della Pietra, Peter V. Desouza, Jennifer C. Lai, and Robert L. Mercer. 1992. Class-based n-gram models of natural language. <i>Computa­tional Linguistics, </i>18(4):467—479.</p><p>Sabine Buchholz and Erwin Marsi. 2006. CoNLL-X shared task on multilingual dependency parsing. In <i>Proceedings </i><i>ofthe</i><i> Tenth Conference on Compu­tational Natural Language Learning, </i>CoNLL-X '06, pages 149-164, Stroudsburg, PA, USA. Association for Computational Linguistics.</p><p>Christos Christodoulopoulos, Sharon Goldwater, and Mark Steedman. 2010. Two decades of unsupervised POS induction: How far have we come? In <i>Proceed­ings of the 2010 Conference on Empirical Methods in Natural Language Processing, </i>pages 575-584, Cam­bridge, MA, October. Association for Computational Linguistics.</p><p>Montserrat Civit, Ma. Marti, and Nuria Buff. 2006. Cat3lb and cast3lb: From constituents to dependen­cies. In Tapio Salakoski, Filip Ginter, Sampo Pyysalo, and Tapio Pahikkala, editors, <i>Advances in Natural Language Processing, </i>volume 4139 of <i>Lecture Notes in Computer Science, </i>pages 141-152. Springer Berlin / Heidelberg.</p><p>Alexander Clark. 2003. Combining distributional and morphological information for part of speech induc­tion. In <i>Proceedings of EACL 2003, </i>pages 59-66, Morristown, NJ, USA.</p><p>Mathias Creutz and Krista Lagus. 2005. Induc­ing the morphological lexicon of a natural language from unannotated text. In <i>In Proceedings of the International and Interdisciplinary Conference on</i> <i>Adaptive Knowledge Representation and Reasoning (AKRR'05), </i>volume 5, pages 106-113.<page local="10" global="647"/> Saso Dzeroski, Tomaz Erjavec, Nina Ledinek, Petr Pajas, Zdenek Zabokrtsky, and AndrejaZele. 2006. Towards a Slovene dependency treebank. In <i>Proceedings Int. Conf. on Language Resources and Evaluation (LREC). </i>Tomaz Erjavec. 2004. MULTEXT-East version 3: Mul­tilingual morphosyntactic specifications, lexicons and corpora. In <i>Fourth International Conference on Lan­guage Resources and Evaluation, (LREC'04), </i>pages 1535 - 1538, Paris. ELRA.</p><p>Sharon Goldwater and Tom Griffiths. 2007. A fully bayesian approach to unsupervised part-of-speech tag­ging. In <i>Proceedings of ACL 2007, </i>pages 744-751, Prague, Czech Republic, June.</p><p>Aria Haghighi and Dan Klein. 2006. Prototype-driven learning for sequence models.   In <i>Proceedings of</i> <i>NAACL 2006, </i>pages 320-327, Morristown, NJ, USA.</p><p>Jan Hajic, Jarmila Panevova, Zdenka Uresova, Alevtina Bemova, and Petr Pajas. 2003. PDTVALLEX: cre­ating a large-coverage valency lexicon for treebank annotation. In <i>Proceedings ofThe Second Workshop on Treebanks and Linguistic Theories, </i>pages 57-68. Vaxjo University Press.</p><p>Mark Johnson. 2007. Why doesn't EM find good HMM POS-taggers? In <i>Proceedings of EMNLP-CoNLL 2007, </i>pages 296-305, Prague, Czech Republic, June.</p><p>Yasushira Kawata and Julia Bartels. 2000. Stylebook for the Japanese treebank in VERMOBIL. Technical report, Universitat Tuubingen. Chen Keh-Jiann, Chu-Ren Huang, Feng-Yi Chen, Chi­Ching Luo, Ming-Chung Chang, Chao-Jan Chen, and Zhao-Ming Gao. 2003. Sinica treebank: Design cri­teria, representational issues and implementation. In Anne Abeille, editor, <i>Treebanks: Building and Us­ing Syntactically Annotated Corpora, </i>pages 231-248. Kluwer Academic Publishers.</p><p>Dan Klein and Christopher D. Manning. 2002. A gener­ative constituent-context model for improved grammar induction. In <i>Proceedings ofACL 40, </i>pages 128-135. Michael Lamar, Yariv Maron, Mark Johnson, and Elie Bienenstock. 2010. SVD and clustering for unsuper-vised POS tagging. In <i>Proceedings ofthe ACL 2010 Conference Short Papers, </i>pages 215-219, Strouds-burg, PA, USA. Association for Computational Lin­guistics.</p><p>Yoong Keok Lee, Aria Haghighi, and Regina Barzilay. 2010. Simple type-level unsupervised POS tagging. In <i>Proceedings ofthe 2010 Conference on Empirical Methods in Natural Language Processing, </i>EMNLP '10, pages 853-861, Stroudsburg, PA, USA. Associ­ation for Computational Linguistics.</p><p>Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated cor­pus of English: the Penn Treebank. <i>Computational Linguistics, </i>19(2):331-330.</p><p>Tahira Naseem, Benjamin Snyder, Jacob Eisenstein, and Regina Barzilay. 2009. Multilingual Part-of-Speech tagging: Two unsupervised approaches. <i>Journal of Ar­tificial Intelligence Research, </i>36:341-385.</p><p>Franz J. Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. <i>Comput. Linguist., </i>29(1):19-51.</p><p>Kemal Oflazer, Bilge Say, Dilek Z. Hakkani-Tur, and Gokhan Tür, 2003. <i>Building A Turkish Treebank, </i>chapter 1, pages 1-17. Kluwer Academic Publishers.</p><p>Carl Rasmussen. 2000. The infinite Gaussian mixture model. In <i>Advances in Neural Information Processing Systems 12.</i></p><p>Sujith Ravi and Kevin Knight. 2009. Minimized mod­els for unsupervised Part-of-Speech tagging. In <i>Pro­ceedings ofACL-IJCNLP 2009, </i>pages 504-512, Sun-</p><p>tec, Singapore, August. Martin Redington, Nick Chater, and Steven Finch. 1998. Distributional information: a powerful cue for acquir­ing syntactic categories. <i>Cognitive Science, </i>22:425 469.</p><p>Andrew Rosenberg and Julia Hirschberg. 2007. V-measure: A conditional entropy-based external clus­ter evaluation measure. In <i>Proceedings ofEMNLP-CoNLL 2007, </i>pages 410-420.</p><p>Hinrich Schutze. 1995. Distributional part-of-speech tagging. In <i>Proceedings of EACL 7, </i>pages 141-148, San Francisco, CA, USA.</p><p>Kiril Simov, Petya Osenova, Alexander Simov, and Milen Kouylekov. 2004. Design and implementation of the Bulgarian HPSG-based treebank. <i>Research on Lan­guage &amp;amp; Computation, </i>2(4):495-522.</p><p>Noah A. Smith and Jason Eisner. 2005. Contrastive esti­mation: training log-linear models on unlabeled data. In <i>Proceedings ofACL 2005, </i>pages 354-362, Morris-town, NJ, USA.</p><p>Otakar Smrzs and Petr Pajas. 2004. Morphotrees of Ara­bic and their annotation in the TrEd environment. In <i>Proceedings ofthe NEMLAR International Conference on Arabic Language Resources and Tools, </i>pages 38­41.</p><p>Ben Snyder and Regina Barzilay. 2008. Unsupervised multilingual learning for morphological segmentation. In <i>Proceedings ofACL.</i></p><p>Kristina Toutanova and Mark Johnson. 2007. A Bayesian LDA-based model for semi-supervised part-of-speech tagging. In <i>Proceedings ofNIPS 2007.</i></p></references></body></article>