<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="678"/><title>The infinite HMM for unsupervised PoS tagging</title><author surname="Van Gael" givenname="Jurgen"><org  name="Cambridge University" country="United Kingdom" city="Cambridge"/></author><author surname="Vlachos" givenname="Andreas"><org  name="Cambridge University" country="United Kingdom" city="Cambridge"/></author><author surname="Ghahramani" givenname="Zoubin"><org  name="Cambridge University" country="United Kingdom" city="Cambridge"/></author></firstpageheader><frontmatter><p><b>The infinite HMM for unsupervised PoS tagging</b></p><p><b>Jürgen Van Gael Andreas Vlachos Zoubin Ghahramani</b></p><p>Department of Engineering        Computer Laboratory Department of Engineering</p><p>University of Cambridge University of Cambridge       University of Cambridge</p><p>jv249@cam.ac.uk av308@cl.cam.ac.uk zoubin@eng.cam.ac.uk</p></frontmatter><abstract>We extend previous work on fully unsu­pervised part-of-speech tagging. Using a non-parametric version of the HMM, called the infinite HMM (iHMM), we ad­dress the problem of choosing the number of hidden states in unsupervised Markov models for PoS tagging. We experi­ment with two non-parametric priors, the Dirichlet and Pitman-Yor processes, on the Wall Street Journal dataset using a paral­lelized implementation of an iHMM in­ference algorithm. We evaluate the re­sults with a variety of clustering evalua­tion metrics and achieve equivalent or bet­ter performances than previously reported. Building on this promising result we eval­uate the output of the unsupervised PoS tagger as a direct replacement for the out­put of a fully supervised PoS tagger for the task of shallow parsing and compare the two evaluations. </abstract></header><body><section number="1" title="Introduction"><p>Many Natural Language Processing (NLP) tasks are commonly tackled using supervised learning approaches. Lhese learning methods rely on the availability of labeled datasets which are usually produced by expensive manual annotation. Lor some tasks, we have the choice to use unsuper­vised learning approaches. While they do not nec­essarily achieve the same level of performance, they are appealing as unlabeled data is usually abundant. In particular, for the purpose of ex­ploring new domains and languages, obtainining labeled material can be prohibitively expensive and unsupervised learning methods are a very at­tractive choice. Recent work (Johnson, 2007; Goldwater and Griffiths, 2007; Gao and Johnson, 2008) explored the task of part-of-speech tagging (PoS) using unsupervised Hidden Markov Models (HMMs) with encouraging results. PoS tagging is a standard component in many linguistic process­ing pipelines, so any improvement on its perfor­mance is likely to impact a wide range of tasks.</p><p>It is important to point out that a completely unsupervised learning method will discover the statistics of a dataset <i>according</i><i> </i><i>to</i><i> </i><i>a</i><i> </i><i>particular</i><i> </i><i>model</i><i> </i><i>choice</i><i> </i>but these statistics might not cor­respond exactly to our intuition about PoS tags. Johnson (2007) and Gao &amp; Johnson (2008) as­sume that words are generated by a hidden Markov model and find that the resulting states strongly correlate with POS tags. Nonetheless, <i>identifying</i><i> </i>the HMM states with appropriate POS tags is hard. Because many evaluation methods often require POS tags (rather than HMM states) this identifica­tion problem makes unsupervised systems difficult to evaluate.</p><p>One potential solution is to add a small amount of supervision as in Goldwater &amp; Griffiths (2007) who assume a dictionary of frequent words asso­ciated with possible PoS tags extracted from a la­beled corpus. Although this technique improves performance, in this paper we explore the com­pletely unsupervised approach. Lhe reason for this is that better unsupervised approaches provide us with better starting points from which to explore how and where to incorporate supervision.</p><p>In previous work on unsupervised PoS tagging a main question was how to set the number of hid­den states appropriately. Johnson (2007) reports results for different numbers of hidden states but it is unclear how to make this choice a priori, while Goldwater &amp; Griffiths (2007) leave this question as future work.</p><p>It is not uncommon in statistical machine learn­ing to distinguish between parameters of a model and the capacity of a model. E.g. in a clustering context, the choice for the number of clusters (ca­pacity) and the parameters of each cluster are often</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">678</doubt><p><i>Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, </i>pages 678-687, Singapore, 6-7 August 2009. ©2009 ACL and AFNLP<page local="2" global="679"/></p><p>treated differently: the latter are estimated using algorithms like EM, MCMC or Variational Bayes while the former is chosen using common sense, heuristics or in a Bayesian framework maybe us­ing evidence maximization.</p><p>Non-parametric Bayesian methods are a class of probability distributions which explicitly treat the capacity of a model as "just another parameter". Potential advantages are</p><p>• the model capacity can automatically adjust to the amount of data: e.g. when clustering a very small dataset, it is unlikely that many fine grained clusters can be distinguished,</p><p>• inference can be more efficient: e.g. instead of running full inference for different model capacities and then choosing the best ca­pacity (according to some choice of "best"), inference in non-parametric Bayesian meth­ods integrates the capacity search in one al­gorithm. This is particularly advantageous when parameters other than capacity need to be explored, since it reduces signifcantly the number of experiments needed.</p><p>None of these potential advantages are guaranteed and in this paper we investigate these two aspects for the task of unsupervised PoS tagging.</p><p>The contributions in this paper extend previous work on unsupervised PoS tagging in five ways. First, we introduce the use of a non-parametric version of the HMM, namely the infinite HMM (iHMM) (Beal et al., 2002) for unsupervised PoS tagging. This answers an open problem from Goldwater &amp; Griffiths (2007). Second, we care­fully implemented a parallelized version of the inference algorithms for the iHMM so we could use it on the Wall Street Journal Penn Treebank dataset. Third, we introduce a new variant of the iHMM that builds on the Pitman-Yor process. Fourth, we evaluate the results with a variety of clustering evaluation methods and achieve equiv­alent or better performances than previously re­ported. Finally, building on this promising result we use the output of the unsupervised PoS tagger as a direct replacement for the output of a fully su­pervised PoS tagger for the task of shallow pars­ing. This evaluation enables us to assess the appli­cability of an unsupervised PoS tagging method and provides us with means of comparing its per­formance against a supervised PoS tagger.</p><p>The rest of the paper is structured as follows: in section 2 we introduce the iHMM as a non-parametric version of the Bayesian HMM used in previous work on unsupervised PoS tagging. Then, in section 3 we describe some details of our implementation of the iHMM. In section 4 we present a variety of evaluation metrics to compare our results with previous work. Finally, in sec­tion 5 we report our experimental results. We con­clude this paper with a discussion of ongoing work and experiments.</p></section><section number="2" title="The Infinite HMM"><p>In this section, we describe a non-parametric hid­den Markov model known as the infinite HMM (iHMM) (Beal et al., 2002; Teh et al., 2006). As we show below, this model is flexible in the num­ber of hidden states which it can accomodate. In other words, <i>the</i><i> </i><i>capacity</i><i> </i><i>is</i><i> </i><i>an</i><i> </i><i>uncertain</i><i> </i><i>quantity</i><i> </i>with an a priori infinite range that is a posteriori inferred by the data. It is instructive to first re­view the finite HMM and its Bayesian treatment: for one, it is the model that has been used in previ­ous work on unsupervised PoS tagging, secondly it allows us to better understand the iHMM.</p><p><b>The Bayesian HMM </b>A finite first-order HMM consists of a hidden state sequence <i>s = </i>(si, <i>s2, </i>• • •, <i>st) </i>and a corresponding observation sequence <i>y = </i>(2/1,2/2, - - - ? <i>Ut)- </i>Each state vari­able <b><i>St </i></b>can take on a finite number of states, say 1... <i>K. </i>Transitions between states are governed by Markov dynamics parameterized by the tran­sition matrix <b>7r, </b>where <b><i>Kij </i></b><i>= p</i><b><i>(st = j\st-i = i), </i></b>while the initial state probabilities are <b><i>iroi = </i></b><i>p</i><b><i>{s\ = i). </i></b>For each state <b><i>st </i></b>G {1... <i>K}</i><i> </i>there is a parameter <b><i>&lt;fiSt </i></b>which parameterizes the obser­vation likelihood for that state: <b><i>yt\st </i></b>~ <i>F</i><i>(&lt;fiSt). </i>Given the parameters <i>{ttq, </i><b>7r, </b><i>4&gt;,</i><i> </i><i>K}</i><i> </i>of the HMM, the joint distribution over hidden states <i>s </i>and ob­servations <i>y </i>can be written (with <b><i>so </i></b>= 0):</p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">t</doubt><doubt alpha="47.5" length="40" tooSmall="False" monospace="0.0">p(s, y\ir0,7T, 0,K)=Y\p(st\st-i)p(yt\st)</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">t=i</doubt><p>As Johnson (2007) clearly explained, training the HMM with EM leads to poor results in PoS tag­ging. However, we can easily treat the HMM in a fully Bayesian way (MacKay, 1997) by introduc­ing priors on the parameters of the HMM. With no further prior knowledge, a typical prior for the transition (and initial) probabilities are symmet­ric Dirichlet distributions. This corresponds to our belief that, a priori, each state is equally likely to transition to every other state.<page local="3" global="680"/> Also, it is com­monly known that the parameter of a Dirichlet distribution controls how sparse its samples are. In other words, by making the hyperprior on the Dirichlet distribution for the rows of the transi­tion matrix small, we can encode our belief that any state (corresponding to a PoS tag in this ap­plication context) will only be followed by a small number of other states. As we explain below, we will be able to include this desirable property in the non-parametric model as well. Secondly, we need to introduce a prior on the observation pa­rameters <b><i>fa. </i></b>Without any further prior knowl­edge, a convenient choice here is another sym­metric Dirichlet distribution with sparsity induc­ing hyperprior. This encodes our belief that only a subset of the words correspond to a particular state.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">679</doubt><p><b>The Infinite HMM </b>A first naive way to obtain a non-parametric HMM with an infinite number of states might be to use symmetric Dirichlet pri­ors over the transition probabilities with parameter <b><i>a/K </i></b>and take <b><i>K </i></b>—&gt; oo. This approach unfortu­nately does not work: <b><i>a/K </i></b>—&gt; 0 when <b><i>K </i></b>—&gt; oo and hence the rows of the matrix will become "in­finitely sparse". Since the sum of the entries must sum to one, the rows of the transition matrix will be zero everywhere and all its mass in a random location. Unfortunately, this random location is out of an infinite number of possible locations and hence with probability 1 will be different for all the rows. As a consequence, at each timestep the HMM moves to a new state and will never revisit old states. As we shall see shortly, we can fix this by using a hierarchical Bayesian formalism where the Dirichlet priors on the rows have a shared pa­rameter.</p><p>Before moving on to the iHMM, let us look at the finite HMM from a different perspective. The finite HMM of length <b><i>T </i></b>with <b><i>K </i></b>hidden states can be seen as a sequence of <b><i>T </i></b>finite mixture models. The following equation illustrates this idea: con­ditioned on the previous state <b><i>st-i, </i></b>the marginal probability of observation <b><i>yt </i></b>can be written as:</p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">K</doubt><doubt alpha="44.2" length="43" tooSmall="False" monospace="0.0">p(yt\st-i = k)   =^2p(st\st-i = k)p(yt\st),</doubt><doubt alpha="57.1" length="7" tooSmall="False" monospace="0.0">st= l K</doubt><doubt alpha="41.7" length="24" tooSmall="False" monospace="0.0">=   J2^k,stp(yt\fat)-(!)</doubt><doubt alpha="60.0" length="5" tooSmall="False" monospace="0.0">st= l</doubt><p>The variable <b><i>St-\ = k </i></b>specifies the mixing weights 7Tfc;. for the mixture distribution, while <b><i>St </i></b>indexes the mixture component generating the ob­servation <b><i>yt. </i></b>In other words, equation (1) says that each row of the transition matrix <b>7r </b>specifies a dif­ferent mixture distribution over the same set of <b><i>K </i></b>mixture components <i>(p.</i></p><p>Our second attempt to define a non-parametric version of the hidden Markov model is to replace the finite mixture by an infinite mixture. The theory of Dirichlet process mixtures (Antoniak, 1974) tells us exactly how to do this. A draw G ~ <b><i>DP(a, </i></b><b><i>H)</i></b><b><i> </i></b>from a Dirichlet process (DP) with base measure <b><i>H </i></b>and concentration parame­ter <b><i>a &gt; </i></b>0 is a discrete distribution which can be written as an infinite mixture of atoms where the <b><i>fa </i></b>are i.i.d. draws from the base mea­sure <b><i>H,</i></b><b><i> </i></b>&lt;^(-) represents a point distribution at <b><i>fa </i></b>and <b><i>TTi = </i></b><i>ViYl</i><b><i>]~1(l </i></b><i>— </i><i>vi) </i>where each <i>vi</i><i> </i><i>~</i><i> </i>Beta(1, a). The distribution over 7Tj is called a stick breaking construction and is essentially an infinite dimensional version of the Dirichlet dis­tribution. We refer to Teh et al. (2006) for more details.</p><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">oo</doubt><doubt alpha="23.1" length="13" tooSmall="False" monospace="0.0">G(-) =E7r^(')</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">i=l</doubt><p>Switching back to the iHMM our next step is to introduce a DP <b><i>Gj </i></b>for each state <b><i>j </i></b>e {1 • • • oo}; we write Gj(-) = <b><i>Y^i^l^^i')- </i></b>There is now a parameter for each state <b><i>j </i></b>and each index <b><i>i </i></b>e {1,2, • • • , oo}. Next, we draw the datapoint at timestep <b><i>t </i></b>given that the previous datapoint was in state <b><i>st-i </i></b>by drawing from DP <b><i>GSt_1. </i></b>We first se­lect a mixture component <b><i>St </i></b>from the vector <b><i>nSt-i,-</i></b>and then sample a datapoint <b><i>yt </i></b>~ <b><i>F(&lt;fiSt_ltSt) </i></b>so we get the following distribution for <b><i>yt</i></b></p><doubt alpha="59.3" length="27" tooSmall="False" monospace="0.0">p(yt\a, st-i) =Ylp(yt\fat-i</doubt><p>This is almost the non-parametric equivalent of equation (1) but there is a subtle difference: each <b><i>Gj </i></b>selects their own set of parameters <b><i>fa.. </i></b>This is unfortunate as it means that the output distribu­tion would not be the same for each state, it would depend on which state we were moving to ! Luck­ily, we can easily fix this: by introducing an in­termediate distribution Go ~ <b><i>DP</i></b><i>{l,</i><i> </i><b><i>H)</i></b><b><i> </i></b>and let <b><i>Gj</i></b><b><i> ~ DP(a, Go) </i></b>we enforce that the i.i.d. draws <b><i>fa </i></b>are draws from a discrete distribution (since Go is a draw from a Dirichlet process) and hence all <i>Gj </i>will share the same infinite set of atoms as cho­sen by Go.<page local="4" global="681"/> Figure 1 illustrates the graphical model for the iHMM.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">680</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">oo</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">i=l</doubt><p><b>The  iHMM  with  Pitman-Yor  Prior </b>The</p><p>Dirichlet process described above defines a very specific distribution over the number of states in the iHMM. One particular generalization of the Dirichlet process that has been studied in the NLP literature before is the <i>Pitman-Yor</i><i> </i>process. Goldwater et al. (2006) have shown that the Pitman-Yor distribution can more accurately cap­ture power-law like distributions that frequently occur in natural language.</p><p>More specifically, a draw G ~ <i>PY(d, a, H) </i>from a Pitman-Yor process (PY) with base mea­sure <i>H, </i>discount parameter 0 &lt; <i>d &lt; </i>1 and con­centration parameter <i>a &gt; — d </i>is a discrete distri­bution which can be written as an infinite mixture of atoms where the <i>fa </i>are i.i.d. draws from the base mea­sure <i>H, </i>&lt;^(-) represents a point distribution at <i>fa </i>and <b><i>TTi = </i></b><i>ViYl</i><b><i>]Z</i></b><i>1(l </i><i>— </i><i>vi) </i>where each <i>vi</i><i> </i><i>~</i><i> </i>Beta(l <i>-d,a + ld). </i>Note the similarity to the DP: in fact, the DP is a special case of PY with <i>d = 0.</i></p><doubt alpha="21.4" length="14" tooSmall="False" monospace="0.0">c(-) = E7r^(-)</doubt><p>In our experiments, we constructed an iHMM where the <i>DP (a, </i><i>H)</i><i> </i>base measure Go is re­placed with its two parameter generalization <i>PY(d, a, H). </i>Because the Dirichlet and Pitman-Yor processes only differ in the way 7r is con­structed, without loss of generality we will de­scribe hyper-parameter choice and inference in the context of the iHMM with Dirichlet process base measure.</p><p><b>Hyperparameter Choice </b>The description above shows that there are 4 parameters which we must specify: the base measure <i>H, </i>the output distribution <i>p(yt</i><i>\&lt;fist), </i>the discount<footnote anchor="1"/> and concentration<footnote anchor="2"/> parameters <i>d, </i>7 for Go and the concentration parameter <i>a </i>for the DP's <i>Gj. </i>Just as in the finite case, the base measure <i>H </i>is the prior distribution on the parameter <i>&lt;fi </i>of <i>p(yt</i><i>\&lt;fist)- </i>We chose to use a symmetric Dirichlet distribution with parameter <i>ö </i>over the word types in our corpus. Since we do not know the sparsity level <i>ö </i>of the output distributions we decided to learn this</p><p>'for Pitman-Yor base measure</p><footnote label="2">for both Dirichlet and Pitman-Yor base measures</footnote><p>parameter from the data. We initially set a vague Gamma prior over <i>ö </i>but soon realized that as we expect hidden states in the iHMM to correspond to PoS tags, it is unrealistic to expect each state to have the same sparsity level. Hence we chose a Dirichlet process as the prior for <i>6; </i>this way we end up with a small discrete set of sparsity levels: e.g. we can learn that states corresponding to verbs and nouns share one sparsity level while states correpsonding to determiners have their own (much sparser) sparsity level. For the output distribution <i>p(yt\&lt;pSt) </i>we chose a simple multinomial distribution.</p><p>The hyperparameters <i>d </i>and 7 mostly control the number of states in the iHMM while - as we dis­cussed above - <i>a </i>controls the sparsity of the tran­sition matrix. In the experiments below we report both fixing the two parameters and learning them by sampling (using vague Gamma hyperpriors). Because of computational constraints, we chose to use vague Bayesian priors for all hyperparameters rather than run the whole experiment over a grid of "reasonable" parameter settings and use the best ones according to cross validation.</p></section><section number="3" title="Inference"><p>The Wall Street Journal part of the Penn Tree-bank that was used for our experiments contains about one million words. In the non-parametric Bayesian literature not many algorithms have been described that scale into this regime. In this sec­tion we describe our parallel implementation of the iHMM which can easily handle a dataset of this scale.</p><p>There is a wealth of evidence (Scott, 2002; Gao and Johnson, 2008) in the machine learning litera­ture that Gibbs sampling for Markov models leads to slow mixing times. Hence we decided our start­ing point for inference needs to be based on dy­namic programming. Because we didn't have a good idea for the number of states that we were go­ing to end up with, we prefered the beam sampler of Van Gael et al. (2008) over a finite truncation of the iHMM. Moreover, the beam sampler also introduces a certain amount of sparsity in the dy­namic program which can speed up computations (potentially at the cost of slower mixing).</p><p>The beam sampler is a blocked Gibbs sampler where we alternate between sampling the param­eters (transition matrix, output parameters), the state sequence and the hyperparameters. Sampling the transition matrix and output distribu­tion parameters requires computing their sufficient statistics and sampling from a Dirichlet distribu­tion; we refer to the beam sampling paper for de­tails.<page local="5" global="682"/> For the hyperparameters we use standard Gibbs sampling. We briefly sketch the resam­pling step for the state sequence for a single se­quence of data (sentence of words). Running stan­dard dynamic programming is prohibitive because the state space of the iHMM is infinitely large. The central idea of the beam sampler is to adap-tively truncate the state space of the iHMM and run dynamic programming. In order to truncate the state space, we sample an auxilary variable <i>Ut </i>for each word in the sequence from the distribu­tion <i>Ut </i>~ Uniform(0,7rSt_1St) where 7r represents the transition matrix.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">681</doubt><figure caption="Figure 1: The graphical model for the iHMM. The variableßrepresents the mixture for the DPGo."></figure><p>Intuitively, when we sample <i>u\:t\si-.t </i>accord­ing to the distribution above, the only valid sam­ples are those for which the <i>ut </i>are smaller than the transition probabilities of the state sequence <i>si</i><i>;t- </i>This means that when we sample <i>s\:t\u\:t </i>at a later point, it must be the case that the <i>ut's </i>are still smaller than the new transition probabil­ities. This significantly reduces the set of valid state sequences that we need to consider. More specifically, Van Gael et al. (2008) show that we can compute <i>p(st\yi:t,ui:t) </i>using the following dynamic programming <i>recursion</i><i> </i><i>p(st\yi-.t, u\:t) =</i></p><doubt alpha="59.4" length="32" tooSmall="False" monospace="0.0">P(yt\St)Ylp(st-l\yi:t-l,Ul:t-l)-</doubt><doubt alpha="56.2" length="16" tooSmall="False" monospace="0.0">st-i:ut&lt;^st_1,st</doubt><p>The summation Y\    .„ ensures that this computation remains finite. When we compute <i>p(st\yi:</i><i>t,ui:t) </i>for <i>t </i>G {1---T}, we can easily sample <i>st </i>and using Bayes rule backtrack sample every other st. It can be shown that this procedure produces samples from the exact posterior.</p><p>Notice that the dynamic program only needs to perform computation when <i>ut &lt; </i>7rSt_1)St. A care­ful implementation of the beam sampler consists of preprocessing the transition matrix 7r and sort­ing its elements in descending order. We can then iterate over the elements of the transition matrix starting from the largest element and stop once we reach the first element of the transition matrix smaller than <i>ut. </i>In our experiments we found that this optimization reduces the amount of computa­tion per sentence by an order of magnitutde.</p><p>A second optimization which we introduced is to use the map-reduce paradigm (Dean and Ghemawat, 2004) to parallelize our computations. More specifically, after we preprocess the transi­tion matrix, the dynamic program computations are independent for each sentence in the dataset. This means we can perform each dynamic pro­gram in parallel; in other words our "map" con­sists of running the dynamic program on one sen­tence in the dataset. Next, we need to resample the transition matrix and output distribution pa­rameters. In order to do so we need to compute their sufficient statistics: the number of transitions from state to state and the number of emissions of each word out of each state. Our "reduce" func­tion consists of computing the sufficient statistics for each sentence and then aggregating the statis­tics for the whole dataset. Our implementation runs on a quad-core shared memory architecture and we find an almost linear speedup going from one to four cores.</p></section><section number="4" title="Evaluation"><p>Evaluating unsupervised PoS tagging is rather dif­ficult mainly due to the fact that the output of such<page local="6" global="683"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">682</doubt><p>Systems are not actual PoS tags but state identi­fiers. Therefore it is impossible to evaluate per­formance against a manually annotated gold stan­dard using accuracy. Recent work (Goldwater and Griffiths, 2007; Johnson, 2007; Gao and Johnson, 2008) on this task explored a variety of method­ologies to address this issue.</p><p>The most common approach followed in pre­vious work is to evaluate unsupervised PoS tag­ging as clustering against a gold standard using the Variation of Information (VI) (Meilä, 2007). VI assesses homogeneity and completeness us­ing the quantities <i>H(C\K) </i>(the conditional en­tropy of the class distribution in the gold stan­dard given the clustering) and <i>H(K\C) </i>(the con­ditional entropy of clustering given the class dis­tribution in the gold standard). However, as Gao &amp; Johnson (2008) point out, VI is biased to­wards clusterings with a small number of clus­ters. A different evaluation measure that uses the same quantities but weighs them differently is the V-measure (Rosenberg and Hirschberg, 2007), which is defined in Equation 2 by setting the pa­rameter <i>ß </i>to 1.</p><doubt alpha="25.0" length="4" tooSmall="False" monospace="0.0">h =1</doubt><doubt alpha="20.0" length="5" tooSmall="False" monospace="0.0">c = 1</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">Vß=</doubt><doubt alpha="50.0" length="6" tooSmall="False" monospace="0.0">H(C\K)</doubt><doubt alpha="50.0" length="6" tooSmall="False" monospace="0.0">H{K\C)</doubt><doubt alpha="40.9" length="22" tooSmall="False" monospace="0.0">H{K) (l + ß)hc (ßh)+ c</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(2)</doubt><p>Vlachos et al. (2009) noted that V-measure favors clusterings with a large number of clusters. Both of these biases become crucial in our experiments, since the number of clusters (states of the iHMM) is not fixed in advance. Vlachos et al. proposed a variation of the V-measure, V-beta, that adjusts the balance between homogeneity and completeness using the parameter <i>ß </i>in Eq. 2.</p><p>It is worth mentioning that, unlike V-measure and V-beta, VI scores are not normalized and therefore they are difficult to interpret. Meilä (2007) presented two normalizations, acknowledging the potential disadvantages they have. The first one normalizes VI by 2<i>\og(max(\K\, </i>|C|)), which is inappropriate when the number of clusters discovered <i>\K\ </i>changes between experiments. The second normalization involves the quantity log <i>N </i>which is appropriate when comparing different algo­rithms on the same dataset <i>(N</i><i> </i>is the number of instances). However, this quantity depends exclusively on the size of the dataset and hence if the dataset is very large it can result in normalized VI scores misleadingly close to 100%. This does not affect rankings, i.e. a better VI score will also be translated into a better normalized VI score. In our experiments, we report results only with the un-normalized VI scores, V-measure and V-beta.</p><p>All the evaluation measures mentioned so far evaluate PoS tagging as a clustering task against a manually annotated gold standard. While this is reasonable, it still does not provide means of assessing the performance in a way that would allow comparisons with supervised methods that output actual PoS tags. Even for the normalized measures V-measure and V-beta, it is unclear how their values relate to accuracy levels. Gao &amp; John­son (2008) partially addressed this issue by map­ping states to PoS tags following two different strategies, cross-validation accuracy, and greedy 1-to-l mapping, which both have shortcomings. We argue that since an unsupervised PoS tagger is trained without taking any gold standard into ac­count, it is not appropriate to evaluate against a particular gold standard, or at least this should not be the sole criterion. The fact that different authors use different versions of the same gold standard to evaluate similar experiments (e.g. Goldwater &amp; Griffiths (2007) versus Johnson (2007)) supports this claim. Furthermore, PoS tagging is seldomly a goal in itself, but it is a component in a linguistic pipeline.</p><p>In order to address these issues, we perform an extrinsic evaluation using a well-explored task that involves PoS tags. While PoS tagging is consid­ered a pre-processing step in many natural lan­guage processing pipelines, the choice of task is restricted by the lack of real PoS tags in the out­put of our system. For our purposes we need a task that relies on discriminating between PoS tags rather than the PoS tag semantics themselves, in other words, a task in which knowing whether a word is tagged as noun instead of a verb is equiv­alent to knowing it is tagged as state 1 instead of state 2. Taking these considerations into account, in Section 5 we experiment with shallow pars­ing in the context of the CoNLL-2000 shared task (Tjong Kim Sang and Buchholz, 2000) in which very good performances were achieved using only the words with their PoS tags. Our intuition is that if the iHMM (or any unsupervised PoS tagging method) has a reasonable level of performance, it should improve on the performance of a system that does not use PoS tags.<page local="7" global="684"/> Moreover, if the per­formance is very good indeed, it should get close to the performance of a system that uses real PoS tags, provided either by human annotation or by a good supervised system. Similar extrinsic evalu­ation was performed by Biemann et al. (2007). It is of interest to compare the results between the clustering evaluation and the extrinsic one.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">683</doubt><p>A different approach in evaluating non-parametric Bayesian models for NLP is state-splitting (Finkel et al., 2007; Liang et al., 2007). In this setting, the model is used in order to re­fine existing annotation of the dataset. While this approach can provide us with some insights and interprétable results, the use of existing annotation influences the output of the model. In this work, we want to verify whether the output of the iHMM (without any supervision) can be used instead of that of a supervised system.</p></section><section number="5" title="Experiments"><p>In all our experiments, the Wall Street Journal (WSJ) part of the Penn Treebank was used. As ex­plained in Section 4, we evaluate the output of the iHMM in two ways, as clustering with respect to a gold standard and as direct replacement of the PoS tags in the task of shallow parsing. In each experi­ment, we obtain a sample from the iHMM over all the sections of WSJ. The states for sections 15-18 and 20 of the WSJ (training and testing sets re-spectvely in the CoNLL shared task) are used for the evaluation based on shallow parsing, while the remaining sections are used for evaluation against the WSJ gold standard PoS tags using clustering evaluation measures.</p><p>As described in Section 2 we performed three runs with the iHMM: one run with DP prior and fixed 7, <i>a, </i>one with PY prior and fixed <i>d, </i>7, <i>a </i>and one with DP prior but where we learn the hyper­parameters 7, <i>a </i>from the data. Our inference algo­rithm uses 1000 burn-in iterations after which we collect a sample every 1000 iterations. Our infer­ence procedure is annealed during the first 1000 burnin and 2400 iterations by powering the likeli­hood of the output distribution with a number that smoothly increases from 0.4 to 1.0 over the 3400 first iterations. The numbers of iterations reported in the remainder of the section refer to the itera­tions after burn-in. We initialized the sampler by:</p><p>a) sampling the hyperparameters from the prior where applicable, b) uniformly assign each word one out of 20 iHMM states. For the DP run with fixed parameters, we chose <i>a = </i>0.8 to encourage some sparsity in the transition matrix and 7 = 5.0 to allow for enough hidden states. For the PY run with fixed parameters, we chose <i>a = </i>0.8 for simi­lar reasons and <i>d = </i>0.1 and 7 = 1.0. We point out that one weakness of MCMC methods is that they are hard to test for convergence. We chose to run the simulations until they became prohibitively ex­pensive to obtain a new sample.</p><p>First, we present results using clustering eval­uation measures which appear in the figures of Table 1. The three runs exhibit different behav­ior. The number of states reached by the iHMM with fixed parameters using the DP prior stabilizes close to 50 states, while for the experiment with learnt hyperparameters the number of states grows more rapidly, reaching 194 states after 8,000 iter­ations. With the PY prior, the number of states reached grows less rapidly reaching 90 states. All runs achieve better performances with respect to all the measures used as the number of iterations grows. An exception is that VI scores tend to in­crease (lower VI scores are better) when the num­ber of states grows larger than the gold standard. It is interesting to notice how the measures exhibit different biases, in particular that VI penalizes the larger numbers of states discovered in the DP run with learnt parameters as well as the run with the PY prior, compared to the more lenient scores pro­vided by V-measure and V-beta. The latter though assigns lower scores to the DP run with learnt pa­rameters because it takes into account that the high homogeneity is achieved using even more states. Finally, the interpretability of these scores presents some interest. For example, in the run with fixed parameters using the DP prior, after burn-in VI was 4.6, which corresponds to 76.65% normalized VI score, while V-measure and V-beta were 12.7% and 9% respectively. In 8,000 iterations after burn-in, VI was 3.94 (80.3% when normalized), while V-measure and V-beta were 53.3%, since the num­ber of states was almost the same as the number of unique PoS tags in the gold standard.</p><p>The closest experiment to ours is the one by Gao &amp; Johnson (2008) who run their Bayesian HMM over the whole WSJ and evaluated against the full gold standard, the only difference being is that we exclude the CoNLL shared task sec-<page local="8" global="685"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">684</doubt><doubt alpha="0.0" length="18" tooSmall="False" monospace="0.0">012345678012345678</doubt><p>Table 1 : Performance of the three iHMM runs according to clustering evaluation measures against num­ber of iteretions (in thousands).</p><p>Table 2: Performance of the output of the three iHMM runs when used in shallow parsing against number of iteretions (in thousands).</p><p>tions from our evaluation, which leaves us with 19 sections instead of 24. Their best VI score was 4.03886 which they achieved using the collapsed, sentence-blocked Gibbs sampler with the number of states fixed to 50. The VI score achieved by the iHMM with fixed parameters using the PY prior reaches 3.73, while using the DP prior VI reaches 4.32 with learnt parameters and 3.93 with fixed parameters.<page local="9" global="686"/> These results, even if they are not directly comparable, are on par with the state-of-the-art, which encouraged us to proceed with the extrinsic evaluation.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">685</doubt><p>For the experiments with shallow parsing we used the CRF++ toolkit<footnote anchor="3"/> which has an efficient implementation of the model introduced by Sha &amp; Pereira (2003) for this task. First we ran an experi­ment using the words and the PoS tags provided in the shared task data and the performances obtained were 96.07% accuracy and 93.81% F-measure. The PoS tags were produced using the Brill tag­ger (Brill, 1994) which employs tranformation-based learning and was trained using the WSJ cor­pus. Then we ran an experiment removing the PoS tags altogether, and the performances were 93.25% accuracy and 88.58% F-measure respec­tively. This gave us some indication as to what the contribution of the PoS tags is in the context of the shallow parsing task at hand.</p><p>The experiments using the output of the iHMM as PoS tags for shallow parsing are presented in Table 2. The best performance achieved was 94.48% and 90.98% in accuracy and F-measure, which is 1.23% and 2.4% better respectively than just using words, but worse by 1.57% and 2.83% compared to using the supervised PoS tagger out­put. Given that the latter is trained on WSJ we be­lieve that this is a good result. Interestingly, this was obtained by using the last sample from the iHMM run using the DP prior with learnt param­eters which has worse overall clustering evalua­tion scores, especially in terms of VI. This sample though has the best homogeneity score (69.39%). We believe that homogeneity is more important than the overall clustering score due to the fact that, in the application considered, it is probably worse to assign tokens that belong to different PoS tags to the same state, e.g. verb and adverbs, rather than generate more than one state for the same PoS. This is likely to be the case in tasks where we are interested in distinguishing between PoS tags rather than the actual tag itself. Also, clus­tering evaluation measures tend to score leniently consistent mixing of members of different classes in the same cluster. However, such mixing results in consistent noise when the clustering output be­comes input to a machine learning method, which is harder to deal with.</p><footnote label="3">http://crfpp . sourceforge.net/</footnote></section><section number="6" title="Conclusions - Future Work"><p>In the context of shallow parsing we saw that the performance of the iHMM does not match the performance of a supervised PoS tagger but does lead to a performance increase over a model us­ing only words as features. Given that it was con­structed without any need for human annotation, we believe this is a good result. At the same time though, it suggests that it is still some way from being a direct drop-in replacement for a supervised method. We argue that the extrinsic evaluation of unsupervised PoS tagging performed in this paper is quite informative as it allowed us to assess our results in a more realistic context. In this work we used shallow parsing for this, but we are consider­ing other tasks in which we hope that PoS tagging performance will be more crucial.</p><p>Our experiments also suggest that the number of states in a Bayesian non-parametric model can be rather unpredictable. On one hand, this is a strong warning towards inference algorithms which per­form finite truncation of non-parametric models. On the other hand, the remarkable difference in behavior between the DP with fixed and learned priors suggests that more research is needed to­wards understanding the influence of hyperparam­eters in Bayesian non-parametric models.</p><p>We are currently experimenting with a semi-supervised PoS tagger where we let the transi­tion matrix of the iHMM depend on annotated PoS tags. This model allows us to: a) use an­notations whenever they are available and do un­supervised learning otherwise; b) use the power of non-parametric methods to possibly learn more fine grained statistical structure than tag sets cre­ated manually.</p><p>On the implementation side, it would be in­teresting to see how our methods scale in a dis­tributed map-reduce architecture where network communication overhead becomes an issue.</p><p>Finally, the ultimate goal of our investigation is to do unsupervised PoS tagging using web-scale datasets. Although the WSJ corpus is reasonably sized, our computational methods do not currently scale to problems with one or two order of magni­tude more data. We will need new breakthroughs to unleash the full potential of unsupervised learn­ing for NLP<page local="10" global="687"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">686</doubt></section><references><p>Charles E. Antoniak. 1974. Mixtures of dirichlet pro­cesses with applications to bayesian nonparametric problems. <i>The Annals of Statistics, </i>2(6): 1152-1174.</p><p>M. J. Beal, Z. Ghahramani, and C. E. Rasmussen. 2002. The infinite hidden markov model. <i>Advances in Neural Information Processing Systems, </i>14:577 -584.</p><p>Chris Biemann, Claudio Giuliano, and Alfio Gliozzo. 2007. Unsupervised part-of-speech tagging support­ing supervised methods. In <i>Proceedings ofRANLP.</i></p><p>Eric Brill. 1994. Some advances in transformation-based part of speech tagging. In <i>National Confer­ence on Artificial Intelligence, </i>pages 722-727.</p><p>Jeffrey Dean and Sanjay Ghemawat. 2004. Mapre-duce: Simplified data processing on large clusters. In <i>Sixth Symposium on Operating System Design and Implementation.</i></p><p>Jenny Rose Finkel, Trond Grenager, and Christo­pher D. Manning. 2007. The infinite tree. In <i>Pro­ceedings of the 45th Annual Meeting of the Associ­ation of Computational Linguistics, </i>pages 272-279, Prague, Czech Republic, June. Association for Com­putational Linguistics.</p><p>J. Van Gael, Y. Saatci, Y. W. Teh, and Z. Ghahra­mani. 2008. Beam sampling for the infinite hidden markov model. In <i>Proceedings of the 25th interna­tional conference on Machine learning, </i>volume 25, Helsinki.</p><p>Jianfeng Gao and Mark Johnson. 2008. A compari­son of bayesian estimators for unsupervised hidden markov model pos taggers. In <i>Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, </i>pages 344-352.</p><p>Sharon Goldwater and Tom Griffiths. 2007. A fully bayesian approach to unsupervised part-of-speech tagging. In <i>Proceedings of the 45th Annual Meet­ing of the Association of Computational Linguistics, </i>pages 744-751, Prague, Czech Republic, June. As­sociation for Computational Linguistics.</p><p>S. Goldwater, T. Griffiths, and M. Johnson. 2006. In­terpolating between types and tokens by estimating power-law generators. <i>Advances in Neural Informa­tion Processing Systems, </i>18.</p><p>Mark Johnson. 2007. Why Doesn't EM Find Good HMM POS-Taggers? In <i>Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, </i>pages 296-305.</p><p>P. Liang, S. Petrov, M. I. Jordan, and D. Klein. 2007. The infinite PCFG using hierarchical Dirichlet pro­cesses. In <i>Proceedings of the 2007 Joint Confer­ence on Empirical Methods in Natural Language Processing and Computational Natural Language Learning.</i></p><p>D. J. C. MacKay. 1997. <i>Ensemble learning for hidden Markov models. </i>Technical report, Cavendish Labo­ratory, University of Cambridge, 1997.</p><p>Marina Meilä. 2007. Comparing clusterings—an in­formation based distance. <i>Journal of Multivariate Analysis, </i>98(5):873-895.</p><p>Andrew Rosenberg and Julia Hirschberg. 2007. V-measure: A conditional entropy-based external clus­ter evaluation measure. In <i>Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, </i>pages 410^420, Prague, Czech Republic, June.</p><p>Steven L. Scott. 2002. Bayesian methods for hidden markov models: Recursive computing in the 21st century. <i>Journal of the American Statistical Asso­ciation, </i>97(457):337-351, March.</p><p>Fei Sha and Fernando Pereira. 2003. Shallow pars­ing with conditional random fields. In <i>Human Lan­guage Technology Conference and the 4th Meeting of the North American Association for Computa­tional Linguistics.</i></p><p>Yee Whye Teh, Michael I. Jordan, Matthew J. Beal, and D. M. Blei. 2006. Hierarchical Dirichlet pro­cesses. <i>Journal of the American Statistical Associa­tion, </i>101(476):1566-1581.</p><p>Erik F. Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the CoNLL-2000 shared task: Chunking. In Claire Cardie, Walter Daelemans, Claire Nedellec, and Erik Tjong Kim Sang, editors, <i>Proceedings of the Fourth Conference on Computa­tional Natural Language Learning, </i>pages 127-132. Lisbon, Portugal, September.</p><p>Andreas Vlachos, Anna Korhonen, and Zoubin Ghahramani. 2009. Unsupervised and Constrained Dirichlet Process Mixture Models for Verb Cluster­ing. In <i>Proceedings of the EACL workshop on GEo-metrical Models of Natural Language Semantics.</i></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">687</doubt></references></body></article>