<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="296"/><title>Why Doesn't EM Find Good HMM POS-Taggers?</title><pubinfo>Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning,pp. 296-305, Prague, June 2007. ©2007 Association for Computational Linguistics</pubinfo><author surname="Johnson" givenname="Mark"><org  name="Microsoft Research" country="USA" city="Redmond"/></author></firstpageheader><frontmatter><p><b>Why doesn't EM find good HMM POS-taggers?</b></p><p><b>Mark Johnson</b></p><p>Microsoft Research Brown University</p><p>Redmond, WA Providence, RI</p><p>t-majoh@microsoft.com Mark Johnson@Brown.edu</p></frontmatter><abstract>This paper investigates why the HMMs es­timated by Expectation-Maximization (EM) produce such poor results as Part-of-Speech (POS) taggers. We find that the HMMs es­timated by EM generally assign a roughly equal number of word tokens to each hid­den state, while the empirical distribution of tokens to POS tags is highly skewed. This motivates a Bayesian approach using a sparse prior to bias the estimator toward such a skewed distribution. We investigate Gibbs Sampling (GS) and Variational Bayes (VB) estimators and show that VB con­verges faster than GS for this task and that VB significantly improves 1-to-1 tagging ac­curacy over EM. We also show that EM does nearly as well as VB when the number of hidden HMM states is dramatically reduced. We also point out the high variance in all of these estimators, and that they require many more iterations to approach conver­gence than usually thought. </abstract></header><body><section number="1" title="Introduction"><p>It is well known that Expectation-Maximization (EM) performs poorly in unsupervised induction of linguistic structure (Carroll and Charniak, 1992; Merialdo, 1994; Klein, 2005; Smith, 2006). In ret­rospect one can certainly find reasons to explain this failure: after all, likelihood does not appear in the wide variety of linguistic tests proposed for identi­fying linguistic structure (Fromkin, 2001).</p><p>This paper focuses on unsupervised part-of-speech (POS) tagging, because it is perhaps the simplest linguistic induction task. We suggest that one reason for the apparent failure of EM for POS tag­ging is that it tends to assign relatively equal num­bers of tokens to each hidden state, while the em­pirical distribution of POS tags is highly skewed, like many linguistic (and non-linguistic) phenomena (Mitzenmacher, 2003). We focus on first-order Hid­den Markov Models (HMMs) in which the hidden state is interpreted as a POS tag, also known as bitag models.</p><p>In this setting we show that EM performs poorly when evaluated using a "1-to-1 accuracy" evalua­tion, where each POS tag corresponds to at most one hidden state, but is more competitive when evaluated using a "many-to-1 accuracy" evaluation, where sev­eral hidden states may correspond to the same POS tag. We explain this by observing that the distribu­tion of hidden states to words proposed by the EM-estimated HMMs is relatively uniform, while the empirical distribution of POS tags is heavily skewed towards a few high-frequency tags. Based on this, we propose a Bayesian prior that biases the sys­tem toward more skewed distributions and show that this raises the 1-to-1 accuracy significantly. Finally, we show that a similar increase in accuracy can be achieved by reducing the number of hidden states in the models estimated by EM.</p><p>There is certainly much useful information that bitag HMMs models cannot capture. Toutanova et al. (2003) describe a wide variety of morphologi­cal and distributional features useful for POS tag­ging, and Clark (2003) proposes ways of incorporat­ing some of these in an unsupervised tagging model. However, bitag models are rich enough to capture at least some distributional information (i.e., the tag for a word depends on the tags assigned to its neigh­bours).<page local="2" global="297"/> Moreover, more complex models add addi­tional complicating factors that interact in ways still poorly understood; for example, smoothing is gen­erally regarded as essential for higher-order HMMs, yet it is not clear how to integrate smoothing into un-supervised estimation procedures (Goodman, 2001; Wang and Schuurmans, 2005).</p><p>Most previous work exploiting unsupervised training data for inferring POS tagging models has focused on semi-supervised methods in the in which the learner is provided with a lexicon specifying the possible tags for each word (Merialdo, 1994; Smith and Eisner, 2005; Goldwater and Griffiths, 2007) or a small number of "prototypes" for each POS (Haghighi and Klein, 2006). In the context of semi-supervised learning using a tag lexicon, Wang and Schuurmans (2005) observe discrepencies between the empirical and estimated tag frequencies similar to those observed here, and show that constraining the estimation procedure to preserve the empirical frequencies improves tagging accuracy. (This ap­proach cannot be used in an unsupervised setting since the empirical tag distribution is not available). However, as Banko and Moore (2004) point out, the accuracy achieved by these unsupervised methods depends strongly on the precise nature of the su­pervised training data (in their case, the ambiguity of the tag lexicon available to the system), which makes it more difficult to understand the behaviour of such systems.</p></section><section number="2" title="Evaluation"><p>All of the experiments described below have the same basic structure: an estimator is used to infer a bitag HMM from the unsupervised training cor­pus (the words of Penn Treebank (PTB) Wall Street Journal corpus (Marcus et al., 1993)), and then the resulting model is used to label each word of that corpus with one of the HMM's hidden states. This section describes how we evaluate how well these sequences of hidden states correspond to the gold­standard POS tags for the training corpus (here, the PTB POS tags). The chief difficulty is determining the correspondence between the hidden states and the gold-standard POS tags.</p><p>Perhaps the most straightforward method of es­tablishing this correspondence is to deterministically map each hidden state to the POS tag it co-occurs most frequently with, and return the proportion of the resulting POS tags that are the same as the POS tags of the gold-standard corpus. We call this the <i>many-to-1 accuracy </i>of the hidden state sequence be­cause several hidden states may map to the same POS tag (and some POS tags may not be mapped to by any hidden states at all).</p><p>As Clark (2003) points out, many-to-1 accuracy has several defects. If a system is permitted to posit an unbounded number of hidden states (which is not the case here) then it can achieve a perfect many-to-1 accuracy by placing every word token into its own unique state. Cross-validation, i.e., identifying the many-to-1 mapping and evaluating on different sub­sets of the data, would answer many of these objec­tions. Haghighi and Klein (2006) propose constrain­ing the mapping from hidden states to POS tags so that at most one hidden state maps to any POS tag. This mapping is found by greedily assigning hidden states to POS tags until either the hidden states or POS tags are exhausted (note that if the number of hidden states and POS tags differ, some will be unas-signed). We call the accuracy of the POS sequence obtained using this map its <i>1-to-1 accuracy.</i></p><p>Finally, several authors have proposed using information-theoretic measures of the divergence between the hidden state and POS tag sequences. Goldwater and Griffiths (2007) propose using the <i>Variation of Information </i>(VI) metric described by Meila (2003). We regard the assignments of hid­den states and POS tags to the words of the cor­pus as two different ways of clustering those words, and evaluate the conditional entropy of each clus­tering conditioned on the other. The VI is the sum of these conditional entropies. Specifically, given a corpus labeled with hidden states and POS tags, if <i>p(y),p(t)</i><i> </i>and p(y, <i>t)</i><i> </i>are the empirical probabilities of a hidden state y, a POS tag t, and the cooccurance of y and t respectively, then the mutual information I, entropies H and variation of information VI are defined as follows:</p><doubt alpha="45.5" length="22" tooSmall="False" monospace="0.0">H (Y) =~Y,P(y)log P(y)</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">y</doubt><doubt alpha="43.5" length="23" tooSmall="False" monospace="0.0">H(T)   =- EP(t)log P(t)</doubt><doubt alpha="50.0" length="6" tooSmall="False" monospace="0.0">P(y,t)</doubt><doubt alpha="50.0" length="8" tooSmall="False" monospace="0.0">P(y)P(t)</doubt><doubt alpha="30.8" length="26" tooSmall="False" monospace="0.0">H(Y|T)   =   H(Y) - I(Y,T)</doubt><doubt alpha="40.0" length="25" tooSmall="False" monospace="0.0">I(Y,T)   =   ]T p(y,t)log</doubt><page local="3" global="298"/><doubt alpha="28.6" length="63" tooSmall="False" monospace="0.0">H(T |Y)   =   H (T) - I(Y,T) VI (Y,T)   =   H (Y |T) + H (T |Y)</doubt><p>As Meila (2003) shows, VI is a metric on the space of probability distributions whose value reflects the divergence between the two distributions, and only takes the value zero when the two distributions are identical.</p></section><section number="3" title="Maximum Likelihood via"><p><b>Expectation-Maximization</b></p><p>There are several excellent textbook presentations of Hidden Markov Models and the Forward-Backward algorithm for Expectation-Maximization (Jelinek, 1997; Manning and Schutze, 1999; Bishop, 2006), so we do not cover them in detail here. Conceptu­ally, a Hidden Markov Model generates a sequence of observations x = (x0,..., xn) (here, the words of the corpus) by first using a Markov model to gen­erate a sequence of hidden states y = (y0,..., yn) (which will be mapped to POS tags during evalua­tion as described above) and then generating each word Xj conditioned on its corresponding state y». We insert endmarkers at the beginning and ending of the corpus and between sentence boundaries, and constrain the estimator to associate endmarkers with a state that never appears with any other observation type (this means each sentence can be processed in­dependently by first-order HMMs; these endmarkers are ignored during evaluation).</p><p>In more detail, the HMM is specified by multi­nomials 0y and 0y for each hidden state y, where 0y specifies the distribution over states following y and 0y specifies the distribution over observations x given state y.</p><doubt alpha="53.8" length="13" tooSmall="False" monospace="0.0">yi-i =y yi =y</doubt><doubt alpha="66.7" length="9" tooSmall="False" monospace="0.0">Multi(0y)</doubt><doubt alpha="54.5" length="11" tooSmall="False" monospace="0.0">Multi((((y)</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(1)</doubt><p>We used the Forward-Backward algorithm to per­form Expectation-Maximization, which is a proce­dure that iteratively re-estimates the model param­eters (0,0), converging on a local maximum of the likelihood. Specifically, if the parameter estimate at time <i>I </i>is &lt;/&gt;(^), then the re-estimated parame­ters at time <i>I </i>+ 1 are:</p><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">V |y</doubt><footnote label="0">x|y</footnote><doubt alpha="48.3" length="29" tooSmall="False" monospace="0.0">= E[rvJ/E[ny]=  E[nx,y]/E[ny]</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(2)</doubt><p>Figure 1: Variation in negative log likelihood with increasing iterations for 10 EM runs from different random starting points.</p><p>where <i>nxy</i><i> </i>is the number of times observation x oc­curs with state y, ny/ y is the number of times state y' follows y and ny is the number of occurences of state y; all expectations are taken with respect to the model (0W,(/&gt;(^).</p><p>We took care to implement this and the other al­gorithms used in this paper efficiently, since optimal performance was often only achieved after several hundred iterations. It is well-known that EM often takes a large number of iterations to converge in like­lihood, and we found this here too, as shown in Fig­ure 1. As that figure makes clear, likelihood is still increasing after several hundred iterations.</p><p>Perhaps more surprisingly, we often found dra­matic changes in accuracy in the order of 5% occur-ing after several hundred iterations, so we ran 1,000 iterations of EM in all of the experiments described here; each run took approximately 2.5 days compu­tation on a 3.6GHz Pentium 4. It's well-known that accuracy often decreases after the first few EM it­erations (which we also observed); however in our experiments we found that performance improves again after 100 iterations and continues improving roughly monotonically. Figure 2 shows how 1-to-1 accuracy varies with iteration during 10 runs from different random starting points. Note that 1-to-1 accuracy at termination ranges from 0.38 to 0.45; a spread of 0.07.</p><p>We obtained a dramatic speedup by working di­rectly with probabilities and rescaling after each ob­servation to avoid underflow, rather than working with log probabilities (thanks to Yoshimasa Tsu-<page local="4" global="299"/></p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>7.15E+06</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b><i>-jj </i></b>7.10E+06</p><p><b>O</b></p><p><b>O </b>.c</p><p>£ 7.05E+06</p><p><b>til</b>l _o</p><p>7.00E+06</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>6.95E+06</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>i                 i                 i </b>i</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>0           250         500         750 1000</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Iteration</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Iteration</p><doubt alpha="37.5" length="8" tooSmall="True" monospace="0.0">(J fD 1_</doubt><doubt alpha="40.0" length="5" tooSmall="True" monospace="0.0">(J (J</doubt><doubt alpha="50.0" length="6" tooSmall="True" monospace="0.0">ru ^ I</doubt><doubt alpha="0.0" length="34" tooSmall="False" monospace="0.0">0.47 0.45 0.43 0.41 0.39 0.37 0.35</doubt><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">■'Mr</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">250</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">500</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">750</doubt><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1000</doubt><p>Figure 2: Variation in 1-to-1 accuracy with increas­ing iterations for 10 EM runs from different random starting points.</p><doubt alpha="25.0" length="4" tooSmall="False" monospace="0.0">2E+5</doubt><doubt alpha="50.0" length="2" tooSmall="True" monospace="0.0">&lt;D</doubt><doubt alpha="25.0" length="4" tooSmall="False" monospace="0.0">1E+5</doubt><doubt alpha="25.0" length="4" tooSmall="False" monospace="0.0">0E+0</doubt><p>PTB VB EM</p><doubt alpha="40.0" length="5" tooSmall="False" monospace="0.0">EM 25</doubt><p>Tag / hidden state (sorted by frequency)</p><p>Figure 3: The average number of words labeled with each hidden state or tag for the EM, VB (with ax = ay = 0.1) and EM-25 estimators (EM-25 is the EM estimator with 25 hidden states).</p><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">0</doubt><p>ruoka for pointing this out). Since we evaluated the accuracy of the estimated tags after each iter­ation, it was important that decoding be done effi­ciently as well. While most researchers use Viterbi decoding to find the most likely state sequence, max­imum marginal decoding (which labels the observa­tion xi with the state <i>yi </i>that maximizes the marginal probability P(yi|x, 0,0)) is faster because it re-uses the forward and backward tables already constructed by the Forward-Backward algorithm. Moreover, in separate experiments we found that the maximum marginal state sequence almost always scored higher than the Viterbi state sequence in all of our evalua­tions, and at modest numbers of iterations (up to 50) often scored more than 5% better.</p><p>We also noticed a wide variance in the perfor­mance of models due to random initialization (both 0 and 0 are initially jittered to break symmetry); this wide variance was observed with all of the estima­tors investigated in this paper. This means we cannot compare estimators on the basis of single runs, so we ran each estimator 10 times from different random starting points and report both mean and standard deviation for all scores.</p><p>Finally, we also experimented with annealing, in which the parameters 0 and ( are raised to the power 1 /T, where T is a "temperature" parameter that is slowly lowered toward 1 at each iteration accord­ing to some "annealing schedule". We experimented with a variety of starting temperatures and annealing schedules (e.g., linear, exponential, etc), but were unable to find any that produced models whose likelihoods were significantly higher (i.e., the models fit better) than those found without annealing.</p><p>The evaluation of the models produced by the EM and other estimators is presented in Table 1. It is difficult to compare these with previous work, but Haghighi and Klein (2006) report that in a completely unsupervised setting, their MRF model, which uses a large set of additional features and a more complex estimation procedure, achieves an av­erage 1-to-1 accuracy of 41.3%. Because they pro­vide no information about the variance in this accu­racy it is difficult to tell whether there is a signifi­cant difference between their estimator and the EM estimator, but it is clear that when EM is run long enough, the performance of even very simple mod­els like the bitag HMM is better than generally rec­ognized.</p><p>As Table 1 makes clear, the EM estimator pro­duces models that are extremely competitive in many-to-1 accuracy and Variation of Information, but are significantly worse in 1-to-1 accuracy. We can understand these results by comparing the dis­tribution of words to hidden states to the distribution of words to POS tags in the gold-standard evaluation corpus. As Figure 3 shows, the distribution of words to POS tags is highly skewed, with just 6 POS tags, NN, IN, NNP, DT, JJ and NNS, accounting for over 55% of the tokens in the corpus. By contrast, the EM distribution is much flatter. This also explains why the many-to-1 accuracy is so much better than the one-to-one accuracy; presumably several hidden<page local="5" global="300"/></p><p>Table 1: Evaluation of models produced by the various estimators. The values of the Dirichlet prior param­eters for ax and ay appear in the estimator name for the VB and GS estimators, and the number of hidden states is given in parentheses. Reported values are means over all runs, followed by standard deviations. 10 runs were performed for each of the EM and VB estimators, while 5 runs were performed for the GS estimators. Each EM and VB run consisted of 1,000 iterations, while each GS run consisted of 50,000 it­erations. For the estimators with 10 runs, a 3-standard error 95% confidence interval is approximately the same as the standard deviation.</p><p>states are being mapped onto a single POS tag. This is also consistent with the fact that the cross-entropy H(T | Y) of tags given hidden states is relatively low (i.e., given a hidden state, the tag is relatively pre­dictable), while the cross-entropy H(Y|T) is rela­tively high.</p><p><b>4   Bayesian estimation via Gibbs Sampling and Variational Bayes</b></p><p>A Bayesian estimator combines a likelihood term P(x|0, 0) and a prior P(0,0) to estimate the poste­rior probability of a model or hidden state sequence. We can use a Bayesian prior to bias our estimator towards models that generate more skewed distri­butions. Because HMMs (and PCFGs) are prod­ucts of multinomials, Dirichlet distributions are a particularly natural choice for the priors since they are conjugate to multinomials, which simplifies both the mathematical and computational aspects of the problem. The precise form of the model we investi­gated is:</p><doubt alpha="44.4" length="18" tooSmall="False" monospace="0.0">0y|   ay~  Dir(ay)</doubt><doubt alpha="47.1" length="17" tooSmall="False" monospace="0.0">0y|   ax~ Dir(ax)</doubt><doubt alpha="45.1" length="51" tooSmall="False" monospace="0.0">yi| yi-i= y ~ Multi(0y) xj|  yi= y     ~  Multi(0y)</doubt><p>Informally, ay controls the sparsity of the state-tostate transition probabilities while ax controls the sparsity of the state-to-observation emission proba­bilities. As ax approaches zero the prior strongly prefers models in which each hidden state emits as few words as possible. This captures the intu­ition that most word types only belong to one POS, since the minimum number of non-zero state-to-observation transitions occurs when each observa­tion type is emitted from only one state. Similarly, as ay approaches zero the state-to-state transitions become sparser.</p><p>There are two main techniques for Bayesian esti­mation of such models: Markov Chain Monte Carlo (MCMC) and Variational Bayes (VB). MCMC en­compasses a broad range of sampling techniques, including component-wise Gibbs sampling, which is the MCMC technique we used here (Robert and techniques do not produce a single model that char­acterizes the posterior, but instead produce a stream of samples from the posterior. The application of MCMC techniques, including Gibbs sampling, to HMM inference problems is relatively well-known: see Besag (2004) for a tutorial introduction and Goldwater and Griffiths (2007) for an application of Gibbs sampling to HMM inference for semisupervised and unsupervised POS tagging.<page local="6" global="301"/></p><doubt alpha="56.5" length="46" tooSmall="False" monospace="0.0">Casella, 2004; Bishop, 2006). In general, MCMC</doubt><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Estimator</p></td><td class="cell"><p>1-to-1</p></td><td class="cell"><p>Many-to-1</p></td><td class="cell"><p>VI</p></td><td class="cell"><p>H (T |Y)</p></td><td class="cell"><p>H (Y |T)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>EM</p></td><td class="cell"><p>(50)</p></td><td class="cell"><p>0.40</p></td><td class="cell"><p>(0.02)</p></td><td class="cell"><p><b>0.62</b></p></td><td class="cell"><p>(0.01)</p></td><td class="cell"><p>4.46</p></td><td class="cell"><p>(0.08)</p></td><td class="cell"><p><b>1.75</b></p></td><td class="cell"><p>(0.04)</p></td><td class="cell"><p>2.71</p></td><td class="cell"><p>(0.06)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>VB(0.1, 0.1)</p></td><td class="cell"><p>(50)</p></td><td class="cell"><p><b>0.47</b></p></td><td class="cell"><p>(0.02)</p></td><td class="cell"><p>0.50</p></td><td class="cell"><p>(0.02)</p></td><td class="cell"><p>4.28</p></td><td class="cell"><p>(0.09)</p></td><td class="cell"><p>2.39</p></td><td class="cell"><p>(0.07)</p></td><td class="cell"><p>1.89</p></td><td class="cell"><p>(0.06)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>VB(0.1,10-4)</p></td><td class="cell"><p>(50)</p></td><td class="cell"><p>0.46</p></td><td class="cell"><p>(0.03)</p></td><td class="cell"><p>0.50</p></td><td class="cell"><p>(0.02)</p></td><td class="cell"><p>4.28</p></td><td class="cell"><p>(0.11)</p></td><td class="cell"><p>2.39</p></td><td class="cell"><p>(0.08)</p></td><td class="cell"><p>1.90</p></td><td class="cell"><p>(0.07)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>vb(io-4, 0.1)</p></td><td class="cell"><p>(50)</p></td><td class="cell"><p>0.42</p></td><td class="cell"><p>(0.02)</p></td><td class="cell"><p>0.60</p></td><td class="cell"><p>(0.01)</p></td><td class="cell"><p>4.63</p></td><td class="cell"><p>(0.07)</p></td><td class="cell"><p>1.86</p></td><td class="cell"><p>(0.03)</p></td><td class="cell"><p>2.77</p></td><td class="cell"><p>(0.05)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>VB(10-4, 10-4)</p></td><td class="cell"><p>(50)</p></td><td class="cell"><p>0.42</p></td><td class="cell"><p>(0.02)</p></td><td class="cell"><p>0.60</p></td><td class="cell"><p>(0.01)</p></td><td class="cell"><p>4.62</p></td><td class="cell"><p>(0.07)</p></td><td class="cell"><p>1.85</p></td><td class="cell"><p>(0.03)</p></td><td class="cell"><p>2.76</p></td><td class="cell"><p>(0.06)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>GS(0.1, 0.1)</p></td><td class="cell"><p>(50)</p></td><td class="cell"><p>0.37</p></td><td class="cell"><p>(0.02)</p></td><td class="cell"><p>0.51</p></td><td class="cell"><p>(0.01)</p></td><td class="cell"><p>5.45</p></td><td class="cell"><p>(0.07)</p></td><td class="cell"><p>2.35</p></td><td class="cell"><p>(0.09)</p></td><td class="cell"><p>3.20</p></td><td class="cell"><p>(0.03)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>GS(0.1, 10-4)</p></td><td class="cell"><p>(50)</p></td><td class="cell"><p>0.38</p></td><td class="cell"><p>(0.01)</p></td><td class="cell"><p>0.51</p></td><td class="cell"><p>(0.01)</p></td><td class="cell"><p>5.47</p></td><td class="cell"><p>(0.04)</p></td><td class="cell"><p>2.26</p></td><td class="cell"><p>(0.03)</p></td><td class="cell"><p>3.22</p></td><td class="cell"><p>(0.01)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>gs(1o-4, 0.1)</p></td><td class="cell"><p>(50)</p></td><td class="cell"><p>0.36</p></td><td class="cell"><p>(0.02)</p></td><td class="cell"><p>0.49</p></td><td class="cell"><p>(0.01)</p></td><td class="cell"><p>5.73</p></td><td class="cell"><p>(0.05)</p></td><td class="cell"><p>2.41</p></td><td class="cell"><p>(0.04)</p></td><td class="cell"><p>3.31</p></td><td class="cell"><p>(0.03)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>GS(10-4,10-4)</p></td><td class="cell"><p>(50)</p></td><td class="cell"><p>0.37</p></td><td class="cell"><p>(0.02)</p></td><td class="cell"><p>0.49</p></td><td class="cell"><p>(0.01)</p></td><td class="cell"><p>5.74</p></td><td class="cell"><p>(0.03)</p></td><td class="cell"><p>2.42</p></td><td class="cell"><p>(0.02)</p></td><td class="cell"><p>3.32</p></td><td class="cell"><p>(0.02)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>EM</p></td><td class="cell"><p>(40)</p></td><td class="cell"><p>0.42</p></td><td class="cell"><p>(0.03)</p></td><td class="cell"><p>0.60</p></td><td class="cell"><p>(0.02)</p></td><td class="cell"><p>4.37</p></td><td class="cell"><p>(0.14)</p></td><td class="cell"><p>1.84</p></td><td class="cell"><p>(0.07)</p></td><td class="cell"><p>2.55</p></td><td class="cell"><p>(0.08)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>EM</p></td><td class="cell"><p>(25)</p></td><td class="cell"><p>0.46</p></td><td class="cell"><p>(0.03)</p></td><td class="cell"><p>0.56</p></td><td class="cell"><p>(0.02)</p></td><td class="cell"><p><b>4.23</b></p></td><td class="cell"><p>(0.17)</p></td><td class="cell"><p>2.05</p></td><td class="cell"><p>(0.09)</p></td><td class="cell"><p>2.19</p></td><td class="cell"><p>(0.08)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>EM</p></td><td class="cell"><p>(10)</p></td><td class="cell"><p>0.41</p></td><td class="cell"><p>(0.01)</p></td><td class="cell"><p>0.43</p></td><td class="cell"><p>(0.01)</p></td><td class="cell"><p>4.32</p></td><td class="cell"><p>(0.04)</p></td><td class="cell"><p>2.74</p></td><td class="cell"><p>(0.03)</p></td><td class="cell"><p><b>1.58</b></p></td><td class="cell"><p>(0.05)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>The Gibbs sampler produces state sequences y sampled from the posterior distribution:</p><doubt alpha="42.2" length="45" tooSmall="False" monospace="0.0">P(y|x,a) ocIP(x, y|0, &lt;&lt;&gt;)P(0|ay)P(&lt;/&gt;Md0dcf)</doubt><p>Because Dirichlet priors are conjugate to multino­mials, it is possible to integrate out the model pa­rameters 0 and &lt;&lt;f to yield the conditional distribu­tion for yi shown in Figure 4. For each observation xi in turn, we resample its state <i>yi </i>conditioned on the states y-i of the other observations; eventually the distribution of state sequences converges to the desired posterior.</p><p>Each iteration of the Gibbs sampler is much faster than the Forward-Backward algorithm (both take time linear in the length of the string, but for an HMM with s hidden states, each iteration of the Gibbs sampler takes O(s) time while each iteration of the Forward-Backward algorithm takes O(s<footnote anchor="2"/>) time), so we ran 50,000 iterations of all samplers (which takes roughly the same elapsed time as 1,000 Forward-Backward iterations).</p><p>As can be seen from Table 1, the posterior state sequences we obtained are not particularly good. Further, when we examined how the posterior like­lihoods varied with increasing iterations of Gibbs sampling, it became apparent that the likelihood was still increasing after 50,000 iterations. Moreover, when comparing posterior likelihoods from differ­ent runs with the same prior parameters but differ­ent random number seeds, none of the likelihoods crossed, which one would expect if the samplers had converged and were mixing well (Robert and Casella, 2004). Just as with EM, we experimented with a variety of annealing regimes, but were unable to find any which significantly improved accuracy or posterior likelihood.</p><p>We also experimented with evaluating state se­quences found using maximum posterior decoding (i.e., model parameters are estimated from the pos­terior sample, and used to perform maximum poste­rior decoding) rather than the samples from the pos­terior produced by the Gibbs sampler. We found that the maximum posterior decoding sequences usually scored higher than the posterior samples, but the scores converged after the first thousand iterations. Since the posterior samples are produced as a by­product of Gibbs sampling while maximum posterior decoding requires an additional time consuming step that does not have much impact on scores, we used the posterior samples to produce the results in</p><p>Table 1.</p><p>In contrast to MCMC, Variational Bayesian in­ference attempts to find the function Q(y, 0, &lt;f) that minimizes an upper bound of the negative log likeli­hood (Jordan et al., 1999):</p><doubt alpha="50.0" length="10" tooSmall="False" monospace="0.0">- log P(x)</doubt><doubt alpha="52.0" length="25" tooSmall="False" monospace="0.0">=-log/Q(y,0,f)^yo0^dyd0df</doubt><doubt alpha="48.8" length="41" tooSmall="False" monospace="0.0">&lt;-JQ(y,0,ff) logP(QXyy00f)&lt;f)dy d0 d&lt;f(3)</doubt><p>The upper bound in (3) is called the <i>Variational Free Energy. </i>We make a "mean-field" assumption that the posterior can be well approximated by a factor-ized model Q in which the state sequence y does not covary with the model parameters 0, &lt;f (this will be true if, for example, there is sufficient data that the posterior distribution has a peaked mode):</p><doubt alpha="29.5" length="44" tooSmall="False" monospace="0.0">P(x, y,0,&lt;f)   w   Q(y,0,&lt;f) = Qi(y)Q2(0,&lt;f)</doubt><p>The calculus of variations is used to minimize the KL divergence between the desired posterior distri­bution and the factorized approximation. It turns out that if the likelihood and conjugate prior be­long to exponential families then the optimal Q<b>i </b>and Q<b>2 </b>do too, and there is an EM-like iterative pro­cedure that finds locally-optimal model parameters (Bishop, 2006).</p><p>This procedure is especially attractive for HMM inference, since it involves only a minor modifica­tion to the M-step of the Forward-Backward algo­rithm. MacKay (1997) and Beal (2003) describe Variational Bayesian (VB) inference for HMMs in detail, and Kurihara and Sato (2006) describe VB for PCFGs (which only involves a minor modifica­tion to the M-step of the Inside-Outside algorithm). Specifically, the E-step for VB inference for HMMs is the same as in EM, while the M-step is as follows:</p><doubt alpha="50.0" length="6" tooSmall="False" monospace="0.0">y'\y\y</doubt><doubt alpha="37.8" length="37" tooSmall="False" monospace="0.0">=  f (E[ny',y]+ay)/f (E[ny]+ say) (4)</doubt><doubt alpha="22.0" length="82" tooSmall="False" monospace="0.0">tp(v)=(v&gt;7)?g(v -2) : (V&gt;(v + 1) - 1)/v g(x)   w   log(x) + 0.04167X-2+ 0.00729x-4</doubt><doubt alpha="6.9" length="29" tooSmall="False" monospace="0.0">+0.00384x"6- 0.00413x-8...(5)</doubt><doubt alpha="41.2" length="34" tooSmall="False" monospace="0.0">=  f (E[nx,y] + ax)/f (E[ny] +max)</doubt><doubt alpha="33.3" length="18" tooSmall="False" monospace="0.0">f(v)   = exp(^(v))</doubt><page local="7" global="302"/><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">0</doubt><doubt alpha="57.1" length="14" tooSmall="False" monospace="0.0">P(yi|x,y-i,a)o</doubt><doubt alpha="60.7" length="28" tooSmall="False" monospace="0.0">(nxi,yi +axA/nyi,yi-i +ayA (</doubt><doubt alpha="60.0" length="30" tooSmall="False" monospace="0.0">nyi+\nyi-i+say7 V+I(yi-i =yi)J</doubt><p>Figure 4: The conditional distribution for state yi used in the Gibbs sampler, which conditions on the states y-i for all observations <i>except </i>xi. Here m is the number of possible observations (i.e., the size of the vocabulary), s is the number of hidden states and I( ) is the indicator function (i.e., equal to one if its argument is true and zero otherwise), n^y is the number of times observation x occurs with state y, ny^y is the number of times state y' follows y, and ny is the number of times state y occurs; these counts are from ( x- i, y- i) , i.e., excluding xi and yi.</p><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">2</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0 1 2</doubt><p>Figure 5: The scaling function y = f(x) = exp -0(x) (curved line), which is bounded above by the line y = x and below by the line y = x - 0.5.</p><p>where ^ is the <i>digamma function </i>(the derivative of the log gamma function; (5) gives an asymptotic ap­proximation), and the remaining quantities are just as in the EM updates (2), i.e., is the number of times observation x occurs with state y, ny' y is the number of times state y' follows y, ny is the number of occurences of state y, s is the number of hidden states and m is the number of observations; all ex­pectations are taken with respect to the variational parameters (0 (^), &lt;&lt;&lt;(^)).</p><p>A comparison between (4) and (2) reveals two dif­ferences between the EM and VB updates. First, the Dirichlet prior parameters a are added to the expected counts. Second, these posterior counts (which are in fact parameters of the Dirichlet pos­terior Q2) are passed through the function f (v) = exp ^(v), which is plotted in Figure 5. When v » 0, f (v) w v - 0.5, so roughly speaking, VB for multinomials involves adding a-0.5 to the expected counts when they are much larger than zero, where a is the Dirichlet prior parameter. Thus VB can be viewed as a more principled version of the well-known ad hoc technique for approximating Bayesian estimation with EM that involves adding a -1 to the expected counts. However, in the ad hoc approach the expected count plus a - 1 may be less than zero, resulting in a value of zero for the corresponding pa­rameter (Johnson et al., 2007; Goldwater and Grif­fiths, 2007). VB avoids this problem because f (v) is always positive when v &gt; 0, even when v is small. Note that because the counts are passed through f, the updated values for 0 and &lt; in (4) are in general <i>not </i>normalized; this is because the variational free energy is only an upper bound on the negative log likelihood (Beal, 2003).</p><p>We found that in general VB performed much bet­ter than GS. Computationally it is very similar to EM, and each iteration takes essentially the same time as an EM iteration. Again, we experimented with annealing in the hope of speeding convergence, but could not find an annealing schedule that signifi­cantly lowered the variational free energy (the quan­tity that VB optimizes). While we had hoped that the Bayesian prior would bias VB toward a common so­lution, we found the same sensitivity to initial condi­tions as we found with EM, so just as for EM, we ran the estimator for 1,000 iterations with 10 different random initializations for each combination of prior parameters. Table 1 presents the results of VB runs with several different values for the Dirichlet prior parameters. Interestingly, we obtained our best per­formance on 1-to-1 accuracy when the Dirchlet prior ax = 0.1, a relatively large number, but best per­formance on many-to-1 accuracy was achieved with a much lower value for the Dirichlet prior, namely ax = 10-4. The Dirichlet prior ay that controls sparsity of the state-to-state transitions had little ef­fect on the results.<page local="8" global="303"/> We did not have computational resources to fully explore other values for the prior (a set of 10 runs for one set of parameter values takes 25 computer days).</p><doubt alpha="53.8" length="13" tooSmall="False" monospace="0.0">P(ylxya)^   l</doubt><doubt alpha="58.6" length="29" tooSmall="False" monospace="0.0">nyi+i,yi+I(yi-i =yi=yi+i)+ay'</doubt><p>As Figure 3 shows, VB can produce distributions of hidden states that are peaked in the same way that POS tags are. In fact, with the priors used here, VB produces state sequences in which only a subset of the possible HMM states are in fact assigned to ob­servations. This shows that rather than fixing the number of hidden states in advance, the Bayesian prior can determine the number of states; this idea is more fully developed in the <i>infinite HMM </i>of Beal et al. (2002) and Teh et al. (2006).</p></section><section number="5" title="Reducing the number of hidden states"><p>EM already performs well in terms of the many-to-1 accuracy, but we wondered if there might be some way to improve its 1-to-1 accuracy and VI score. In section 3 we suggested that one reason for its poor performance in these evaluations is that the distri­butions of hidden states it finds tend to be fairly flat, compared to the empirical distribution of POS tags. As section 4 showed, a suitable Bayesian prior can bias the estimator towards more peaked distribu­tions, but we wondered if there might be a simpler way of achieving the same result.</p><p>We experimented with dramatic reductions in the number of hidden states in the HMMs estimated by EM. This should force the hidden states to be more densely populated and improve 1-to-1 accu­racy, even though this means that there will be no hidden states that can possibly map onto the less fre­quent POS tags (i.e., we will get these words wrong). In effect, we abandon the low-frequency POS tags in the hope of improving the 1-to-1 accuracy of the high-frequency tags.</p><p>As Table 1 shows, this markedly improves both the 1-to-1 accuracy and the VI score. A 25-state HMM estimated by EM performs effectively as well as the best VB model in terms of both 1-to-1 accu­racy and VI score, and runs 4 times faster because it has only half the number of hidden states.</p></section><section number="6" title="Conclusion and future work"><p>This paper studied why EM seems to do so badly in HMM estimation for unsupervised POS tagging. In fact, we found that it doesn't do so badly at all: the bitag HMM estimated by EM achieves a mean 1-to-1 tagging accuracy of 40%, which is approximately the same as the 41.3% reported by (Haghighi and Klein, 2006) for their sophisticated MRF model.</p><p>Then we noted the distribution of words to hidden states found by EM is relatively uniform, compared to the distribution of words to POS tags in the eval­uation corpus. This provides an explanation of why the many-to-1 accuracy of EM is so high while the 1-to-1 accuracy and VI of EM is comparatively low. We showed that either by using a suitable Bayesian prior or by simply reducing the number of hidden states it is possible to significantly improve both the 1-to-1 accuracy and the VI score, achieving a 1-to-1 tagging accuracy of 46%.</p><p>We also showed that EM and other estimators take much longer to converge than usually thought, and often require several hundred iterations to achieve optimal performance. We also found that there is considerable variance in the performance of all of these estimators, so in general multiple runs from different random starting points are necessary in or­der to evaluate an estimator's performance.</p><p>Finally, there may be more sophisticated ways of improving the 1-to-1 accuracy and VI score than the relatively crude methods used here that primar­ily reduce the number of available states. For ex­ample, we might obtain better performance by us­ing EM to infer an HMM with a large number of states, and then using some kind of distributional clustering to group similar HMM states; these clus­ters, rather than the underlying states, would be in­terpreted as the POS tag labels. Also, the Bayesian framework permits a wide variety of different priors besides Dirichlet priors explored here. For example, it should be possible to encode linguistic knowledge such markedness preferences in a prior, and there are other linguistically uninformative priors, such the "entropic priors" of Brand (1999), that may be worth exploring.</p></section><section title="Acknowledgements"><p>I would like to thank Microsoft Research for pro­viding an excellent environment in which to con­duct this work, and my friends and colleagues at Microsoft Research, especially Bob Moore, Chris Quirk and Kristina Toutanova, for their helpful com­ments on this paper.</p><page local="9" global="304"/></section><references><p>Michele Banko and Robert C. Moore. 2004. Part of speech tagging in context. In <i>Proceedings, 20th In­ternational Conference on Computational Linguistics (Coling 2004), </i>pages 556-561, Geneva, Switzerland.</p><p>M.J. Beal, Z. Ghahramani, and C.E. Rasmussen. 2002. The infinite Hidden Markov Model. In T. Dietterich, S. Becker, and Z. Ghahramani, editors, <i>Advances in Neural Information Processing Systems, </i>volume 14, pages 577-584. The MIT Press.</p><p>Matthew J. Beal. 2003. <i>Variational Algorithms for Ap­proximate Bayesian Inference. </i>Ph.D. thesis, Gatsby Computational Neuroscience unit, University College London.</p><p>Julian Besag. 2004. An introduction to Markov Chain Monte Carlo methods. In Mark Johnson, Sanjeev P. Khudanpur, Mari Ostendorf, and Roni Rosenfeld, ed­itors, <i>Mathematical Foundations ofSpeech and Lan­guage Processing, </i>pages 247-270. Springer, New York.</p><p>Christopher M. Bishop. 2006. <i>Pattern Recognition and Machine Learning. </i>Springer.</p><p>M. Brand. 1999. An entropic estimator for structure dis­covery. <i>Advances in Neural Information Processing Systems, </i>11:723-729.</p><p>Glenn Carroll and Eugene Charniak. 1992. Two experi­ments on learning probabilistic dependency grammars from corpora. In <i>Proceedings ofthe AAAI Workshop on Statistically-Based Natural Language Processing Techniques, </i>San Jose, CA.</p><p>Alexander Clark. 2003. Combining distributional and morphological information for part of speech induc­tion. In <i>10th Conference of the European Chapter of the Association for Computational Linguistics, </i>pages 59-66. Association for Computational Linguistics.</p><p>Victoria Fromkin, editor. 2001. <i>Linguistics: An Intro­duction to Linguistic Theory. </i>Blackwell, Oxford, UK.</p><p>Sharon Goldwater and Tom Griffiths. 2007. A fully Bayesian approach to unsupervised part-of-speech tag­ging. In <i>Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics.</i></p><p>Joshua Goodman. 2001. A bit of progress in language modeling. <i>Computer Speech and Language, </i>14:403­434.</p><p>Aria Haghighi and Dan Klein. 2006. Prototype-driven learning for sequence models. In <i>Proceedings of the Human Language Technology Conference </i><i>ofthe</i> <i>NAACL, Main Conference, </i>pages 320-327, New York City, USA, June. Association for Computational Lin­guistics.</p><p>Frederick Jelinek. 1997. <i>Statistical Methods for Speech Recognition. </i>The MIT Press, Cambridge, Mas­sachusetts.</p><p>Mark Johnson, Tom Griffiths, and Sharon Goldwater. 2007. Bayesian inference for PCFGs via Markov chain Monte Carlo. In <i>Human Language Technologies 2007: The Conference of the North American Chap­ter </i><i>ofthe</i><i> Association for Computational Linguistics; Proceedings of the Main Conference, </i>pages 139-146, Rochester, New York. Association for Computational Linguistics.</p><p>Michael I. Jordan, Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Sau. 1999. An introduc­tion to variational methods for graphical models. <i>Ma­chine Learning, </i>37(2):183-233.</p><p>Dan Klein. 2005. <i>The Unsupervised Learning of Natural Language Structure. </i>Ph.D. thesis, Stanford Univer­sity.</p><p>Kenichi Kurihara and Taisuke Sato. 2006. Variational Bayesian grammar induction for natural language. In <i>8th International Colloquium on Grammatical Infer­ence.</i></p><p>David J.C. MacKay. 1997. Ensemble learning for hidden Markov models. Technical report, Cavendish Labora­tory, Cambridge.</p><p>Chris Manning and Hinrich Schutze. 1999. <i>Foundations ofStatistical Natural Language Processing. </i>The MIT Press, Cambridge, Massachusetts.</p><p>Michell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated cor­pus of English: The Penn Treebank. <i>Computational Linguistics, </i>19(2):313-330.</p><p>Marina Meila. 2003. Comparing clusterings by the vari­ation of information. In Bernhard Scholkopf and Man­fred K. Warmuth, editors, <i>COLT 2003: The Sixteenth Annual Conference on Learning Theory, </i>volume 2777 of <i>Lecture Notes in Computer Science, </i>pages 173-187. Springer.</p><p>Bernard Merialdo. 1994. Tagging English text with a probabilistic model. <i>Computational Linguistics, </i>20:155-171.</p><p>M. Mitzenmacher. 2003. A brief history of generative models for power law and lognormal distributions. <i>In­ternet Mathematics, </i>1(2):226-251.</p><p>Christian P. Robert and George Casella. 2004. <i>Monte Carlo Statistical Methods. </i>Springer.</p><p>Noah A. Smith and Jason Eisner. 2005. Contrastive estimation: Training log-linear models on unlabeled data. In <i>Proceedings of the 43rd Annual Meeting of the</i> <i>Association for Computational Linguistics (ACL'05), </i>pages 354-362, Ann Arbor, Michigan, June.<page local="10" global="305"/> Associa­tion for Computational Linguistics.</p><p>Noah A. Smith. 2006. <i>Novel Estimation Methods for Unsupervised Discovery of Latent Structure in Natu­ral Language Text. </i>Ph.D. thesis, Johns Hopkins Uni­versity.</p><p>Y. W. Teh, M. I. Jordan, M. J. Beal, and D. M. Blei. 2006. Hierarchical Dirichlet processes. <i>Journal of the Amer­ican Statistical Association, </i>101(476):1566-1581.</p><p>Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In <i>Pro­ceedings of the 2003 Human Language Technology Conference ofthe North American Chapter ofthe As­sociation for Computational Linguistics, </i>pages 252­259.</p><p>Qin Iris Wang and Dale Schuurmans. 2005. Improved estimation for unsupervised part-of-speech tagging. In <i>Proceedings ofthe 2005 IEEE International Confer­ence on Natural Language Processing and Knowledge Engineering (IEEE NLP-KE'2005), </i>pages 219-224, Wuhan, China.</p></references></body></article>