<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="1308"/><title>Bayesian Learning of Phrasal Tree-to-String Templates</title><author surname="Liu" givenname="Ding"><org  name="University of Rochester" country="USA" city="Rochester"/></author><author surname="Gildea" givenname="Daniel"><org  name="University of Rochester" country="USA" city="Rochester"/></author></firstpageheader><frontmatter><p><b>Bayesian Learning of Phrasal Tree-to-String Templates</b></p><p><b>Ding Liu </b>and <b>Daniel Gildea</b></p><p>Department of Computer Science University of Rochester Rochester, NY 14627 {dliu,  gildea}@cs.rochester.edu</p></frontmatter><abstract>We examine the problem of overcoming noisy word-level alignments when learn­ing tree-to-string translation rules. Our approach introduces new rules, and re-estimates rule probabilities using EM. The major obstacles to this approach are the very reasons that word-alignments are used for rule extraction: the huge space of possible rules, as well as controlling overfitting. By carefully controlling which portions of the original alignments are re­analyzed, and by using Bayesian infer­ence during re-analysis, we show signifi­cant improvement over the baseline rules extracted from word-level alignments. </abstract></header><body><section number="1" title="Introduction"><p>Non-parametric Bayesian methods have been suc­cessfully applied to directly learn phrase pairs from a bilingual corpus with little or no depen­dence on word alignments (Blunsom et al., 2008; DeNero et al., 2008). Because such approaches di­rectly learn a generative model over phrase pairs, they are theoretically preferable to the standard heuristics for extracting the phrase pairs from the many-to-one word-level alignments produced by the IBM series models (Brown et al., 1993) or the Hidden Markov Model (HMM) (Vogel et al., 1996). We wish to apply this direct, Bayesian ap­proach to learn better translation rules for syntax-based statistical MT (SSMT), by which we specif­ically refer to MT systems using Tree-to-String (TTS) translation templates derived from syntax trees (Liu et al., 2006; Huang et al., 2006; Gal­ley et al., 2006; May and Knight, 2007), as op­posed to <i>formally syntactic </i>systems such as Hi-ero (Chiang, 2007). The stumbling block pre­venting us from taking this approach is the ex­tremely large space of possible TTS templates when no word alignments are given. Given a sen­tence pair and syntax tree over one side, there are an exponential number of potential TTS tem­plates and a polynomial number of phrase pairs. In this paper, we explore methods for restricting the space of possible TTS templates under con­sideration, while still allowing good templates to emerge directly from the data as much as possible. We find an improvement in translation accuracy through, first, using constraints to limit the number of new templates, second, using Bayesian methods to limit which of these new templates are favored when re-analyzing the training data with EM, and, third, experimenting with different renormaliza-tion techniques for the EM re-analysis.</p><p>We introduce two constraints to limit the num­ber of TTS templates that we extract directly from tree/string pairs without using word alignments. The first constraint is to limit direct TTS tem­plate extraction to the part of the corpus where word alignment tools such as GIZA++ do poorly. There is no reason not to re-use the good align­ments from GIZA++, which holds a very compet­itive baseline performance. As already mentioned, the noisy alignments from GIZA++ are likely to cross the boundaries of the tree constituents, which leads to comparatively big TTS templates. We use this fact as a heuristic to roughly distin­guish noisy from good word alignments.<footnote anchor="1"/> Here we define <i>big templates </i>as those with more than 8 symbols in their right hand sides (RHSs). The word alignments in big templates are considered to be noisy and will be recomposed by extracting smaller TTS templates. Another reason to do ex­traction on big templates is that the applicability of big templates to new sentences is very limited due to their size, and the portion of the training data from which they are extracted is effectively wasted. The second constraint, after choosing the</p><footnote label="1">Precisely differentiating the noisy/good word alignments is as hard as correctly aligning the words.</footnote><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1308</doubt><p><i>Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, </i>pages 1308-1317, Singapore, 6-7 August 2009. ©2009 ACL and AFNLP<page local="2" global="1309"/></p><p>extraction site, is to extract the TTS templates all the way down to the leaves of the hosting tem­plates. This constraint limits the number of possi­ble left hand sides (LHSs) to be equal to the num­ber of tree nodes in the hosting templates. The entire extraction process can be summarized in 3 steps:</p><p>1. Compute word alignments using GIZA++, and generate the basic TTS templates.</p><p>2. Select big templates from the basic TTS tem­plates in step 1, and extract smaller TTS tem­plates all the way down to the bottom from big templates, without considering the pre-computed word alignments.</p><p>3. Combine TTS templates from step 1 and step 2 and estimate their probabilities using Vari­ational Bayes with a Dirichlet Process prior.</p><p>In step 2, since there are no constraints from the pre-computed word alignments, we have complete freedom in generating all possible TTS templates to overcome noisy word alignments. We use vari­ational EM to approximate the inference of our Bayesian model and explore different normaliza­tion methods for the TTS templates. A two-stage normalization is proposed by combining LHS-based normalization with normalization based on the root of the LHS, and is shown to be the best model when used with variational EM.</p><p>Galley et al. (2006) recompose the TTS tem­plates by inserting unaligned target words and combining small templates into bigger ones. The recomposed templates are then re-estimated using the EM algorithm described in Graehl and Knight (2004). This approach also generates TTS tem­plates beyond the precomputed word alignments, but the freedom is only granted over unaligned tar­get words, and most of the pre-computed word alignments remain unchanged. Other prior ap­proaches towards improving TTS templates fo­cus on improving the word alignment performance over the classic models such as IBM series mod­els and Hidden Markov Model (HMM), which do not consider the syntactic structure of the align­ing languages and produce syntax-violating align­ments. DeNero and Klein (2007) use a syntax-based distance in an HMM word alignment model to favor syntax-friendly alignments. Fossum et al. (2008) start from the GIZA++ alignment and in­crementally delete bad links based on a discrim-</p><doubt alpha="37.5" length="8" tooSmall="False" monospace="0.0">' Tip" '</doubt><doubt alpha="11.1" length="9" tooSmall="False" monospace="0.0">■ - - w -</doubt><doubt alpha="50.0" length="6" tooSmall="False" monospace="0.0">"ÄÜX"'</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">Bg</doubt><p>Figure 1: 5 small TTS templates are extracted based on the correct word alignments (left), but only 1 big TTS template (right) can be extracted when the cross-boundary noisy align­ments are added in.</p><p>inative model with syntactic features. This ap­proach can only find a better subset of the GIZA++ alignment and requires a parallel corpus with gold-standard word alignment for training the discrim­inative model. May and Knight (2007) factorize the word alignment into a set of re-orderings rep­resented by the TTS templates and build a hierar­chical syntax-based word alignment model. The problem is that the TTS templates are generated by the word alignments from GIZA++, which lim­its the potential of the syntactic re-alignment. As shown by these prior approaches, directly improv­ing the word alignment either falls into the frame­work of many-to-one alignment, or is substantially confined by the word alignment it builds upon.</p><p>The remainder of the paper focuses on the Bayesian approach to learning TTS templates and is organized as follows: Section 2 describes the procedure for generating the candidate TTS tem­plates; Section 3 describes the inference methods used to learn the TTS templates; Section 4 gives the empirical results, Section 5 discusses the char­acteristics of the learned TTS templates, and Sec­tion 6 presents the conclusion.</p></section><section number="2" title="Extracting Phrasal TTS Templates"><p>The Tree-to-String (TTS) template, the most im­portant component of a SSMT system, usually contains three parts: a fragment of a syntax tree in its left hand side (LHS), a sequence of words and variables in its right hand side (RHS), and a probability indicating how likely the template is to be used in translation. The RHS of a TTS template shows one possible translation and re­ordering of its LHS. The variables in a TTS tem­plate are further transformed using other TTS tem­plates, and the recursive process continues until there are no variables left. There are two ways<page local="3" global="1310"/></p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1309</doubt><p>Table 1 : Percentage of corpus used to generate big templates, based on different word alignments</p><p>in NP3of 2</p><doubt alpha="40.0" length="5" tooSmall="False" monospace="0.0">NP,61</doubt><p>Figure 2: Examples of valid and invalid templates extracted from a big Template. Template 1, invalid, doesn't go all the way down to the bottom. Template 2 is valid. Template 3, in­valid, doesn't have the same set of variables in its LHS/RHS. Template 4, invalid, is not a phrasal TTS template.</p><p>that TTS templates are commonly used in ma­chine translation. The first is <i>synchronous pars­ing </i>(Galley et al., 2006; May and Knight, 2007), where TTS templates are used to construct syn­chronous parse trees for an input sentence, and the translations will be generated once the syn­chronous trees are built up. The other way is the TTS <i>transducer </i>(Liu et al., 2006; Huang et al., 2006), where TTS templates are used just as their name indicates: to transform a source parse tree (or forest) into the proper target string. Since synchronous parsing considers all possible syn­chronous parse trees of the source sentence, it is less constrained than TTS transducers and hence requires more computational power. In this paper, we use a TTS transducer to test the performance of different TTS templates, but our techniques could also be applied to SSMT systems based on syn­chronous parsing.</p><subsection number="2.1" title="Baseline Approach: TTS Templates Obeying Word Alignment"><p>TTS templates are commonly generated by de­composing a pair of aligned source syntax tree and target string into smaller pairs of tree frag­ments and target string (i.e., the TTS templates). To keep the number of TTS templates to a manage­able scale, only the non-decomposable TTS tem­plates are generated. This algorithm is referred to as GHKM (Galley et al., 2004) and is widely used in SSMT systems (Galley et al., 2006; Liu et al., 2006; Huang et al., 2006). The word alignment used in GHKM is usually computed independent of the syntactic structure, and as DeNero and Klein (2007) and May and Knight (2007) have noted,</p><p>Table 2: In the selected big templates, the distribution of words in the templates of different sizes, which are measured based on the number of symbols in their RHS s</p><p>is not the best for SSMT systems. In fact, noisy word alignments cause more damage to a SSMT system than to a phrase based SMT system, be­cause the TTS templates can only be derived from tree constituents. If some noisy alignments happen to cross over the boundaries of two constituents, as shown in Figure 2, a much bigger tree frag­ment will be extracted as a TTS template. Even though the big TTS templates still carry the orig­inal alignment information, they have much less chance of getting matched beyond the syntax tree where they were extracted, as we show in Sec­tion 4. In other words, a few cross-boundary noisy alignments could disable a big portion of a training syntax tree, while for a phrase-based SMT system, their effect is limited to the phrases they align. As a rough measure of how the training corpus is af­fected by the big templates, we calculated the dis­tribution of target words in big and non-big TTS templates. The word alignment is computed using GIZA++<footnote anchor="2"/> for the selected 73,597 sentence pairs in the FBIS corpus in both directions and then com­bined using union and heuristic diagonal growing (Koehn et al., 2003). Table 1 shows that big templates consume 20.1% to 45.9% of the training corpus depending on different types of word align­ments. The statistics indicate that a significant portion of the training corpus is simply wasted, if the TTS templates are extracted based on word alignments from GIZA++. On the other hand, it shows the potential for improving an SSMT sys­tem if we can efficiently re-use the wasted train­ing corpus. By further examining the selected big templates, we find that the most common form of big templates is a big skeleton template starting</p><p>GIZA++ is available at http://www.fjoch.com/GIZA++.html<page local="4" global="1311"/></p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1310</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Ch-En</p></td><td class="cell"><p>En-Ch</p></td><td class="cell"><p>Union</p></td><td class="cell"><p>Heuristic</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>28.6%</p></td><td class="cell"><p>33.0%</p></td><td class="cell"><p>45.9%</p></td><td class="cell"><p>20.1%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>AUX</p><p><b>1</b></p></td><td class="cell"><p>j ADJ</p></td><td class="cell"><p></p></td><td class="cell"><p>9-12</p></td><td class="cell"><p>13-20</p></td><td class="cell"><p>&gt;21</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>is</p></td><td class="cell"><p>Ibeautiful</p></td><td class="cell"><p>Ch-En</p></td><td class="cell"><p>18.2%</p></td><td class="cell"><p>17.4%</p></td><td class="cell"><p>64.4%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>j</b></p></td><td class="cell"><p>En-Ch</p></td><td class="cell"><p>15.9%</p></td><td class="cell"><p>20.7%</p></td><td class="cell"><p>63.4%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3</p></td><td class="cell"><p><b>i <footnote anchor="4"/></b></p></td><td class="cell"><p>Union</p></td><td class="cell"><p>9.8%</p></td><td class="cell"><p>15.1%</p></td><td class="cell"><p>75.1%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NPj</p></td><td class="cell"><p></p></td><td class="cell"><p>Heuristic</p></td><td class="cell"><p>24.6%</p></td><td class="cell"><p>27.9%</p></td><td class="cell"><p>47.5%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>from the root of the source syntax tree, and hav­ing many terminals (words) misaligned in the bot­tom. Table 2 shows, in the selected big templates, the distribution of words in the templates of differ­ent sizes (measured based on the number of sym­bols in their RHS). We can see that based on ei­ther type of word alignment, the most common big templates are the TTS templates with more than 20 symbols in their RHSs, which are gen­erally the big skeleton templates. The advantage of such big skeleton templates is that they usually have good marginal accuracy<footnote anchor="3"/> and allow accurate smaller TTS templates to emerge.</p></subsection><subsection number="2.2" title="Liberating Phrasal TTS Templates From Noisy Word Alignments"><p>To generate better TTS templates, we use a more direct way than modifying the underlying word alignment: extract smaller phrasal TTS tem­plates from the big templates without looking at their pre-computed word alignments. We define <i>phrasal </i>TTS templates as those with more than one symbol (word or non-terminal) in their LHS. The reason to consider only phrasal TTS tem­plates is that they are more robust than the word-level TTS templates in addressing the complicated word alignments involved in big templates, which are usually not the simple type of one-to-many or many-to-one. Abandoning the pre-computed word alignments in big templates, an extracted smaller TTS template can have many possible RHSs, as long as the two sides have the same set of vari­ables. Note that the freedom is only given to the alignments of the words; for the variables in the big templates, we respect the pre-computed word alignments. To keep the extracted smaller TTS templates to a manageable scale, the following two constraints are applied:</p><p>1. The LHS of extracted TTS templates should go all the way down to the bottom of the LHS of the big templates. This constraint ensures that at most <i>N </i>LHSs can be extracted from one big Template, where <i>N </i>is the number of tree nodes in the big Template's LHS.</p><p>2. The number of leaves (including both words and variables) in an extracted TTS template's LHS should not exceed 6. This constraint limits the size of the extracted TTS templates.</p><footnote label="3">Here, marginal accuracy means the correctness of the TTS template's RHS corresponding to its LHS.</footnote><doubt alpha="61.9" length="113" tooSmall="False" monospace="0.0">(VP(AUXis)(ADJ beautiful))—► ffiit(PP(INof) NP3) —► NP3(NPNP^PPONof) NP3))—►NP3NP,(NPNP^PPONof) NP3))—►NP3NP,ffiH</doubt><figure caption="Figure 3: All valid templates that can be extracted from the example in Figure 2.1"></figure><p><b>for all </b>template <i>t </i><b>do if </b>size(t.rhs) &gt; 8 <b>then</b></p><p><b>for all </b>tree node <i>s </i>in t.lhs <b>do </b><i>subt = </i>subtree(s, t.lhs); <b>if </b>leaf_num(s-ii&amp;i) &lt; 6 <b>then for </b>i=l:size(t.rhs) <b>do for </b>j=i:size(t.rhs) <b>do</b> <b>if </b>valid(swW, i, j) <b>then</b></p><doubt alpha="66.7" length="30" tooSmall="False" monospace="0.0">create_template(s-ii&amp;i, i, j);</doubt><figure caption="Figure 4: Algorithm that liberates smaller TTS Templates from big templates"></figure><p>As we show in Section 4, use of bigger TTS templates brings very limited performance gain.</p><p>Figure 2.2 describes the template liberating algo­rithm running in <i>0(NM<footnote anchor="2"/>),</i><i> </i>where <i>N</i><i> </i>denotes the number of tree nodes in the LHS of the input big Template and <i>M </i>denotes the length of the RHS. In the algorithm, function <i>valid </i>returns <i>true </i>if there are the same set of variables in the left/right hand side of an extracted TTS template; <i>subtree(x, </i><i>y)</i><i> </i>denotes the sub-tree in <i>y </i>which is rooted at <i>x </i>and goes all the way down to <i>y's </i>bottom. Figure 2.1 shows valid and invalid TTS templates which can be extracted from an example hosting TTS tem­plate. Note that, in order to keep the example simple, the hosting TTS template only has 4 sym­bols in its RHS, which does not qualify as a big template according to our definition. Figure 2.2 shows the complete set of valid TTS templates which can be extracted from the example TTS template. The subscripts of the non-terminals are used to differentiate identical non-terminals in dif­ferent positions. The extraction process blindly releases smaller TTS templates from the big tem­plates, among which only a small fraction are cor­rect TTS templates. Therefore, we need an infer­ence method to raise the weight of the correct tem­plates and decrease the weight of the noisy tem­plates.</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1311</doubt><page local="5" global="1312"/></subsection></section><section number="3" title="Estimating TTS Template Probability"><p>The Expectation-Maximization (EM) algorithm (Dempster et al., 1977) can be used to estimate the TTS templates' probabilities, given a genera­tive model addressing how a pair of source syn­tax tree and target string is generated. There are two commonly used generative models for syntax-based MT systems, each of which corresponds to a normalization method for the TTS templates. The LHS-based normalization (LHSN) (Liu et al., 2006; Huang et al., 2006), corresponds to the generative process where the source syntax sub­tree is first generated, and then the target string is generated given the source syntax subtree. The other one is normalization based on the root of the LHS (ROOTN) (Galley et al., 2006), corre­sponding to the generative process where, given the root of the syntax subtree, the LHS syntax sub­tree and the RHS string are generated simultane­ously. By omitting the decomposition probability in the LHS-based generative model, the two gen­erative models share the same formula for comput­ing the probability of a training instance:</p><doubt alpha="46.9" length="32" tooSmall="False" monospace="0.0">Pr{T, S) =J2MT, S,R) =J2( IIPr^J</doubt><p>where <i>T </i>and <i>S </i>denote the source syntax tree and target string respectively, <i>R </i>denotes the decompo­sition of (T, <i>S)</i>, and <i>t </i>denotes the TTS template. The expected counts of the TTS templates can then be efficiently computed using an inside-outside-like dynamic programming algorithm (May and Knight, 2007).</p><p>LHSN, as shown by Galley et al. (2006), cannot accurately restore the true conditional probabili­ties of the target sentences given the source sen­tences in the training corpus. This indicates that LHSN is not good at predicting unseen sentences or at translating new sentences. But this deficiency does not affect its ability to estimate the expected counts of the TTS templates, because the posteri­ors of the TTS templates only depend on the com­parative probabilities of the different derivations of a training instance (a pair of tree and string). In fact, as we show in Section 4, LHSN is bet­ter than ROOTN in liberating smaller TTS tem­plates out of the big templates, since it is less bi­ased to the big templates in the EM training.<footnote anchor="4"/> Be­cause the two normalization methods have their</p><footnote label="4">Based on LHSN, the difference between the probabil­ity of a big Template and the product of the probabilities of</footnote><doubt alpha="71.4" length="7" tooSmall="False" monospace="0.0">E-step:</doubt><p><b>for all </b>pair of syntax tree <i>T </i>and target string <i>S </i><b>do for all </b>TTS Template <i>t </i><b>do</b></p><doubt alpha="59.5" length="37" tooSmall="False" monospace="0.0">EC(t)+ -Efi;pr(T&gt;s&gt;fl/)/i .Increaseß;</doubt><doubt alpha="71.4" length="7" tooSmall="False" monospace="0.0">M-step:</doubt><p><b>for all </b>TTS Template <i>t </i><b>do if </b>it is the last iteration <b>then</b></p><doubt alpha="46.7" length="15" tooSmall="False" monospace="0.0">Pr(t) -_ëÇiî)_•</doubt><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">else</doubt><doubt alpha="30.0" length="10" tooSmall="False" monospace="0.0">Pr(t) -__•</doubt><figure caption="Figure 5: EM Algorithm For Estimating TTS Templates"></figure><p>own strength and weakness, both of them are used in our EM algorithm: LHSN is used in all EM iterations except the last one to compute the ex­pected counts of the TTS templates, and ROOTN is used in the last EM iteration to compute the final probabilities of the TTS templates. This two-stage normalization method is denoted as MIXN in this paper.</p><p>Deterministic Annealing (Rose et al., 1992) is is used in our system to speed up the training process, similar to Goldwater et al. (2006). We start from a high temperature and gradually de­crease the temperature to 1; we find that the ini­tial high temperature can also help small templates to survive the initial iterations. The complete EM framework is sketched in Figure 3, where <i>ß </i>is the inverse of the specified temperature, and <i>EC </i>de­notes the expected count.</p><subsection number="3.1" title="Bayesian Inference with the Dirichlet Process Prior"><p>Bayesian inference plus the Dirichlet Process (DP) have been shown to effectively prevent MT mod­els from overfitting the training data (DeNero et al., 2008; Blunsom et al., 2008). A similar ap­proach can be applied here for SSMT by consider­ing each TTS template as a cluster, and using DP to adjust the number of TTS templates according to the training data. Note that even though there is a size limitation on the liberated phrasal TTS templates, standard EM will still tend to overfit the training data by pushing up the probabilities of the big templates from the noisy word alignments. The complete generative process, integrating the DP prior and the generative models described in</p><p>its decomposing TTS templates is much less than the one based on ROOTN, thus LHSN gives comparably more ex­pected counts to the smaller TTS templates than ROOTN.</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1312</doubt><page local="6" global="1313"/><p><b>for all </b>TTS Template <i>t </i><b>do if </b>it is the last iteration <b>then</b></p><p><i>pr(f) </i><b>= </b><b>_</b><b><u>exp(^(gC(t) + aG0(t)))</u></b><b>_</b><b>.</b></p><doubt alpha="46.2" length="39" tooSmall="False" monospace="0.0">^ IeXp(*((£t,;t,root=troot£C(tO) + «))'</doubt><doubt alpha="42.9" length="28" tooSmall="False" monospace="0.0">pr(f\=exp(^(gC(t) + aG0(t)))</doubt><doubt alpha="44.4" length="36" tooSmall="False" monospace="0.0">^ &gt;exp(*((Et,:t,Iks=tJklEC(i'))+«))'</doubt><figure caption="Figure 6: M-step of the Variational EM"></figure><p>Section 3.1, is given below:</p><doubt alpha="42.3" length="26" tooSmall="False" monospace="0.0">6r\{ar,Gr0}   ~ DP(ar,Gr0)</doubt><doubt alpha="56.5" length="23" tooSmall="False" monospace="0.0">^ jßt.root    ^ ßt.root</doubt><doubt alpha="37.9" length="29" tooSmall="False" monospace="0.0">(T,S)I{SG, {t},9}~ SG({t},tf)</doubt><p>where Go is a base distribution of the TTS tem­plates, <i>t </i>denotes a TTS template, #*-ro°* denotes the multinomial distribution over TTS templates with the same root as <i>t, SG </i>denotes the generative model for a pair of tree and string in Section 3.1, and a is a free parameter which adjusts the rate at which new TTS templates are generated.</p><p>It is intractable to do exact inference under the Bayesian framework, even with a conjugate prior such as DP. Two methods are commonly used for approximate inference: Markov chain Monte Carlo (MCMC) (DeNero et al., 2008), and Vari­ational Bayesian (VB) inference (Blunsom et al., 2008). In this paper, the latter approach is used be­cause it requires less running time. The E-step of VB is exactly the same as standard EM, and in the M-step the digamma function <i>tp </i>and the base dis­tribution Go are used to increase the uncertainty of the model. Similar to standard EM, both LHS- and root-based normalizations are used in the M-step, as shown in Figure 3.1. For the TTS templates, which are also pairs of subtrees and strings, a natu­ral choice of Go is the generative models described in Section 3.1. Because Go estimates the probabil­ity of the new TTS templates, the root-based gen­erative model is superior to the LHS-based gener­ative model and used in our approach.</p></subsection><subsection number="3.2" title="Initialization"><p>Since the EM algorithm only converges to a lo­cal minimum, proper initializations are needed to achieve good performance for both standard EM and variational EM. For the baseline templates derived from word alignments, the initial counts are set to the raw counts in the training corpus. For the templates blindly extracted from big tem­plates, the raw count of a LHS tree fragment is distributed among their RHSs based on the like­lihood of the template, computed by combining</p><p><b>for all </b>big template <i>t </i><b>do</b> <b>for all </b>template <i>g </i>extracted from <i>t </i><b>do</b></p><p><i>g. count = g.lhs. count = </i>0; <b>for all </b>template <i>g </i>extracted from <i>t </i><b>do</b></p><doubt alpha="51.5" length="33" tooSmall="False" monospace="0.0">g.count+= w_in((jr)xw_out((7, i);</doubt><p><i>g.lhs.count </i>+= w_m(g)xw_out((7, i); <b>for all </b>template <i>g </i>extracted from <i>t </i><b>do</b></p><doubt alpha="52.9" length="17" tooSmall="False" monospace="0.0">g.init+=f.c°"nt,;</doubt><doubt alpha="52.9" length="17" tooSmall="True" monospace="0.0">::&gt;g .Ins .count1</doubt><figure caption="Figure 7: Compute the initial counts of the liberated TTS templates"></figure><p>the word-based inside/outside scores. The algo­rithm is sketched in Figure 3.2, where the inside score <i>wJn(g)</i><i> </i>is the product of the IBM Model 1 scores in both directions, computed based on the words in <i>g's</i><i> </i>LHS and RHS. The outside score <i>W-OUt(g, </i><i>t)</i><i> </i>is computed similarly, except that the IBM Model 1 scores are computed based on the words in the hosting template i's LHS/RHS ex­cluding the words in <i>g's</i><i> </i>LHS/RHS. The initial probabilities of the TTS templates are then com­puted by normalizing their initial counts using LHSN or ROOTN.</p></subsection></section><section number="4" title="Experiments"><p>We train an English-to-Chinese translation sys­tem using the FBIS corpus, where 73,597 sentence pairs are selected as the training data, and 500 sen­tence pairs with no more than 25 words on the Chi­nese side are selected for both the development and test data.<footnote anchor="5"/> Charniak (2000)'s parser, trained on the Penn Treebank, is used to generate the En­glish syntax trees. Modified Kneser-Ney trigram models are trained using SRILM (Stolcke, 2002) upon the Chinese portion of the training data. The trigram language model, as well as the TTS tem­plates generated based on different methods, are used in the TTS transducer. The model weights of the transducer are tuned based on the develop­ment set using a grid-based line search, and the translation results are evaluated based on a single Chinese reference<footnote anchor="6"/> using BLEU-4 (Papineni et al., 2002). Huang et al. (2006) used character-based BLEU as a way of normalizing inconsistent Chi­nese word segmentation, but we avoid this prob­lem as the training, development, and test data are from the same source.</p><footnote label="5">The total 74,597 sentence pairs used in experiments are those in the FBIS corpus whose English part can be parsed using Charniak (2000)'s parser.</footnote><footnote label="6">BLEU-4 scores based on a single reference are much lower than the ones based on multiple references.</footnote><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1313</doubt><page local="7" global="1314"/><p>Table 4: BLEU-4 scores (test set) of the <i>union </i>alignment, us­ing TTS templates up to a certain size, in terms of the number of leaves in their LHS s</p><subsection number="4.1" title="Baseline Systems"><p>GHKM (Galley et al., 2004) is used to generate the baseline TTS templates based on the word alignments computed using GIZA++ and different combination methods, including <i>union </i>and the di­agonal growing heuristic (Koehn et al., 2003). We also tried combining alignments from GIZA++ based on intersection, but it is worse than both single-direction alignments, due to its low cover­age of training corpus and the incomplete transla­tions it generates. The baseline translation results based on ROOTN are shown in Table 4.1. The first two columns in the table show the results of the two single direction alignments. <i>e2c </i>and <i>c2e </i>de­note the many English words to one Chinese word alignment and the many Chinese words to one En­glish word alignment, respectively. The two rows show the results with and without the big tem­plates, from which we can see that removing the big templates does not affect performance much; this verifies our postulate that the big templates have very little chance of being used in the trans­lation. Table 4.1, using the <i>union </i>alignments as the representative and measuring a template's size by the number of leaves in its LHS, also demon­strates that using big TTS templates brings very limited performance gain.</p><p>The result that the union-based combination outperforms either single direction alignments and even the heuristic-based combination, combined with the statistics of the disabled corpus in Sec­tion 2.2, shows that more disabled training cor­pus actually leads to better performance. This can be explained by the fact that the <i>union </i>alignments have the largest number of noisy alignments gath­ered together in the big templates, and thus have the least amount of noisy alignments which lead to small and low-quality TTS templates.</p><doubt alpha="22.1" length="95" tooSmall="False" monospace="0.0">temperature parameter ß 0.1     0.3     0.5     0.7     0.8     0.9     1.0     1.0     1.0 1.0</doubt><doubt alpha="0.0" length="11" tooSmall="False" monospace="0.0">12345678910</doubt><p>iteration</p><figure caption="Figure 8: BLEU-4 scores (development set) of annealing EM and annealing VB in each iteration."></figure></subsection><subsection number="4.2" title="Learning Phrasal TTS Templates"><p>To test our learning methods, we start with the TTS templates generated based on <i>e2c, c2e, </i>and <i>union </i>alignments using GHKM. This gives us 0.98M baseline templates. We use the big tem­plates from the <i>union </i>alignments as the basis and extract 10.92M new phrasal TTS templates, which, for convenience, are denoted by NEW-PHR. Because based on Table 1 and Table 2 the <i>union </i>alignment has the greatest number of alignment links and therefore produces the largest rules, this gives us the greatest flexibility in re­aligning the input sentences. The baseline TTS templates as well as NEW-PHR are initialized us­ing the method in Section 3.3 for both annealing EM and annealing VB. To simplify the experi­ments, the same Dirichlet Process prior is used for all multinomial distributions of the TTS templates with different roots. Go in the Dirichlet prior is computed based on the 1-level TTS templates se­lected from the baseline TTS templates, so that the big templates are efficiently penalized. The train­ing algorithms follow the same annealing sched­ule, where the temperature parameter <i>ß </i>is initial­ized to 0.1, and gradually increased to 1.</p><p>We experiment with the two training algo­rithms, annealing EM and annealing VB, with dif­ferent normalization methods. The experimental results based on the development data are shown in Figure 4.2, where the free parameter <i>a </i>of an­nealing VB is set to 1, 100, and 100 respec­tively for ROOTN, LHSN, and MIXN. The re­sults verify that LHSN is worse than ROOTN in predicting the translations, since MIXN outper­forms LHSN with both annealing EM and VB. ROOTN is on par with MIXN and much better<page local="8" global="1315"/></p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1314</doubt><table caption="Table 3: BLEU-4 scores (test set) of systems based on GIZA++ word alignments" class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>E2C</p></td><td class="cell"><p>C2E</p></td><td class="cell"><p>Union</p></td><td class="cell"><p>Heuristic</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>w/ Big</p></td><td class="cell"><p>13.37</p></td><td class="cell"><p>12.66</p></td><td class="cell"><p><b>14.55</b></p></td><td class="cell"><p>14.28</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>w/o Big</p></td><td class="cell"><p>13.20</p></td><td class="cell"><p>12.62</p></td><td class="cell"><p>14.53</p></td><td class="cell"><p>14.21</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>i</b></p></td><td class="cell"><p><b>i</b></p></td><td class="cell"><p><b>1</b></p></td><td class="cell"><p><b>1</b></p></td><td class="cell"><p><footnote anchor="1"/></p></td><td class="cell"><p><b>1 1</b></p></td><td class="cell"><p><b>i</b></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>-</b></p></td><td class="cell"><p>p&lt;</p></td><td class="cell"><p></p></td><td class="cell"><p><b>—»—</b></p><p><b>„.-©---</b></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p><b>_____©--"""</b></p><p><i>]^£^=^^</i></p></td><td class="cell"><p><b>-M-—--</b></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>-</b></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p><b>--£]...</b></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>o"'</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>J</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>h //</p></td><td class="cell"><p><i>x'</i></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p><b>.....*.......*.....</b></p></td><td class="cell"><p><b>-*......</b></p></td><td class="cell"><p><b>-"</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>y y</i></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>y</i></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>MIXN-EM LHSN-VB LHSN-EM ROOTN-EM ROOTN-VB MIXN-VB</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>r*"</p></td><td class="cell"><p><i>M </i>■</p></td><td class="cell"><p></p></td><td class="cell"><p><b>1</b></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p><b>—X—</b></p><p>......€3......</p><p><b>--0--</b><b></b>i</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>&lt; 5</p></td><td class="cell"><p>&lt; 6      &lt; 7</p></td><td class="cell"><p>&lt; 8     &lt; oo</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>BLEU-4</p></td><td class="cell"><p>14.27</p></td><td class="cell"><p>14.42 14.43</p></td><td class="cell"><p>14.45 14.55</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Max Likelihood</p><p>Annealing EM</p><p>Annealing VB</p><p>Table 6: The total number of templates and the percentage of NEW-PHR, in the beginning and end of annealing VB</p><p>than LHSN when annealing EM is used; but with annealing VB, it is outperformed by MIXN by a large margin and is even slightly worse than LHSN. This indicates that ROOTN is not giv­ing large expected counts to NEW-PHR and leaves very little space for VB to further improve the re­sults. For all the normalization methods, anneal­ing VB outperforms annealing EM and maintains a longer ascending path, showing better control of overfitting for the Bayesian models. Figure 4.2 shows the optimized results of the development set based on annealing VB with different <i>a. </i>The best performance is achieved as <i>a </i>approaches 1, 100, and 100 for ROOTN, LHSN and MIXN re­spectively. The <i>a </i>parameter can be viewed as a weight used to balance the expected counts and the probabilities from Go. Thus it is reasonable for LHSN and MIXN to have bigger optimal <i>a </i>than ROOTN, since ROOTN gives lower expected counts to NEW-PHR than LHSN and MIXN do.</p><p>To see the contribution of the phrasal template extraction in the performance gain, MT experi­ments are conducted by turning this component on and off. Results on the test set, obtained by using parameters optimized on the development set, are shown in Table 4.2. The template counts used in the Max-Likelihood training are the same as the ones used in the initialization of anneal­ing EM and VB. Results show that for annealing EM and VB, use of NEW-PHR greatly improves performance, while for the Max-Likelihood train­ing, use of NEW-PHR hurts performance. This is not surprising, because Max-Likelihood train­ing cannot efficiently filter out the noisy phrasal templates introduced in the initial NEW-PHR. An­other observation is that annealing VB does not al­ways outperform annealing EM. With NEW-PHR turned on, annealing VB shows consistent supe­riority over annealing EM; while without NEW-PHR, it only outperforms annealing EM based on LHSN and MIXN, and the improvement is not as big as when NEW-PHR is turned on. This indi­cates that without NEW-PHR, there is less need to use VB to shrink down the size of the tem­plate set. Table 4.2 shows the statistics of the ini­tial template set including NEW-PHR and the final TTS template set after annealing VB is conducted, where we can see annealing VB efficiently re­duces NEW-PHR to a relatively small size and re­sults in much more compact systems than the sys­tem based on the baseline templates from GIZA++ alignments. Comparing with the best GIZA++-based system <i>union, </i>our best system, utilizing NEW-PHR and the two-stage template normaliza­tion, demonstrates the strength of annealing VB by an absolute improvement of 2.29% in BLEU4 score, from 14.55 to 16.84. This improvement is significant at <i>p &lt; </i>0.005 based on 2000 itera­tions of paired bootstrap re-sampling of the test set (Koehn, 2004).</p><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">19. £</doubt><doubt alpha="55.6" length="27" tooSmall="False" monospace="0.0">MIXN —h-ROOTN—x-LHSN ---*--</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">0.1</doubt><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">10</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">100</doubt><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1000</doubt><figure caption="Figure 9: BLEU-4 scores (development set) of annealing VB with differenta."></figure></subsection></section><section number="5" title="Discussion"><p>Our experimental results are obtained based on a relatively small training corpus, the improved performance may be questionable when a larger training corpus is used. Someone may wonder if the performance gain primarily comes from the reduced out of vocabulary (OOV) ratio.<page local="9" global="1316"/> We ex­amined the OOV ratio of the test set with/without the learned TTS templates, and found the differ­ence was very small. In fact, our method is de­signed to learn the phrasal TTS templates, and ex-plictly avoids lexical pairs. To further understand the characteristics of the learned TTS templates, we list some representative templates in Figure 4.2 classified in 3 groups. The group <i>Many-to-one Alignment </i>and <i>Many-to-many Alignment </i>show the TTS templates based on complicated word align­ments, which are difficult to compute based on the existing word alignment models. These templates do not have rare English words, whose translation cannot be found outside the big templates. The difficulty lies in the non-literal translation of the source words, which are unlikely to learnt by soly increasing the size of the training corpus. One other interesting observation is that our learning method is tolerant to noisy Chinese word segmen­tation, as shown in group <i>Allowance of Bad Word Segmentation.</i></p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1315</doubt><table caption="Table 5: BLEU-4 scores (test set) of different systems." class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>w/o new-phr</p></td><td class="cell"><p>with new-phr</p></td><td class="cell"><p>w/o new-phr</p></td><td class="cell"><p>with new-phr</p></td><td class="cell"><p>w/o new-phr</p></td><td class="cell"><p>with new-phr</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>LHSN</p></td><td class="cell"><p>14.05</p></td><td class="cell"><p>13.16</p></td><td class="cell"><p>14.31</p></td><td class="cell"><p>15.33</p></td><td class="cell"><p>14.82</p></td><td class="cell"><p>16.15</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ROOTN</p></td><td class="cell"><p>14.50</p></td><td class="cell"><p>13.49</p></td><td class="cell"><p>14.90</p></td><td class="cell"><p>16.06</p></td><td class="cell"><p>14.76</p></td><td class="cell"><p>16.12</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MIXN</p></td><td class="cell"><p>NA</p></td><td class="cell"><p>NA</p></td><td class="cell"><p>14.82</p></td><td class="cell"><p>16.37</p></td><td class="cell"><p>14.93</p></td><td class="cell"><p><b>16.84</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Initial Template number new-phr%</p></td><td class="cell"><p>Final Template number new-phr%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ROOTN LHSN MIXN</p></td><td class="cell"><p>11.9M 91.8% 11.9M 91.8% 11.9M 91.8%</p></td><td class="cell"><p>408.0K 21.9% 557.2K 29.8% 500.5K 27.6%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><section number="6" title="Conclusion"><p>This paper proposes a Bayesian model for extract­ing the Tree-to-String templates directly from the data. By limiting the extraction to the big tem­plates from the pre-computed word alignments and applying a set of constraints, we restrict the space of possible TTS templates under consider­ation, while still allowing new and more accurate templates to emerge from the training data. The empirical results demonstrate the strength of our approach, which outperforms the GIZA++-based systems by a large margin. This encourages a move from word-alignment-based systems to sys­tems based on consistent, end-to-end probabilistic modeling. Because our Bayesian model employs a very simple prior, more sophisticated generative models provide a possible direction for further ex­perimentation.</p><p><b>Acknowledgments </b>This work was supported by NSF grants IIS-0546554 and ITR-0428020.</p><table caption="Figure 10: Examples of the learned TTS templates" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Many-to-one Alignment</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>( vp ( vb make ) ( np ( dt a ) ( jj complete ) ( nn statement ) ) )</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>( s ( vp vbg ( np ( dt the ) ( nn mass ) ( nn line ) ) pp ) )</p></td><td class="cell"><p>ppvbg <i>mxim</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>( pp ( to to ) ( np ( dt the ) ( jjs greatest ) ( nn extent ) ) )</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>( pp ( in of) (np ( jj peaceful ) ( nnp coexistence ) ) )</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Many-to-many Alignment</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>( vp ( vbn based ) ( pp ( in on ) ( np ( jj actual ) ( nns needs ) ) ) )</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>( pp ( in into ) ( np ( np ( dt the ) ( nns hands ))pp))</p></td><td class="cell"><p>M ft pp ¥±</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>( vp ( vbp exercise ) ( np ( jj strict ) ( nn self-discipline ) ) )</p></td><td class="cell"><p><i>r u w </i>a</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>( sb AR ( s ( np ( dt the ) ( vbg aging ) nn ) ( vp ( aux is ) np ) ) )</p></td><td class="cell"><p><b>NN </b><i>% m it % </i>np</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>( np NPi pp (,, ) ( vp ( vbn centered ) ( pp ( in around ) np2 ) ) )</p></td><td class="cell"><p><i>u </i>np2 <i>%     </i>#j NPi pp</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Allowance of Bad Word Segmentation</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(np (np (nnpjapan ) ( pos 's ) ) (nnp sdf ) (nnpnavy ) )</p></td><td class="cell"><p>b« <i>± </i>gm</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>( np ( pdt all ) ( np ( nns people ) ( pos 's ) ) ( nns organizations ) )</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><references><p>Phil Blunsom, Trevor Cohn, and Miles Osborne. 2008. Bayesian synchronous grammar induction. In <i>Neu­ral Information Processing Systems (NIPS).</i></p><p>Peter F. Brown, Stephen A. Delia Pietra, Vincent j. Delia Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Pa­rameter estimation. <i>Computational Linguistics, </i>19(2):263-311.</p><p>Eugene Charniak. 2000. A maximum-entropy-inspired parser. In <i>Proceedings ofNAACL-01, </i>pages 132-139.</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1316</doubt><page local="10" global="1317"/><p>David Chiang. 2007. Hierarchical phrase-based trans­lation. <i>Computational Linguistics, </i>33(2).</p><p>A. P. Dempster, N. M. Laird, and D. B. Rubin. 1977. Maximum likelihood from incomplete data via the <i>EM </i>algorithm. <i>Journal of the Royal Statistical Soci­ety, </i>39(1):1-21.</p><p>John DeNero and Dan Klein. 2007. Tailoring word alignments to syntactic machine translation. In <i>Pro­ceedings ofACL-07, </i>pages 17-24.</p><p>John DeNero, Alexandre Bouchard-Cote, and Dan Klein. 2008. Sampling alignment structure under a bayesian translation model. In <i>EMNLP08.</i></p><p>Victoria Fossum, Kevin Knight, and Steven Abney. 2008. Using syntax to improveword alignment pre­cision for syntax-based machine translation. In <i>Pro­ceedings of the Third Workshop on Statistical Ma­chine Translation, </i>Columbus, Ohio. ACL.</p><p>Michel Galley, Mark Hopkins, Kevin Knight, and Daniel Marcu. 2004. What's in a translation rule? In <i>Proceedings ofNAACL-04, </i>pages 273-280.</p><p>Michel Galley, Jonathan Graehl, Kevin Knight, Daniel Marcu, Steve DeNeefe, Wei Wang, and Ignacio Thayer. 2006. Scalable inference and training of context-rich syntactic translation models. In <i>Pro­ceedings ofCOLING/ACL-06, </i>pages 961-968, July.</p><p>Sharon Goldwater, Thomas L. Griffiths, and Mark Johnson. 2006. Contextual dependencies in un­supervised word segmentation. In <i>Proceedings of the Human Language Technology Conference/North American Chapter of the Association for Computa­tional Linguistics (HLT/NAACL).</i></p><p>Jonathan Graehl and Kevin Knight. 2004. Training tree transducers. In <i>Proceedings ofNAACL-04.</i></p><p>Liang Huang, Kevin Knight, and Aravind Joshi. 2006. Statistical syntax-directed translation with extended domain of locality. In <i>Proceedings of the 7th Bi­ennial Conference of the Association for Machine Translation in the Americas (AMTA), </i>Boston, MA.</p><p>Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In <i>Pro­ceedings ofNAACL-03, </i>Edmonton, Alberta.</p><p>Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In <i>Proceedings of EMNLP, </i>pages 388-395, Barcelona, Spain, July.</p><p>Yang Liu, Qun Liu, and Shouxun Lin. 2006. Tree-to-string alignment template for statistical machine translation. In <i>Proceedings of COLING/ACL-06, </i>Sydney, Australia, July.</p><p>J. May and K. Knight. 2007. Syntactic re-alignment models for machine translation. In <i>Proceedings of EMNLP.</i></p><p>Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In <i>Proceedings ofACL-02.</i></p><p>K. Rose, E. Gurewitz, and G. C. Fox. 1992. Vec­tor quantization by deterministic annealing. <i>IEEE Transactions on Information Theory, </i>38(4): 1249-1257, July.</p><p>Andreas Stolcke. 2002. Srilm - an extensible language modeling toolkit. In <i>International Conference on Spoken Language Processing, </i>volume 2, pages 901-904.</p><p>Stephan Vogel, Hermann Ney, and Christoph Tillmann. 1996. Hmm-based word alignment in statistical translation. In <i>COLING-96, </i>pages 836-841.</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1317</doubt></references></body></article>