<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="352"/><title>A Bayesian Model of Syntax-Directed Tree to String Grammar Induction</title><author surname="Cohn" givenname="Trevor"><org  name="University of Edinburgh" country="United Kingdom" city="Edinburgh"/></author><author surname="Blunsom" givenname="Phil"><org  name="University of Edinburgh" country="United Kingdom" city="Edinburgh"/></author></firstpageheader><frontmatter><p><b>A Bayesian Model of Syntax-Directed Tree to String Grammar Induction</b></p><p><b>Trevor Cohn </b>and <b>Phil Blunsom</b></p><p>School of Informatics University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB Scotland, United Kingdom</p><p>{tcohn,pblunsom}@inf.ed.ac.uk</p></frontmatter><abstract>Tree based translation models are a com­pelling means of integrating linguistic in­formation into machine translation. Syn­tax can inform lexical selection and re­ordering choices and thereby improve translation quality. Research to date has focussed primarily on decoding with such models, but less on the difficult problem of inducing the bilingual grammar from data. We propose a generative Bayesian model of tree-to-string translation which induces grammars that are both smaller and pro­duce better translations than the previous heuristic two-stage approach which em­ploys a separate word alignment step. </abstract></header><body><section number="1" title="Introduction"><p>Many recent advances in statistical machine trans­lation (SMT) are a result of the incorporation of syntactic knowledge into the translation process (Marcu et al., 2006; Zollmann and Venugopal, 2006). This has been facilitated by the use of syn­chronous grammars to model translation as a gen­erative process over pairs of strings in two lan­guages. Such models are particularly attractive for translating between languages with divergent word orders, such as Chinese and English, where syntax-inspired translation rules can succinctly de­scribe the requisite reordering operations. In con­trast, standard phrase-based models (Koehn et al., 2003) assume a mostly monotone mapping be­tween source and target, and therefore cannot adequately model these phenomena. Currently the most successful paradigm for the use of syn­chronous grammars in translation is that of string-to-tree transduction (Galley et al., 2004; Zollmann and Venugopal, 2006; Galley et al., 2006; Marcu et al., 2006). In this case a grammar is extracted from a parallel corpus, with strings on its source side and syntax trees on its target side, which is then used to translate novel sentences by perform­ing inference over the space of target syntax trees licensed by the grammar.</p><p>To date grammar-based translation models have relied on heuristics to extract a grammar from a word-aligned parallel corpus. These heuristics are extensions of those developed for phrase-based models (Koehn et al., 2003), and involve sym­metrising two directional word alignments fol­lowed by a projection step which uses the align­ments to find a mapping between source words and nodes in the target parse trees (Galley et al., 2004). However, such approaches leave much to be de­sired. Word-alignments rarely factorise cleanly with parse trees (i.e., alignment points cross con­stituent structures), resulting in large and implau­sible translation rules which generalise poorly to unseen data (Fossum et al., 2008). The principal reason for employing a grammar based formal­ism is to induce rules which capture long-range reorderings between source and target. However if the grammar itself is extracted using word-alignments induced with models that are unable to capture such reorderings, it is unlikely that the grammar will live up to expectations.</p><p>In this work we draw on recent advances in Bayesian modelling of grammar induction (lohn-son et al., 2007; Cohn et al., 2009) to propose a non-parametric model of synchronous tree substi­tution grammar (STSG), continuing a recent trend in SMT to seek principled probabilistic formula­tions for heuristic translation models (Zhang et al., 2008; DeNero et al., 2008; Blunsom et al., 2009b; Blunsom et al., 2009a). This model leverages a hierarchical Bayesian prior to induce a compact translation grammar directly from a parsed paral­lel corpus, unconstrained by word-alignments. We show that the induced grammars are more plausi­ble and improve translation output.</p><p>This paper is structured as follows: In Section</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">352</doubt><p><i>Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, </i>pages 352-361, Singapore, 6-7 August 2009. ©2009 ACL and AFNLP<page local="2" global="353"/></p><p>2 we introduce the STSG formalism and describe current heuristic approaches to grammar induc­tion. We define a principled Bayesian model of string-to-tree translation in Section 3, and describe an inference technique using Gibbs sampling in Section 4. In Section 5 we analyse an induced grammar on a corpus of Chinese^English trans­lation, comparing them with a heuristic grammar in terms of grammar size and translation quality.</p></section><section number="2" title="Background"><p>Current tree-to-string translation models are a form of <i>Synchronous Tree Substitution Grammar </i>(STSG; Eisner (2003)). Formally, a STSG is a 5-tuple, <i>G</i><i> </i><i>=</i><i> </i><i>(T,T',N,S,R),</i><i> </i>where <i>T</i><i> </i>and <i>V</i><i> </i>are sets of <i>terminal symbols </i>in the target and source languages respectively, <i>N </i>is a set of <i>non­terminal symbols, S </i>e <i>N </i>is the distinguished <i>root non-terminal </i>and <i>R </i>is a set of productions (a.k.a. rules). Each production is a tuple compris­ing an <i>elementary tree </i>and a string, the former referring to a tree fragment of depth &gt; 1 where each internal node is labelled with a non-terminal and each leaf is labelled with either a terminal or a non-terminal. The string part of the rule de­scribes the lexical component of the rule in the source language and includes a special variable for each <i>frontier non-terminal </i>in the elementary tree. These variables describe the reordering and form the recursion sites in the generative process of cre­ating tree and string pairs with the grammar. For example, the rule rewrites a noun-phrase (NP) as a NP and prepo­sitional phrase (PP) headed by 'of in the target language. The rule generates the token <i>'t£j' </i>in the source and reverses the order of the two child noun-phrases, indicated by the numbering of the variables in the string part.</p><doubt alpha="54.1" length="37" tooSmall="False" monospace="0.0">((NP NPs(PP (IN of)NPi)),MIft CD) (1)</doubt><p>A <i>derivation </i>creates a (tree, string) pair by start­ing with the root non-terminal and an empty string, then choosing a rule to rewrite (substitute) the non­terminal and expand the string. This process re­peats by rewriting all frontier non-terminals until there are none remaining. A <i>Probabilistic STSG </i>assigns a probability to each rule in the grammar. The probability of a derivation is the product of the probabilities of its component rules, and the probability of a (tree, string) pair is the sum of the probabilities over all its derivations.</p><subsection number="2.1" title="Heuristic Grammar Induction"><p>Grammar based SMT models almost exclusively follow the same two-stage approach to gram­mar induction developed for phrase-based meth­ods (Koehn et al., 2003). In this approach they induce a finite-state grammar with phrase-pairs as rules by taking a sentence aligned parallel cor­pus and 1) predicting word alignments before 2) extracting transduction rules that are 'consistent' with the word aligned data. Although empiri­cally effective, this two stage approach is less than ideal due to the disconnect between the word-based models used for alignment and the phrase-based translation model. This is problematic as the word-based model cannot recognise phrase-based phenomena. Moreover, it raises the problem of identifying and weighting the rules from the word alignment.</p><p>The same criticisms levied at the phrase-based models apply equally to the two-stage technique used for synchronous grammar induction (Galley et al., 2004; Zollmann and Venugopal, 2006; Gal­ley et al., 2006; Marcu et al., 2006). Namely that the word alignment models typically do not use any syntax and therefore will not be able to model, e.g., consistent syntactic reordering effects, or the impact of the syntactic category on phrase transla­tions. The identification and estimation of gram­mar rules from word aligned data is also non-trivial. Galley et al. (2004) describe an algorithm for inducing a string-to-tree grammar using a par­allel corpus with syntax trees on target side. Their method projects the source strings onto nodes of the target tree using the word alignment, and then extracts the minimal transduction rules as well as rules composed of adjacent minimal units. The production weights are estimated either by heuris­tic counting (Koehn et al., 2003) or using the EM algorithm. Both estimation techniques are flawed. The heuristic method is inconsistent in the limit (lohnson, 2002) while EM is <i>degenerate, </i>placing disproportionate probability mass on the largest rules in order to describe the data with as few a rules as possible (DeNero et al., 2006). With no limit on rule size this method will learn a single rule for every training instance, and therefore will not generalise to unseen sentences. These prob­lems can be ameliorated by imposing limits on rule size or early stopping of EM training, how­ever neither of these techniques addresses the un­derlying problems.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">353</doubt><page local="3" global="354"/><p>In contrast, our model is trained in a single step, i.e., the alignment model <i>is </i>the translation model. This allows syntax to directly inform the align­ments. We infer a grammar without resorting to word alignment constraints or limits on rule size. The model uses a prior to bias towards a compact grammar with small rules, thus solving the degen­eracy problem.</p></subsection></section><section number="3" title="Model"><p>Our training data comprises parallel target trees and source strings and our aim is to induce a STSG that best describes this data. This is achieved by inferring a distribution over the derivations for each training instance, where the set of derivations collectively specify the grammar. In the follow­ing, we denote the source trees as t, target strings s, and derivations r which are sequences of gram­mar rules, <i>r.</i></p><p>As described in section 2.1, previous methods for estimating a STSG have suffered from degen­eracy. A principled way to correct for such degen­erated behaviour is to use a <i>prior </i>over rules which biases towards small rules. This matches our intu­ition: we expect good translation rules to be small, with few internal nodes, frontier non-terminals and terminal strings. However, we recognise that on occasion larger rules will be necessary; we al­low such rules when there is sufficient support in the data.</p><p>We model the grammar as a set of distributions, <i>Gc, </i>over the productions for each non-terminal symbol, c. We adopt a non-parametric Bayesian approach by treating each <i>Gc </i>as a random variable with a Dirichlet process (DP) prior, where <i>Po(-\c) </i>(the <i>base distribution) </i>is a distribu­tion over the infinite space of trees rooted with c, and <i>ac </i>(the <i>concentration parameter) </i>controls the model's tendency towards either reusing existing rules or creating novel ones as each training in­stance is encountered (and consequently, the ten­dency to infer larger or smaller grammars). We discuss the base distribution in more detail below.</p><doubt alpha="42.1" length="38" tooSmall="False" monospace="0.0">r\c       ~ GcGc|ac,Po-DP(ac,P0(-|c)),</doubt><p>Rather than representing the distribution <i>Gc </i>ex­plicitly, we integrate over all possible values of <i>Gc. </i>This leads to the following predictive distribution for the rule <i>ri </i>given the previously observed rules where <i>n~% </i>is the number number of times <i>r% </i>has been used to rewrite c in r~\ and <i>n~% = </i><i>J2r</i><i> R(r)=cnr% ls </i>the total count of rewrit­ing c (here <i>R(r) </i>is the root non-terminal of <i>r).</i><i> </i>The distribution is <i>exchangeable, </i>meaning that all per­mutations of the input sequence are assigned the same probability. This allows us to treat any item as being the last, which is fundamental for efficient Gibbs sampling. Henceforth we adopt the notation r~ and <i>n~ </i>to refer to the rules and counts for the whole data set excluding the current rules under consideration, irrespective of their location in the corpus.</p><doubt alpha="35.7" length="14" tooSmall="False" monospace="0.0">r1= n ...Ti-i,</doubt><doubt alpha="42.9" length="21" tooSmall="False" monospace="0.0">,  , xn~%+ acPo{ri\c)</doubt><doubt alpha="43.5" length="23" tooSmall="False" monospace="0.0">p(rt\r-\c,ac,Po)=n, (2)</doubt><doubt alpha="66.7" length="6" tooSmall="False" monospace="0.0">nc+ ac</doubt><p>The base distribution, <i>P</i><i>q,</i><i> </i>assigns a prior prob­ability to an infinite number of rules, where each rule is an (elementary tree, source string) pair de­noted <i>r </i>= (e, w). While there are a myriad of possible distributions, we developed a very sim­ple one. We decompose the probability into two factors, the probability of the target elementary tree and the probability of the source string, where c = <i>R(r).</i><i></i></p><doubt alpha="38.5" length="26" tooSmall="False" monospace="0.0">P0(e,w|c)=P(e|c)P(w|e),(3)</doubt><p>The tree probability, <i>P(e\c) </i>in (3), is modelled using generative process whereby the root cate­gory c is expanded into a sequence of child non­terminals, then each of these are either expanded or left as-is. This process continues until there are no unprocessed children. The number of child nodes for each expansion is drawn from a geo­metric prior with parameter pcniid&gt; except in the case of pre-terminals where the number of chil­dren is always one. The binary expansion deci­sions are drawn from a Bernoulli prior with pa­rameter Pexpand&gt; and non-terminals and terminals are drawn uniformly from <i>N </i>and <i>T </i>respectively. For example, the source side of rule (1) was gen­erated as follows: 1) the NP was rewritten as two children; 2) an NP; and 4) a PP; 5) the NP child was not expanded; 6) the PP child was expanded; 7) as an IN; and 8) a NP; 9) the IN was expanded to the terminal 'of ; and 10) the final NP was not ex­panded. Each of these steps is a draw from the rel­evant distribution, and the total probability is the product of the probabilities for each step.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">354</doubt><page local="4" global="355"/><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">A'1</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">3 1</doubt><doubt alpha="18.2" length="11" tooSmall="False" monospace="0.0">(ft     % —</doubt><doubt alpha="10.0" length="10" tooSmall="False" monospace="0.0">\      5 A</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">* 7</doubt><doubt alpha="40.0" length="5" tooSmall="False" monospace="0.0">T.-H-</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">3tM</doubt><doubt alpha="57.1" length="7" tooSmall="False" monospace="0.0">riof 11</doubt><p>Figure 1: Example derivation. Each node is annotated with their span in the target string (aligned nodes are shaded). The dotted edges show the implied alignments. Preterminals are displayed with their child terminal in the leaf nodes.</p><p>The second factor in (3) is P(w|e), the prob­ability of the source string (a sequence of source terminals and variables). We assume that the el­ementary tree is generated first, and condition the string probability on <i>I = F(e), </i>its number of fron­tier nodes (i.e., variables). The string is then cre­ated by choosing a number of terminals from a ge­ometric prior with parameter pterm then drawing each terminal from a uniform distribution over <i>T<footnote anchor="1"/>. </i>Finally each of the <i>I </i>variables are inserted into the string one at a time using a uniform distribution over the possible placements. For the example rule in (1) the generative process corresponds to 1) de­ciding to create one terminal; 2) with value Ö^J; 3) inserting the first variable after the terminal; and 4) inserting the second variable before the termi­nal. Again, the probability of the string is simply the product of the probabilities for each step.</p><p>Together both the factors in the base distribu­tion penalise large trees with many nodes and long strings with many terminals and variables. Po de­creases exponentially with rule size, thus discour­aging the model from using larger rules; for this to occur the rules must significantly increase the likelihood.</p></section><section number="4" title="Training"><p>To train our model we use Gibbs sampling (Geman and Geman, 1984), a Markov chain Monte Carlo method (Gilks et al., 1996) in which variables are repeatedly sampled conditioned on the values of all other variables in the model. After a period of burn-in, each sampler state (set of variable assign­ments) is a sample from the posterior distribution of the model. In our case, we wish to sample from the posterior over the grammar, P(r| t, s, <i>a).</i></p><p>To simplify matters we associate an alignment variable, <i>a, </i>with every internal node of the trees in the training set. This variable specifies the span of source tokens to which node is aligned. Alter­natively, the node can be unaligned, which is en­coded as an empty span. I.e. <i>a</i><i> </i>e ( <i>J</i><i> </i>x J) U 0 where J is the set of target word indices. Spans are written inclusive of <i>i</i><i> </i>and exclusive of <i>j.</i><i> </i>Each aligned node <i>(a </i>/ 0) forms the root of a rule as well as being a frontier non-terminal of an ancestor rule, while unaligned nodes form part of an ancestor rule.<footnote anchor="2"/> The set of valid align­ments are constrained by the tree in a number of ways. Child nodes can be aligned only to sub-spans of their ancestor nodes' alignments and no two nodes' alignments can overlap. Finally, the root node of the tree must be aligned to the full<page local="5" global="356"/></p><footnote label="1">Previous approaches to bilingual grammar induction have used variational inference to optimise a bound on the data log-likelihood (Zhang et al., 2008; Blunsom et al., 2009b). Both these approaches truncated the grammar a pri­ori in order to permit tractable inference. In contrast our Gibbs sampler can perform inference over the full space of grammars. See also Blunsom et al. (2009a) where we present a Gibbs sampler for inducing SCFGs without truncation.</footnote><footnote label="2">The Gibbs sampler is an extension of our sampler for monolingual tree-substitution grammar (Cohn et al., 2009), which used a binary substitution variable at each node to en­code the segmentation of a training tree into elementary trees.</footnote><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">355</doubt><doubt alpha="58.6" length="29" tooSmall="False" monospace="0.0">((s(np npeppm)vps.s), m m us)</doubt><doubt alpha="57.7" length="26" tooSmall="False" monospace="0.0">((NPDTsNN0),E^)((DTEvery),</doubt><doubt alpha="55.0" length="20" tooSmall="False" monospace="0.0">((NNcorner),—-f"^IH)</doubt><doubt alpha="60.0" length="20" tooSmall="False" monospace="0.0">((PP(INof)NPs),ffl^)</doubt><doubt alpha="64.0" length="25" tooSmall="False" monospace="0.0">((NP (NNPHong)(NNPKong)),</doubt><doubt alpha="61.1" length="18" tooSmall="False" monospace="0.0">((VP(VBZis)VPs),E)</doubt><p>((VPVBNsPPi),fPfflS)</p><doubt alpha="62.5" length="16" tooSmall="False" monospace="0.0">((VBNfilled), ?E</doubt><doubt alpha="62.5" length="24" tooSmall="False" monospace="0.0">((PP (INwith)(NPNNeû),m)</doubt><doubt alpha="54.5" length="11" tooSmall="False" monospace="0.0">((NNfun),M)</doubt><doubt alpha="0.0" length="10" tooSmall="False" monospace="0.0">((■•), • )</doubt><p>Table 1 : Grammar rules specified by the derivation in Fig­ure 1. Each rule is shown as a tuple comprising a tar­get elementary tree and a source string. Boxed numbers show the alignment between string variables and frontier non­terminals.</p><doubt alpha="0.0" length="5" tooSmall="True" monospace="0.0">[2,6)</doubt><p>span of source words.</p><p>Collectively, the training trees and alignment variables specify the sequence of rules r, which in turn specify the grammar. Figure 1 shows an example derivation with alignment variables. The corresponding STSG rules are shown in Table 1.</p><subsection number="4.1" title="Gibbs operators"><p>The Gibbs sampler works by sampling new val­ues of the alignment variables, using two differ­ent Gibbs operators to make the updates. The first operator, Expand, takes a tree node, <i>v, </i>and sam­ples a new alignment, <i>av</i>, given the alignments of all other nodes in the same tree and all other trees in the corpus, denoted a~. The set of valid la­bels is constrained by the other alignments in the tree, specifically that of the node's closest aligned ancestor, <i>ap, </i>its closest aligned descendants, a^, and its aligned siblings, <i>as </i>(the aligned descen­dants of a). The alignment variable may be empty, <i>av </i>= 0, while non-empty values must obey the tree constraints. Specifically the span must be a subspan of its ancestor, <i>av </i>ç <i>ap, </i>subsume its de­scendants, <i>av </i>5 U a&lt;i. and not overlap its siblings, <i>j </i>U <i>as </i>&gt; Vj G <i>av. </i>Figure 2 shows an exam­ple with the range of valid values for <i>corner/NN's </i>alignment variable and the corresponding align­ments that these encode.</p><p>Each alignment in the set of valid outcomes de­fines a set of grammar rules. The non-aligned out­come results in a single rule <i>rp </i>rooted at ancestor node <i>p.</i><i> </i>While the various aligned outcomes resuit in a pair of rules, ry and <i>rv, </i>rooted at <i>p </i>and <i>v </i>respectively. In the example in Figure 2, the top-right outcome has <i>av = </i>0 and</p><figure caption="Figure 2: Possible state updates for the(nncorner) node us­ing theExpandoperator."></figure><doubt alpha="63.6" length="22" tooSmall="False" monospace="0.0">((NPDTs (NNcorner)), H</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">1^) •</doubt><p>The bottom-right outcome, <i>av = </i>[4, 5), describes the pair of rules:</p><p>V = ((NP DTcd NNm), ffl-^ftl) and <i>rv = </i>((NN corner), -f") .</p><p>The set of valid options are then scored according to the probability of their rules as follows:</p><doubt alpha="57.1" length="7" tooSmall="False" monospace="0.0">P(rp\r~</doubt><doubt alpha="38.5" length="13" tooSmall="False" monospace="0.0">n   + aP0(rp\</doubt><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">Cp )</doubt><doubt alpha="66.7" length="6" tooSmall="False" monospace="0.0">nCp+ a</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(4)</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(5)</doubt><doubt alpha="52.5" length="40" tooSmall="False" monospace="0.0">P(rp&gt;,rv\r)=P(ry|r)P(rv\r,ry)n~,+&lt;xP0(vl</doubt><doubt alpha="20.0" length="10" tooSmall="False" monospace="0.0">=   —--z-x</doubt><doubt alpha="51.4" length="35" tooSmall="False" monospace="0.0">n~v+5{rp&gt;,rv) + aP0(rnöv+ö(cp,cv)+a</doubt><p>where <i>cp </i>is the non-terminal at node <i>p </i>(simi­larly for <i>cv), n~ </i>denote counts of trees (e.g., <i>n~ ) </i>or the sum over all trees expanding a non­terminal (e.g., <i>n~v) </i>in the conditioning context, r~, and £(•,•) is the Kronecker delta function, which returns 1 when its arguments are identi­cal and 0 otherwise. For clarity, we have omit­ted some items from the conditioning context in (4) and (5), namely t, s and hyper-parameters «Schild, Pexpand,Pterm- The <i>ö </i>terms in the sec­ond factor of (5) account for the changes to <i>n~</i><i> </i>that would occur after observing ry, which forms part of the conditioning context for <i>rv.</i><page local="6" global="357"/><i></i><i> </i>If the rules ry and <i>rv</i><i> </i>are identical, then the count <i>n~v</i><i> </i>would increase by one, and if the rules expand the same root non-terminal, then <i>n~v</i><i> </i>would increase by one. Equation (4) is evaluated once for the unaligned outcome, <i>av</i><i> </i><i>=</i><i> </i>0, and (5) is evaluated for each valid alignment. The probabilities are normalised and an outcome sampled.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">356</doubt><p>The EXPAND operator is sufficient to move from one derivation to any other valid derivation, however it may take many steps to do so. These intermediate steps may require the sampler to pass through highly improbable regions of the state space, and consequently such moves are unlikely. The second operator, Swap, is designed to help address this problem by increasing the mobility of the sampler, allowing it to mix more quickly. The operator considers pairs of nodes, <i>v,w,m </i>one tree and attempts to swap their alignment values.<footnote anchor="3"/> This is illustrated in the example in Figure 3. There are two options being compared: preserving the align­ments (left) or swapping them (right). This can change three rules implied by the derivation: that rooted at the nodes' common aligned ancestor, <i>p, </i>and those rooted at <i>v </i>and <i>w. </i>For the example, the left option implies rules and the right option implies rules</p><doubt alpha="53.0" length="66" tooSmall="False" monospace="0.0">{fp=((NP DTcn NNn), Q]\2\),rv=((DTEvery),U),rw=((NNcorner),-fÄI}},</doubt><p><i>{fP</i><i> </i><i>=</i><i> </i>((NP DTcn NNnO, m [Tj), <i>rv</i><i> </i><i>=</i><i> </i>((DT Every), — <i>faW), </i><i>rw</i><i> </i>= ((NN corner), .</p><p>the probability of under   our model,</p><doubt alpha="41.2" length="17" tooSmall="False" monospace="0.0">= P(rp\r)P(r\,|r"</doubt><p>We   simply evaluate both   triples   of rules <i>P</i><i>{fp,</i><i> Tv, Tw </i>IF <i>P</i><i>{r </i><i>w\r</i><i> </i><i>,rp,rv), </i>where the additional rules in the conditioning context signify their inclusion in the counts r~ before applying (2) to evaluate the probability (much the same as in (5) where</p><footnote label="3">We rarely need to consider the full quadratic space of node pairs, as the validity constraints mean that the only candidates for swapping are siblings (i.e., share the closest aligned ancestor) which do not have any aligned descendants.</footnote><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">{23</doubt><doubt alpha="9.1" length="22" tooSmall="False" monospace="0.0">2      |&gt;3i4 *      5X</doubt><p><i>m</i><i> </i><i>- -t- </i><i>p.</i></p><p>Figure 3: Possible state updates for the pair of nodes (DT every) and (NN corner) using the Swap operator.</p><p>the <i>ö </i>functions encode the changes to the counts). An outcome is then sampled according to the normalised probabilities of the preserve vs. swap rules.</p><p>The Gibbs sampler makes use of both operators. The algorithm visits each (tree, string) pair in the training set in random order and applies the EX­pand operator to every node in the tree. After the tree has been processed, the Swap operator is ap­plied to all candidate pairs of nodes. Visiting all sentence pairs in this way constitutes a single sam­ple from the Gibbs sampler.</p></subsection></section><section number="5" title="Experiments"><p>We evaluate our non-parametric model of gram­mar induction on a subset of the NIST Chinese-English translation evaluation, representing a real­istic SMT experiment with millions of words and long sentences. The Chinese-English training data consists of the FBIS corpus (LDC2003E14) and the first 100k sentence pairs from the Sinorama corpus (LDC2005E47). The Chinese text was seg­mented with a CRF-based Chinese segmenter op­timized for MT (Chang et al., 2008), and the En­glish text was parsed using the Stanford parser (Klein and Manning, 2003).</p><p>As a baseline we implemented the heuristic grammar extraction technique of Galley et al. (2004) (henceforth GHKM). This method finds the minimum sized translation rules which are consistent with a word-aligned sentence pair, as described in section 2.<page local="7" global="358"/>1. The rules are then treated as events in a relative frequency esti­mate.<footnote anchor="4"/> We used Giza++ Model 4 to obtain word alignments (Och and Ney, 2003), using the grow-diag-f inal-and heuristic to sym­metrise the two directional predictions (Koehn et al., 2003).</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">357</doubt><table caption="Table 2: NIST Chinese-English corpora statistics (LDC2003E14, LDC2005E47)." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>English &lt;— Chinese</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Sentences Words or Segments Avg. Sent. Length Longest Sent.</p></td><td class="cell"><p>300k ll.OM 8.6M 36 28 80 80</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>The model was sampled for 300 iterations to 'burn-in', where in each iteration we applied both sampling operators to all nodes (or node pairs) of all training instances. We initialised the sam­pler using the GHKM derivation of the training data (the baseline system). The final state of the sampler was used to extract the grammar. The hyperparameters were set by hand to a = 10<footnote anchor="6"/>, Pchild = 0.5, Expand = 0.5, and pterm = 0.5.<footnote anchor="5"/>Overall the model took on average 2,218s per full iteration of Gibbs sampling and 1 week in total to train, using a single core of a 2.3Ghz AMD Opteron machine.</p><subsection number="5.1" title="Grammar Analysis"><p>The resulting grammar had 1.62M rules, al­most identical to the GHKM grammar which had 1.63M. Despite their similarity in size the gram­mars were quite different, as illustrated in Fig­ure 4, which shows histograms over various mea­sures of rule size for the two grammars. Under each measure the sampled grammar finds many more simple rules - shallower with fewer internal nodes, fewer variables and fewer terminals - than the GHKM method. This demonstrates that the prior is effective in shifting mass away from com­plex rules. To show how the rules themselves dif­fer, Table 3 lists rules in the sampled grammar that are not in the GHKM grammar. Note that many of these rules are highly plausible, describing regular tree structures and lexicalisation. These rules have not been specified to the same extent in the GHKM grammar. For example the first rule incorporates</p><footnote label="4">Our implementation of the GHKM algorithm attaches unaligned source words to the highest possible node in the source tree, rather than allowing all attachment points as in the original presentation (Galley et al., 2004). Allowing all attachments made no difference to translation performance, but did make the grammar considerably larger. We imple­mented only the minimal rule extraction, i.e., with no rule composition (Galley et al., 2006). Consequently there is no derivational ambiguity, obviating the need for expectation maximisation or similar for rule estimation.</footnote><footnote label="5">Note that although a seems large, it still encourages sparse distributions as the Po values are typically much smaller than its reciprocal, 10~ 6 , especially if the rule is large. aPo &lt; 1 implies a sparse Dirichlet prior.</footnote><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">I</doubt><doubt alpha="66.7" length="6" tooSmall="False" monospace="0.0">■ GKHM</doubt><doubt alpha="71.4" length="7" tooSmall="False" monospace="0.0">■ Gibbs</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">■ ■</doubt><p>maximum tree depth source terminals target terminals</p><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">to</doubt><doubt alpha="50.0" length="2" tooSmall="False" monospace="0.0">d]</doubt><p>Figure 4: Histograms over rule statistics comparing the heuristic grammar (GHKM) and learnt grammar (Gibbs).</p><p>the TOP symbol, while the GHKM grammar in­stead relies on the rule ((TOP Srjj), uj) to produce the same fragment. The model has learnt to dis­tinguish between sentence-spanning and subsen-tential S constituents, which typically do not in­clude final punctuation. The third and ninth (last) rule are particularly interesting. These rules en­code reordering effects relating to noun phrases and subordinate prepositional phrases, in partic­ular that Chinese prepositional modifiers precede the nouns they modify. Differences in word or­der such as these are quite common in Chinese-English corpora, so it is imperative that they are modelled accurately.</p><p>The rules in the GHKM grammar that do not appear in the sampled grammar are shown in Table 4. In contrast to the rules only present in the sampled grammar, these have much lower counts, i.e., are less probable. Each of these rules has been specified further by the Bayesian model.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">358</doubt><page local="8" global="359"/><doubt alpha="51.7" length="29" tooSmall="False" monospace="0.0">((TOP (S NPH VPHm)),ÜJ[2J[3J&gt;</doubt><doubt alpha="52.2" length="23" tooSmall="False" monospace="0.0">((s(VP (toto)vph)), LH)</doubt><doubt alpha="53.1" length="32" tooSmall="False" monospace="0.0">((NP NPH (PP (IN of) NPH)),LHQ]}</doubt><doubt alpha="54.2" length="24" tooSmall="False" monospace="0.0">((PP (IN in) NPH), ftcd)</doubt><doubt alpha="50.0" length="34" tooSmall="False" monospace="0.0">((NP NP[0 (PP (IN of) NPH)),[TjHI)</doubt><doubt alpha="51.9" length="27" tooSmall="False" monospace="0.0">((NP (DT the) NNH), ffj Q]}</doubt><doubt alpha="45.5" length="22" tooSmall="False" monospace="0.0">((S (VPTOHVPH)),h][2])</doubt><doubt alpha="63.6" length="22" tooSmall="False" monospace="0.0">((VP(VBZis)npti]),êLU)</doubt><doubt alpha="47.9" length="48" tooSmall="False" monospace="0.0">((NP (NP (DT the) NNH) (PP (IN of) NPH)), [2]LU)</doubt><p>Table 3: Top ten rules in the sampled grammar that do not appear in the GHKM grammar. All the above rules are quite high probability, with counts between 37,118 and 7,275 from first to last.</p><doubt alpha="64.9" length="37" tooSmall="False" monospace="0.0">((pp (inat)(npDTlO(nnslevels))),LUB.)</doubt><doubt alpha="61.1" length="36" tooSmall="False" monospace="0.0">((np nph,nnph (,,)cesnph),ldlhl3sh]}</doubt><doubt alpha="56.1" length="57" tooSmall="False" monospace="0.0">((NP NPH,hNPH,hNPH (,,) (CC and) NPH),ltjlh lh lhH], [ë]}</doubt><doubt alpha="53.0" length="115" tooSmall="False" monospace="0.0">((s sh (np (prpThey))vps ,h),lulhs)((s pph ,h nph vph -h"Ml),lh lh lh s\§\[5]}((s pph ,h nph vph -h),lu ^ lh 1s lu)</doubt><doubt alpha="66.7" length="63" tooSmall="False" monospace="0.0">((NP (NNP Foreign) (NNP Ministry) NNH (NNP Zhu) (NNP Bangzao)),</doubt><doubt alpha="64.3" length="14" tooSmall="False" monospace="0.0">((sshsh),ldlh)</doubt><doubt alpha="55.9" length="34" tooSmall="False" monospace="0.0">((s sh (np (prpWe))vph -h),lu lus)</doubt><doubt alpha="66.7" length="36" tooSmall="False" monospace="0.0">((np(DT the)(nnspeople)posh), aslu)_</doubt><p>Table 4: Top ten rules in the GHKM grammar that do not ap­pear in the sampled grammar. These are quite low probability rules: their counts range from 1,137 to 103.</p><p>For example, every instance of the first rule had the same determiner and target translation, ((PP (IN at) (NP (DT all) (NNS levels))), # <i>M),</i><i> </i>and therefore the model specified the determiner, resulting in a single rule. The model has correctly learnt that other translations for (DT all) are not appropriate in this context (e.g., <i>M,</i><i> </i><i>PftM</i><i> </i>or —<i>'W). </i>In a number of the remaining rules the commas were lexicalised, or S rules were extended to include the TOP symbol.</p><p>To further illustrate the differences between the grammars, Table 5 shows the rules which include the possessive particle, Ö^J, and at least one vari­able. In both grammars there are many fully lex­icalised rules which translate the token to, e.g., a determiner or a preposition. The grammars differ on the complex rules which combine lexicalisa­tion and frontier non-terminals. The GHKM rules are all very simple depth-1 SCFG rules, contain­ing minimal information. In contrast, the sampled rules are more lexicalised, licensing the insertion of various English tokens and tree substructure. Note particularly the second and forth rule which succinctly describe the reordering of prepositional</p><p>Sampled Grammar <u>GHKM Grammar</u></p><doubt alpha="59.1" length="22" tooSmall="False" monospace="0.0">((np(DT the)nnh),W LU)</doubt><doubt alpha="61.8" length="68" tooSmall="False" monospace="0.0">((np (np(DT the)nnh) (pp (inof)nph)),LHfftLTJ)((np(DTthe)nnh),LI] W)</doubt><doubt alpha="60.0" length="55" tooSmall="False" monospace="0.0">((NP (NP (DT the) JJH NNH) (PP (IN of) NPH)),LHffiLHLH)</doubt><doubt alpha="57.1" length="21" tooSmall="False" monospace="0.0">((pp (inof)nph),lum)_</doubt><doubt alpha="59.1" length="22" tooSmall="False" monospace="0.0">((np jjh nnsh),LUWl2|)</doubt><p>((np jjhnnhxLUWLH) ((np dtjjj jjh nnh), LU LH W LH) ((np prp$h nnh),LU W LH) phrases with an noun phrase. <b>5.2 Translation</b></p><doubt alpha="61.9" length="21" tooSmall="False" monospace="0.0">((np nph pps),LHWlp)_</doubt><table caption="Table 5: Top five rules which include the possessive particleFBand at least one variable."></table><p>In order to test the translation performance of the grammars induced by our model and the GHKM method<footnote anchor="6"/> we report bleu (Papineni et al., 2002) scores on sentences of up to twenty words in length from the MT03 NIST evaluation. We built a synchronous beam search decoder to find the maximum scoring derivation, based on the CYK+ chart parsing algorithm and the cube-pruning method of Chiang (2007). Parse edges for all constituents spanning a given chart cell were cube-pruned together using a beam of width 1000, and only edges from the top ten constituents in each cell were retained. No artificial glue-rules or rule span limits were employed.<footnote anchor="7"/> The parameters of the translation system were trained to maximize Bleu on the MT02 test set (Och, 2003). Decoding took roughly 10s per sentence for both grammars, using a 8-core 2.6Ghz Intel Xeon machine.</p><p>Table 6 shows the Bleu scores for the baseline using the GHKM rule induction algorithm, and our non-parametric Bayesian grammar induction method. We see a small increase in generalisation performance from our model. Our previous analysis (Section 5.<page local="9" global="360"/>1) of the grammars produced by the two approaches showed our method produced better lexicalised rules than those induced by the GHKM algorithm. Galley et al. (2006) noted that the GHKM algorithm often over generalised and proposed combining minimal rules to form com­posed rules as a solution. Although composing rules was effective at improving bleu scores, the result was a massive expansion in the size of the grammar. By learning the appropriate level of lex­icalisation we believe that our inference algorithm is having a similar effect as composing rules (Gal­ley et al., 2006), however the resulting grammar remains compact, a significant advantage of our approach.</p><footnote label="6">Our decoder was unable to process unary rules (those which consume nothing in the source). Monolingual pars­ing with unary productions is fairly straightforward (Stolcke, 1995), however in the transductive setting these rules can li­cence infinite insertions in the target string. This is further complicated by the language model integration. Therefore we composed each unary rule instance with its descendant rule(s) to create a non-unary rule.</footnote><footnote label="7">Our decoder lacks certain features shown to be beneficial to synchronous grammar decoding, in particular rule binari-sation (Zhang et al., 2006). As such the reported results for MT03 lag the state-of-the-art: the Moses phrase-based de­coder (Koehn et al., 2007) achieves 26.8. We believe that im­provements from a better decoder implementation would be orthogonal to the improvements presented here (and would allow us to relax the length restriction on the test set).</footnote><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">359</doubt><doubt alpha="62.8" length="43" tooSmall="False" monospace="0.0">Model     BLEU score GHKM26X)Our model 26.6</doubt><table caption="Table 6: Translation results on the NIST test set MT03 for sentences of length &lt; 20."></table></subsection></section><section number="6" title="Conclusion"><p>In this paper we have presented a method for in­ducing a tree-to-string grammar which removes the need for various heuristics and constraints from models of word alignment. Instead the model is capable of directly inferring a grammar in one step, using the syntactic fragments that it has learnt to better align the source and target data. Using a prior which favours sparse distributions and sim­pler rules, we demonstrate that the model finds a more parsimonious grammar than the heuristic technique. Moreover, this grammar results in im­proved translations on a standard evaluation set.</p><p>We expect that various extensions to the model would improve its performance. One avenue is to develop a more sophisticated prior over rules, e.g., one that recognises common types of rule via the shape of the tree and ordering pattern in the tar­get. A second avenue is to develop better means of inference under the grammar, in order to ensure faster mixing and a means to escape from local optima. Finally, we wish to develop a method for decoding under the full Bayesian model, instead of the current beam search. With these extensions we expect that our model of grammar induction has the potential to greatly improve translation output.</p></section><section title="Acknowledgements"><p>The authors acknowledge the support of the EP-SRC (grants GR/T04557/01 and EP/D074959/1). This work has made use of the resources pro­vided by the Edinburgh Compute and Data Facility (ECDF). The ECDF is partially supported by the eDIKT initiative.</p></section><references><p>P. Blunsom, T. Cohn, C. Dyer, M. Osborne. 2009a. A Gibbs sampler for phrasal synchronous grammar induction. In <i>Proceedings of the Joint conference of the 47th Annual Meeting of the Association for Computational Linguistics and the 4th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Pro­cessing, </i>Singapore. To appear.</p><p>P. Blunsom, T. Cohn, M. Osborne. 2009b. Bayesian synchronous grammar induction. In D. Koller, D. Schuurmans, Y. Bengio, L. Bottou, eds., <i>Ad­vances in Neural Information Processing Systems 21, </i>161-168. MIT Press, Cambridge, MA.</p><p>P.-C. Chang, M. Galley, C. D. Manning. 2008. Op­timizing Chinese word segmentation for machine translation performance. In <i>Proceedings of the Third Workshop on Statistical Machine Translation, </i>224-232, Columbus, Ohio.</p><p>D. Chiang. 2007. Hierarchical phrase-based transla­tion. <i>Computational Linguistics, </i>33(2):201-228.</p><p>T. Cohn, S. Goldwater, P. Blunsom. 2009. Inducing compact but accurate tree-substitution grammars.</p><p>In <i>Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Lin­guistics, </i>548-556, Boulder, Colorado.</p><p>J. DeNero, D. Gillick, J. Zhang, D. Klein. 2006. Why generative phrase models underperform sur­face heuristics. In <i>Proceedings on the Workshop on Statistical Machine Translation, </i>31-38, New York City, NY.</p><p>J. DeNero, A. Bouchard-Côté, D. Klein. 2008. Sam­pling alignment structure under a Bayesian transla­tion model. In <i>Proceedings of the 2008 Conference on Empirical Methods in Natural Language Pro­cessing, </i>314-323, Honolulu, Hawaii.</p><p>J. Eisner. 2003. Learning non-isomorphic tree map­pings for machine translation. In <i>The Companion Volume to the Proceedings of 41 st Annual Meeting of the Association for Computational Linguistics, </i>205-208, Sapporo, Japan.</p><p>V. Fossum, K. Knight, S. Abney. 2008. Using syn­tax to improve word alignment precision for syntax-based machine translation.  In <i>Proceedings of the</i> <i>Third Workshop on Statistical Machine Translation, </i>44-52, Columbus, Ohio.<page local="10" global="361"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">360</doubt><p>M. Galley, M. Hopkins, K. Knight, D. Marcu. 2004. What's in a translation rule? In <i>Proceedings of the 2004 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, </i>273-280, Boston, MA.</p><p>M. Galley, J. Graehl, K. Knight, D. Marcu, S. DeNeefe, W. Wang, I. Thayer. 2006. Scalable inference and training of context-rich syntactic translation mod­els. In <i>Proceedings of the 21st International Con­ference on Computational Linguistics and 44th An­nual Meeting of the Association for Computational Linguistics, </i>961-968, Sydney, Australia.</p><p>S. Geman, D. Geman. 1984. Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images. <i>IEEE Transactions on Pattern Analysis and Machine Intelligence, 6:121-1 Al.</i></p><p>W. Gilks, S. Richardson, D. J. Spiegelhalter, eds. 1996. <i>Markov Chain Monte Carlo in Practice. </i>Chapman and Hall, Suffolk.</p><p>M. Johnson, T. L. Griffiths, S. Goldwater. 2007. Adaptor grammars: A framework for specifying compositional nonparametric Bayesian models. In</p><p>B. Schölkopf, J. Platt, T. Hoffman, eds., <i>Advances in Neural Information Processing Systems 19, </i>641-648. MIT Press, Cambridge, MA.</p><p>M. Johnson. 2002. The DOP estimation method is biased and inconsistent. <i>Computational Lingusitics, </i>28(l):71-76.</p><p>D. Klein, CD. Manning. 2003. Fast exact inference with a factored model for natural language parsing. In <i>In Advances in Neural Information Processing Systems 15, </i>3-10. MIT Press.</p><p>P. Koehn, F. J. Och, D. Marcu. 2003. Statisti­cal phrase-based translation. In <i>Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, </i>48-54, Edmonton, Canada.</p><doubt alpha="65.6" length="93" tooSmall="False" monospace="0.0">P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen,</doubt><p>C. Moran, R. Zens, C. Dyer, O. Bojar, A. Con­stantin, E. Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In <i>Pro­ceedings of the 45th Annual Meeting of the Associa­tion for Computational Linguistics Companion Vol­ume Proceedings of the Demo and Poster Sessions, </i>177-180, Prague, Czech Republic.</p><p>D. Marcu, W. Wang, A. Echihabi, K. Knight. 2006. SPMT: Statistical machine translation with syntact-ified target language phrases. In <i>Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, </i>44-52, Sydney, Australia.</p><p>F. J. Och, H. Ney. 2003. A systematic comparison of various statistical alignment models. <i>Computational Linguistics, </i>29(1):19-51.</p><p>F. J. Och. 2003. Minimum error rate training in sta­tistical machine translation. In <i>Proceedings of the 41st Annual Meeting of the Association for Compu­tational Linguistics, </i>160-167, Sapporo, Japan.</p><p>K. Papineni, S. Roukos, T. Ward, W.-J. Zhu. 2002. BLEU: a method for automatic evaluation of ma­chine translation. In <i>Proceedings of the 40th Annual Meeting of the Association for Computational Lin­guistics, </i>311-318, Philadelphia, PA.</p><p>A. Stolcke. 1995. An efficient probabilistic context-free parsing algorithm that computes prefix proba­bilities. <i>Computational Linguistics, </i>21(2).</p><p>H. Zhang, L. Huang, D. Gildea, K. Knight. 2006. Syn­chronous binarization for machine translation. In <i>Proceedings of the 2006 Human Language Technol­ogy Conference of the North American Chapter of the Association for Computational Linguistics, </i>256-263.</p><p>H. Zhang, C. Quirk, R. C. Moore, D. Gildea. 2008. Bayesian learning of non-compositional phrases with synchronous parsing. In <i>Proceedings ofACL-08: HLT, </i>97-105, Columbus, Ohio.</p><p>A. Zollmann, A. Venugopal. 2006. Syntax augmented machine translation via chart parsing. In <i>Proceed­ings of the 2006 Human Language Technology Con­ference of the North American Chapter of the As­sociation for Computational Linguistics, </i>138-141, New York City, NY.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">361</doubt></references></body></article>