<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="215"/><title>Probabilistic Inference for Machine Translation</title><author surname="Blunsom" givenname="Phil"><org  name="University of Edinburgh" country="United Kingdom" city="Edinburgh"/></author><author surname="Osborne" givenname="Miles"><org  name="University of Edinburgh" country="United Kingdom" city="Edinburgh"/></author></firstpageheader><frontmatter><p><b>Probabilistic Inference for Machine Translation</b></p><p><b>Phil Blunsom </b>and <b>Miles Osborne</b></p><p>School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh, EH8 9AB, UK {pblunsom,miles}@inf.ed.ac.uk</p></frontmatter><abstract>We advance the state-of-the-art for discrimi-natively trained machine translation systems by presenting novel probabilistic inference and search methods for synchronous gram­mars. By approximating the intractable space of all candidate translations produced by inter­secting an ngram language model with a synchronous grammar, we are able to train and decode models incorporating millions of sparse, heterogeneous features. Further, we demonstrate the power of the discriminative training paradigm by extracting structured syntactic features, and achieving increases in translation performance. </abstract></header><body><section number="1" title="Introduction"><p>The goal of creating statistical machine translation (SMT) systems incorporating rich, sparse, features over syntax and morphology has consumed much recent research attention. Discriminative approaches are widely seen as a promising technique, poten­tially allowing us to further the state-of-the-art. Most work on discriminative training for SMT has focussed on linear models, often with margin based algorithms (Liang et al., 2006; Watanabe et al., 2006), or rescaling a product of sub-models (Och, 2003; Ittycheriah and Roukos, 2007).</p><p>Recent work by Blunsom et al. (2008) has shown how translation can be framed as a probabilistic log-linear model, where the distribution over trans­lations is modelled in terms of a latent variable on derivations. Their approach was globally opti­mised and discriminative trained. However, a lan­guage model, an information source known to be crucial for obtaining good performance in SMT, was notably omitted. This was because adding a lan­guage model would mean that the normalising parti­tion function could no longer be exactly calculated, thereby preventing efficient parameter estimation.</p><p>Here, we show how language models can be incorporated into large-scale discriminative transla­tion models, without losing the probabilistic inter­pretation of the model. The key insight is that we can use Monte-Carlo methods to approximate the partition function, thereby allowing us to tackle the extra computational burden associated with adding the language model. This approach is theoreti­cally justified and means that the model contin­ues to be both probabilistic and globally optimised. As expected, using a language model dramatically increases translation performance.</p><p>Our second major contribution is an exploita­tion of syntactic features. By encoding source syn­tax as features allows the model to use, or ignore, this information as it sees fit, thereby avoiding the problems of coverage and sparsity associated with directly incorporating the syntax into the grammar translation gains using this approach.</p><doubt alpha="49.0" length="51" tooSmall="False" monospace="0.0">(Huang et al., 2006; Mi et al., 2008). We report on</doubt><p>We begin by introducing the synchronous gram­mar approach to SMT in Section 2. In Section 3 we define the parametric form of our model and describe techniques for approximating the intractable space of all translations for a given source sentence. In Section 4 we evaluate the abil­ity of our model to effectively estimate the highly dependent weights for the sparse features and real-valued language model. In addition we describe how</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">215</doubt><p><i>Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, </i>pages 215-223, Honolulu, October 2008. ©2008 Association for Computational Linguistics<page local="2" global="216"/></p><doubt alpha="21.1" length="19" tooSmall="False" monospace="0.0">S =&gt;  &lt;X,M . ,X, .&gt;</doubt><doubt alpha="50.9" length="55" tooSmall="False" monospace="0.0">X, =&gt; &lt;X2m Bf^W&amp;X3,X2X3late last night&gt;X2=&gt;  &lt;^, Brown&gt;</doubt><doubt alpha="58.2" length="67" tooSmall="False" monospace="0.0">X3=&gt;&lt;ÄX4SäX5, arrived inX5fromX4&gt; X4=&gt;&lt;±Ä , Shanghai&gt;X5=&gt;, Beijing&gt;</doubt><p><b>Figure 1. </b>An example SCFG derivation from a Chi­nese source sentence which yields the English sentence: <i>"Brown arrived in Shanghai from Beijing late last night."</i></p><p>our model can easily integrate rich features over source syntax trees and compare our training meth­ods to a state-of-the-art benchmark.</p></section><section number="2" title="Synchronous context free grammar"><p>A synchronous context free grammar (SCFG, (Lewis II and Stearns, 1968)) describes the gener­ation of pairs of strings. A string pair is generated by applying a series of paired context-free rewrite rules of the form, <i>X — (a, 7, </i>~), where <i>X </i>is a non­terminal, <i>a </i>and 7 are strings of terminals and non­terminals and ~ specifies a one-to-one alignment between non-terminals in <i>a </i>and <i>7. </i>In the context of SMT, by assigning the source and target languages to the respective sides of a SCFG it is possible to describe translation as the process of parsing the source sentence, while generating the target trans­lation (Chiang, 2007).</p><p>In this paper we only consider grammars extracted using the heuristics described for the Hiero SMT system (Chiang, 2007). Note however that our approach is general and could be used with other synchronous grammar transducers (e.g., (Galley et al., 2006)). SCFG productions can specify that the order of the child non-terminals is the same in both languages (a <i>monotone </i>production), or is reversed (a <i>reordering </i>production). Without loss of generality, here we add the restriction that non-terminals on the source and target sides of the grammar must have the same category. Figure 1 shows an example deriva­tion for Chinese to English translation.</p></section><section number="3" title="Model"><p>We start by defining a log-linear model for the con­ditional probability distribution over target transla­tions of a given source sentence. A sequence of SCFG rule applications which produce a translation from a source sentence is referred to as a <i>derivation, </i>and each translation may be produced by many dif­ferent derivations. As the training data only provides source and target sentences, the derivations are mod­elled as a latent variable.</p><p>The conditional probability of a derivation, d, for a target translation, e, conditioned on the source, f, is given by:</p><p>exPE<b>k Afc</b><i>Hk</i><i>(d, e, f </i>)</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(1)</doubt><doubt alpha="50.0" length="6" tooSmall="False" monospace="0.0">ZA(f )</doubt><doubt alpha="43.6" length="39" tooSmall="False" monospace="0.0">whereHk(d, e, f) = ^hk(f, r, q(r,d))(2)</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">red</doubt><p>Using Equation (1), the conditional probability of a target translation given the source is the sum over all of its derivations:</p><doubt alpha="47.4" length="19" tooSmall="False" monospace="0.0">PA(e|f)=£PA(d, e|f)</doubt><doubt alpha="55.6" length="9" tooSmall="False" monospace="0.0">deA(e,f )</doubt><p>where A(e, f) is the set of all derivations of the target sentence <b>e </b>from the source <b>f.</b></p><p>Here <i>k </i>ranges over the model's features, and A = {Ak} are the model parameters (weights for their corresponding features). The function <i>q(r, </i>d) returns the target ngram context, for a language model with order m, of rule r in derivation d. For a rule which spans the target words <i>) </i>and <i>targetjyield</i><i>(d) = </i><i>{t0, • </i>• • <i>,t</i><i>i}:</i></p><doubt alpha="37.5" length="8" tooSmall="False" monospace="0.0">q(r,d) =</doubt><doubt alpha="40.5" length="42" tooSmall="False" monospace="0.0">ti "'ti-\-rn —2~*rtj — rn-\-2'''tjti''' tj</doubt><p>if <i>j — i &gt; m </i>otherwise</p><p>The feature functions <i>h</i><i>k </i>are real-valued functions over the source and target sentences, and can include overlapping and non-independent features of the data. The features must decompose with the deriva­tion and the ngram context defined by the function q, as shown in Equation (2). The features can reference the entire source sentence coupled with each rule, <i>r, </i>and its target context, in a derivation.</p><p>By directly incorporating the language model context <i>q </i>into the model formulation, we will not be able to exactly compute the partition function Zv(f), which sums over all possible derivations.<page local="3" global="217"/> Even though a dynamic program over this space would still run in polynomial time, as shown by Chi­ang (2007), a packed chart representation of the par­tition function for the binary Hiero grammars used in this work would require O(n<footnote anchor="3"/>|T|4(m-1)) space,<footnote anchor="1"/>which is far too large to be practical.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">216</doubt><p>Instead we approximate the partition function using a sum over a large subset of the possible derivations (A(e, f)):</p><doubt alpha="45.5" length="33" tooSmall="False" monospace="0.0">ZA(f) »E     Eexp £ AkHk(d, e, f)</doubt><doubt alpha="47.6" length="21" tooSmall="False" monospace="0.0">ed€{cA(e,f)}k = ZA(f)</doubt><p>This model formulation raises the questions of what an appropriate <i>large subset </i>of derivations for training is, and how to efficiently calculate the sum over all derivations in decoding. In the following sections we elucidate and evaluate our solutions to these problems.</p><subsection number="3.1" title="Sampling Derivations"><p>The training and decoding algorithms presented in the following sections rely upon Monte-Carlo tech­niques, which in turn require the ability to draw derivation samples from the probability distribution defined by our log-linear model. Here we adapt previously presented algorithms for sampling from a PCFG (Goodman, 1998) for use with our syn­chronous grammar model. Algorithm 1 describes the algorithm for sampling derivations. The sampling algorithm assumes the pre-existance of a packed chart representation of all derivations for a given source sentence. The <i>inside algorithm </i>is then used to calculate the scores needed to define a multino­mial distribution over all partial derivations associ­ated with expanding a given child rule. These ini­tial steps are performed once and then an unlim­ited number of samples can be drawn by calling the recursive sample procedure. multi draws a sample from the distribution over rules for a given chart cell, children enumerates the chart cells connected to a rule as variables, and derivation is a recursive tree data structure for derivations. The algorithm is <b>Algorithm 1 </b>Top-down recursive derivation sam­pling algorithm. 1: <b>procedure </b>SAMPLE(X, <i>i,</i><i> </i><i>k)</i><i> </i>2:      <i>rule — </i>MULTl(inside_chart(X, <i>i,</i><i> </i>k))</p><footnote label="1">where |T| is the size of the terminal alphabet, i.e. the num­ber of unique English words.</footnote><doubt alpha="22.2" length="9" tooSmall="False" monospace="0.0">3:c = &lt;f&gt;</doubt><p>4:      <b>for </b><i>(child-category, </i><i>x,y)</i><i> </i>G CHILDREN(rule) 5: <i>c — c U </i><i>SAMPLE</i><i>(child_category, </i><i>x,</i><i> </i><i>y)</i> 7:      <b>return </b>DERIVATION(X, children) 8: <b>end procedure</b></p><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">do</doubt><doubt alpha="66.7" length="9" tooSmall="False" monospace="0.0">6:end for</doubt><p>first called on a category and chart cell spanning the entire chart, and then proceeds top down by using the function MULTI to draw the next rule to expand from the distribution defined by the inside scores.</p></subsection><subsection number="3.2" title="Approximate Inference"><p>Approximating the partition function with Zv(f) could introduce biases into inference and in the fol­lowing discussion we describe measures taken to minimise the effects of the approximation bias.</p><p>An obvious approach to approximating the parti­tion function, and the feature expectations required for calculating the derivative in training, is to use the packed chart of derivations produced by running the cube pruning beam search algorithm of Chiang (2007) on the source sentence. In this case Zv(f) includes all the derivations that fall within the cube pruning beam, hopefully representing the majority of the probability mass. We denote the partition function estimated with this cube beam approxima­tion as <i>Z</i><i>cA</i><i>(f </i>). This approach has the advantage of using the same beam search dynamic program dur­ing training as is used for decoding. As the approxi­mated partition function does not contain all deriva­tions, it is possible that some, or all, of the deriva­tions of the reference translation from the parallel corpus may be excluded. We must therefore intersect the packed chart built from the cube beam with that of the reference derivations to ensure consistency.</p><p>Although, as would be done using cube-pruning, it would seem intuitively sensible to approximate the partition function using only high probability derivations, it is possible that doing so will bias our model in odd ways. The space of derivations contained within the beam will be tightly clustered about a maximum, and thus a model trained with such an approximation will only see a very small<page local="4" global="218"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">217</doubt><p>Alles / Everything</p><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">(a)</doubt><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">(c)</doubt><doubt alpha="66.7" length="9" tooSmall="False" monospace="0.0">und / and</doubt><p>jedes / anything</p><doubt alpha="60.0" length="5" tooSmall="False" monospace="0.0">ist /</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">is</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">2</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">3</doubt><doubt alpha="40.0" length="5" tooSmall="False" monospace="0.0">.X-*r</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">4</doubt><doubt alpha="71.4" length="7" tooSmall="False" monospace="0.0">Alles /</doubt><p>Everything</p><p>jedes / everyone vorstellbar / conceivable</p><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">.X*r</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">X</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">\</doubt><p>VranceivaMe/</p><p><b>Figure 2. </b>A German-English translation example of building <i>Z</i><b><i>iam(f</i></b>) from samples. (a) Two sample derivations are drawn from the model, (b) these samples are then combined into a packed representation, here represented by a hypergraph with target translations elided for a bigram language model. The derivation in (c) is contained within the hypergraph even though it was never explicitly inserted.</p><p>part of the overall distribution, possibly leading it astray. Consider the example of a language model feature: as this is a very strong indicator of transla­tion quality, we would expect all derivations within the beam to have a similar (high) language model score, thereby robbing this feature ofits discriminat­ing power. However if our model could also see the low probability derivations it would be clear that this feature is indeed very strongly correlated with good translations. Thus a good approximation of the space of derivations is one that includes both good and bad examples, not just a cluster around the maximum.</p><p>A principled solution to the problem of approx­imating the partition function would be to use a Markov Chain Monte Carlo (MCMC) sampler to estimate the sum with a large number of samples. Most of the sampled derivations would be in the high probability region of the distribution, however there would also be a number of samples drawn from the rest of the space, giving the model a more global view of the distribution, avoiding the pit­falls of the narrow view obtained by a beam search. Although effective, the computational cost of such an approach is prohibitive as we would need to draw hundreds of thousands of samples to obtain conver­gence, for every training iteration.</p><p>Here we mediate between the computational advantages of a beam and the broad view of the dis­tribution provided by sampling. Using the algorithm outlined in Section 3.1 we draw samples from the distribution of derivations and then insert these sam­ples into a packed chart representation. This process is illustrated in Figure 2. The packed chart created by intersecting the sample derivations represents a space of derivations much greater than the original samples. In Figure 2 the chart is built from the first two sampled derivations, while the third derivation can be extracted from the chart even though it was never explicitly entered. This approximation of the partition function (denoted <i>Z</i><i>sA</i><i>m(f</i>)) allows us to build an efficient packed chart representation of a large number of derivations, centred on those with high probability while still including a significant representation of the low probability space. Deriva­tions corresponding to the reference can be detected during sampling and thus we can build the chart for the reference derivations at the same time as the one approximating the partition function. This could lead to some, or none of, the possible ref­erence derivations being included, as they may not have been sampled. Although we could intersect all of the reference derivations with the sampled chart, this could distort the distribution over derivations, and we believe it to be advantageous to keep the distributions between the partition function and ref­erence charts consistent.<page local="5" global="219"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">218</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">S</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">5</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">6</doubt><p>Both of the approximations proposed above, ZAb(f) and <i>Z']A</i><i>'m(f</i>), rely on the pre-existence of a trained translation model in order to either guide the cube-pruning beam, or define the probability distri­bution from which we draw samples. We solve this chicken and egg problem by first training an exact translation model <i>without </i>a language model, and then use this model to create the partition function approximations for training <i>with </i>a language model. We denote the distribution without a language model as <i>p-LM</i>(e|f) and that with as <i>p+LM</i>(e|f).</p><p>A final training problem that we need to address is the appropriate initialisation of the model param­eters. In theory we could simply randomly initialise A for p+LM(e|f), however in practice we found that this resulted in poor performance on the develop­ment data. This is due to the complex non-convex optimisation function, and the fact that many fea­tures will fall outside the approximated charts result­ing in random, or zero, weights in testing. We intro­duce a novel solution in which we use the Gaus­sian prior over model weights to tie the exact model trained without a language model, which assigns sensible values to all rule features, with the approx­imated model. The prior over model parameters for p+LM(e|f) is defined as:</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">{\\2</doubt><doubt alpha="61.5" length="13" tooSmall="False" monospace="0.0">p+LM(Ak) oce~</doubt><p>Here we have set the mean parameters of the Gaus­sian distribution for the approximated model to those learnt for the exact one. This has the effect that any features that fall outside the approximated model will simply retain the weight assigned by the exact model. While for other feature weights the prior will penalise substantial deviations away from A-LM, essentially encoding the intuition that the rule rule parameters should not change substantially with the inclusion of language model features.</p><p>This results in the following log-likelihood objec­tive and corresponding gradient:</p><doubt alpha="61.3" length="31" tooSmall="False" monospace="0.0">L=£log p+LM(et|fi)+£logp+LM(Ak)</doubt><p><b>(e</b>i<b>,f</b>i<b>)GD </b>k</p><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">dL</doubt><doubt alpha="61.5" length="39" tooSmall="False" monospace="0.0">dAk=EP++LM(d|ei,fi)[hk]—EP+LM(e|fi)thk]</doubt><doubt alpha="80.0" length="5" tooSmall="False" monospace="0.0">Ak—Ak</doubt><doubt alpha="50.0" length="2" tooSmall="False" monospace="0.0">a2</doubt></subsection><subsection number="3.3" title="Decoding"><p>As stated in Equation 3 the probability of a given translation string is calculated as the sum of the probabilities of all the derivations that yield that string. In decoding, where the reference translation is not known, the exact calculation of this summa­tion is NP-Hard. This problem also arises in mono­lingual parsing with probabilistic tree substitution grammars and has been tackled in the literature using Monte-Carlo sampling methods (Chappelier and Rajman, 2000). Their approach is directly appli­cable to our SCFG decoding problem and we can use Algorithm 1 to draw sample translation derivations for the source sentence. The probability of a trans­lation can be calculated simply from the number of times a derivation that yields it was sampled, divided by the total number of samples. For the p-LM(e|f ) model we can build the full chart of all possible derivations and thus sample from the true distribu­tion over derivations. For the p+LM(e|f) model we suffer the same problem as in training and cannot build the full chart. Instead a chart is built using the cube-pruning algorithm with a wide beam and we then draw samples from this chart. Although sampling from a reduced chart will result in biased samples, in Section 4 we show this approach to be effective in practice.<footnote anchor="2"/> In Section 4 we compare our sampling approach to the heuristic beam search pro­posed by Blunsom et al. (2008).</p><p>It is of interest to compare our proposed decoding algorithms to minimum Bayes risk (MBR) decoding (Kumar and Byrne, 2004), a commonly used decod­ing method. From a theoretical standpoint, the sum­ming of derivations for a given translation is exactly equivalent to performing MBR with a 0/1 loss func­tion over derivations.<page local="6" global="220"/> From a practical perspective, MBR is normally performed with <i>B</i><i>leu </i>as the loss and approximated using n-best lists. These n-best lists are produced using algorithms tuned to remove multiple derivations of the same translation (which have previously been seen as undesirable). However, it would be simple to extend our sampling based decoding algorithm to calculate the MBR estimate using <i>Bleu</i>, in theory providing a lower variance estimate than attained with n-best lists.</p><footnote label="2">We have experimented with using a Metropolis Hastings sampler, with p —LM (e|f) as the proposal distribution, to sam­ple from the true distribution with the language model. Unfor­tunately the sample rejection rate was very high such that this method proved infeasibly slow.</footnote><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">219</doubt></subsection></section><section number="4" title="Evaluation"><p>We evaluate our model on the IWSLT 2005 Chinese to English translation task (Eck and Hori, 2005), using the 2004 test set as development data for tuning the hyperparameters and MERT training the benchmark systems. The statistics for this data are presented in Table 1.<footnote anchor="3"/> The training data made avail­able for this task consisted of 40k pairs of tran­scribed utterances, drawn from the travel domain. The development and test data for this task are some­what unusual in that each sentence has a single human translated reference, and fifteen paraphrases of this reference, provided by monolingual anno-tators. Model performance is evaluated using the standard <i>Bleu </i>metric (Papineni et al., 2002) which measures average n-gram precision, <i>n &lt; </i>4, and we use the NIST definition of the brevity penalty for multiple reference test sets. We provide evaluation against both the entire multi-reference sets, and the single human translation.</p><p>Our translation grammar is induced using the standard alignment and rule extraction heuristics used in hierarchical translation models (Chiang, 2007).<footnote anchor="4"/> As these heuristics aren't based on a genera­tive model, and don't guarantee that the target trans­lation will be reachable from the source, we discard those sentence pairs for which we cannot produce a derivation, leaving 38,405 sentences for training.</p><p>Our base model contains a single feature for each rule which counts the number of times it appeared in a particular derivation. For models which include a language model, we train a standard Kneser-Ney tri-gram model on the target side of the training corpus. We also include a word penalty feature to compen­sate for the shortening effect of the language model. In total our model contains 2.9M features.</p><footnote label="3">Development and test set statistics are for the single human translation reference.</footnote><footnote label="4">With the exception that we allow unaligned words at the boundary of rules. This improves training set coverage.</footnote><p>The aims of our evaluation are: (1) to deter­mine that our proposed training regimes are able to realise performance increase when training sparse rule features and a real valued language model fea­ture together, (2) that the model is able to effectively use rich features over the source sentence, and (3) to confirm that our model obtains performance com­petitive with the current state-of-the-art.</p><subsection number="4.1" title="Inference and Decoding"><p>We have described a number of modelling choices which aim to compensate for the training biases introduced by incorporating a language model fea­ture through approximate inference. Our a priori knowledge from other SMT systems suggests that incorporating a language model should lead to large increases in <i>Bleu </i>score. In this evaluation we aim to determine whether our training regimes are able to realises these expected gains.</p><p>Table 2 shows a matrix of development <i>Bleu </i>scores achieved by varying the approximation of the partition function in training, and varying the decod­ing algorithm. If we consider the vertical axis we can see that the sampling method for approximat­ing the partition function has a small but consistent advantage over using the cube-pruning beam. The charts produced by the sampling approach occupy roughly half the disc space as those produced by the beam search, so in subsequent experiments we present results using the ZAam(f ) approximation.</p><p>Comparing the decoding algorithms on the hori­zontal axis we can reconfirm the findings of Blun-som et al. (2008) that the max-translation decod­ing outperforms the Viterbi max-derivation approx­imation. It is also of note that this <i>B</i><i>leu </i>increase is robust to the introduction of the language model feature, assuaging fears that the max-translation approach may have been doing the job of the lan­guage model. We also compare using Monte-Carlo sampling for decoding with the previously pro­posed heuristic beam search algorithm. The differ­ence between the two algorithms is small, however<page local="7" global="221"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">220</doubt><p><b>Table 1. </b>IWSLT Chinese to English translation corpus statistics.</p><p><b>Table 2. </b>Development set results for varying the approximation of the partition function in training, ZX°(f ) vs. <i>Z</i><b><i>lam(f </i></b>), and decoding using the Viterbi max-derivation algorithm, or the max-translation algorithm with either a beam approxi­mation or Monte-Carlo sampling.</p><p>we feeling the sampling approach is more theoreti­cally justified and adopt it for our later experiments.</p><p>The most important result from this evaluation is that both our training regimes realise substantial gains from the introduction of the language model feature. Thus we can be confident that our model is capable of modelling the distribution over trans­lations even when the space over all derivations is intractable to dynamically program exactly.</p></subsection><subsection number="4.2" title="A Discriminative Syntactic Translation Model"><p>In the previous sections we've described and evalu­ated a statistical model of translation that is able to estimate a probability distribution over translations using millions of sparse features. A prime motiva­tion for such a model is the ability to define com­plex features over more than just the surface forms of the source and target strings. There are limit­less options for such features, and previous work has focused on defining token based features such as part-of-speech and morphology (Ittycheriah and Roukos, 2007). Although such features are applica­ble to our model, here we attempt to test the model's ability to incorporate complex features over source-side syntax trees, essentially subsuming and extend­ing previous work on tree-to-string translation mod­els (Huang et al., 2006; Mi et al., 2008).</p><p>We first parse the source side of our training, development and test corpora using the Stanford parser.<footnote anchor="5"/> Next, while building the synchronous charts required for training, whenever a rule is used in a derivation a feature is activated which captures: (1) the constituent spanning the rule's source side in the syntax tree (if any) (2) constituents spanning any variables in the rule, and (3) the rule's target side surface form. Figure 3 illustrates this process.</p><footnote label="5">http://nlp.stanford.edu/software/lex-parser.shtml</footnote><p>These syntactic features are equivalent to the grammar rules extracted for tree-to-string translation systems. The key difference in our model is that the source syntax tree is treated as conditioning context and it's information encoded as features, thus this information can be used or ignored as the model sees fit. This avoids the problems associated with explic­itly encoding the source syntax in the grammar, such as sparsity and overly constraining the model. In addition we could easily incorporate features over multiple source trees, for example mixing labelled syntax trees with dependency graphs.</p><p>We limit the extraction of syntactic features to those that appear in at least two training derivations, giving a total of 4.2M syntactic features, for an over­all total of 7.1M features.</p></subsection><subsection number="4.3" title="Discussion"><p>Table 3 shows the results from applying our described models to the test set. We benchmark our results against a model (Hiero) which was directly trained to optimise <i>Bleunist </i>using the standard</p><p>MERT algorithm (Och, 2003) and the full set of translation and lexical weight features described for the Hiero model (Chiang, 2007). As well as<page local="8" global="222"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">221</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Training</p></td><td class="cell"><p>Development</p></td><td class="cell"><p>Test</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Chinese English</p></td><td class="cell"><p>Chinese English</p></td><td class="cell"><p>Chinese English</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Utterances Segments/Words Av. Utterances Length Longest Utterance</p></td><td class="cell"><p>38405 317621 353116 8 9 55 68</p></td><td class="cell"><p>500</p><p>3464 3752 6 7 58 62</p></td><td class="cell"><p>506</p><p>3784 3823 7 7 61 56</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Model</p></td><td class="cell"><p>Max-derivation</p></td><td class="cell"><p>Max-translation(Beam)</p></td><td class="cell"><p>Max-translation(Sampling)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>p-LM </i>(e|f )</p></td><td class="cell"><p>31.0</p></td><td class="cell"><p>32.5</p></td><td class="cell"><p>32.6</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>p+LM</i>(e|f) (Z£*(f))</p></td><td class="cell"><p>39.1</p></td><td class="cell"><p>39.8</p></td><td class="cell"><p>39.8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>p+LM</i>(e|f) <i>(Z</i><b><i>iam(f</i></b>))</p></td><td class="cell"><p>39.9</p></td><td class="cell"><p>40.5</p></td><td class="cell"><p>40.6</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">(a)</doubt><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">(c)</doubt><p><b>Table 3. </b>Test set results.</p><p><b>(b)   </b>Example Derivation:</p><doubt alpha="55.7" length="97" tooSmall="False" monospace="0.0">(Step 1) X1-&gt;&lt;ffirf5&amp;i&amp;£h,currency exchange office&gt;(Step 2) X2-&gt; &lt; [X1]ft1M?,Where is the [X1] ?&gt;</doubt><doubt alpha="61.1" length="54" tooSmall="False" monospace="0.0">Example Syntax feature =    ( SQ )&gt;Where is the [X1] ?</doubt><p>for Step 2</p><doubt alpha="55.6" length="9" tooSmall="False" monospace="0.0">( NHft1M?</doubt><doubt alpha="50.0" length="2" tooSmall="False" monospace="0.0">X1</doubt><p><b>Figure 3. </b>Syntax feature example: For the parsed source and candidate translation (a), with the derivation (b), we extract the syntax feature in (c) by combining the grammar rule with the source syntax of the constituents contained within that rule.</p><p><b>Table 4. </b>Example test set output produced when: not using a language model, using a language model, also using syntax, output optimised using MERT and finally the reference</p><p>Where is   the   currency   exchange office ?</p><p><i>Bleunist </i>(brevity penalty uses the shortest ref­erence), we also include results from the IBM <i>(Bleuibm</i>) metric (brevity penalty uses the closest reference), and using only the actual human transla­tion in the test set, not the monolingual paraphrase multiple references <i>(Bleuhumanref </i>).</p><p>The first result of interest is that we see an increase in performance through the incorporation of the source syntax features. This is an encourag­ing result as the transcribed speech source sentences are well out of the domain of the data on which the parser was trained, suggesting that our model is able to sift the good information from the noisy in the unreliable source syntax trees. Table 4 shows illus­trative system output on the test set.</p><p>On the <i>Bleunist </i>metric we see that our mod­els under-perform the MERT trained system. We hypothesise that this is predominately due to the interaction of the brevity penalty with the unusual nature of the multiple paraphrase reference train­ing and development data. Their performance is however quite consistent across the different inter­pretations of the brevity penalty (NIST vs. IBM). This contrasts with the MERT trained model, which clearly over-fits to the NIST metric that it was trained on and underperforms our models when eval­uated on the single human test translations. If we directly compare the brevity penalties of the MERT model (0.868) and our discriminative model incor­porating source syntax (0.942), on the these single references, we can see that the MERT training has optimised to the shortest paraphrase reference.<page local="9" global="223"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">222</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Model</p></td><td class="cell"><p><i>Bleu nist</i></p></td><td class="cell"><p><i>Bleu ibm</i></p></td><td class="cell"><p><i>B leu </i>HumanRef</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Palm </i>(e|f )</p></td><td class="cell"><p>33.5</p></td><td class="cell"><p>35.2</p></td><td class="cell"><p>25.2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>P+alm</i><i> </i>(e|f )</p></td><td class="cell"><p>44.6</p></td><td class="cell"><p>44.6</p></td><td class="cell"><p>31.2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>pALM (e|f ) + syntax</p></td><td class="cell"><p>45.3</p></td><td class="cell"><p>45.2</p></td><td class="cell"><p>31.8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MERT <i>(Bleunist</i>)</p></td><td class="cell"><p>46.2</p></td><td class="cell"><p>44.5</p></td><td class="cell"><p>30.2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Source</p></td><td class="cell"><p><b>m </b><b><i>à ± </i></b><b>&amp; w    </b><b><i>m </i></b><b>n </b><b><i>* </i></b><b>m - &amp; m </b><b><i>m    </i></b><b>m »</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PaLM (e|f ) PALM (e|f ) pALM (e|f ) + syntax</p></td><td class="cell"><p>don 't have enough bag on me change please go purchase a new by plane .</p><p>i have enough money to buy a new one by air .</p><p>i don 't have enough money to buy a new airline ticket.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MERT <i>(Bleunist</i>)</p></td><td class="cell"><p>i don 't have enough money to buy a new ticket.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Reference</p></td><td class="cell"><p>i do n't have enough money with me to buy a new airplane ticket.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>From these results it is difficult to draw any hard conclusions on the relative performance of the dif­ferent training regimes. However we feel confident in claiming that we have achieved our goal of train­ing a probabilistic model on millions of sparse fea­tures which obtains performance competitive with the current state-of-the-art training algorithm.</p></subsection></section><section number="5" title="Conclusion"><p>In this paper we have shown that statistical machine translation can be effectively modelled as a well posed machine learning task. In doing so we have described a model capable of estimating a probabil­ity distribution over translations using sparse com­plex features, and achieving performance compara­ble to the state-of-the-art on standard metrics. With further work on scaling these models to large data sets, and engineering high performance features, we believe this research has the potential to provide sig­nificant increases in translation quality.</p></section><section title="Acknowledgements"><p>The authors acknowledge the support of the EPSRC grant EP/D074959/1.</p></section><references><p>Phil Blunsom, Trevor Cohn, and Miles Osborne. 2008. A discriminative latent variable model for statistical machine translation. In <i>Proc. of the 46th Annual Con­ference of the Association for Computational Linguis­tics: Human Language Technologies (ACL-08:HLT), </i>pages 200-208, Columbus, Ohio, June.</p><p>Jean-Cedric Chappelier and Martin Rajman. 2000. Monte-carlo sampling for np-hard maximization prob­lems in the framework of weighted parsing. In <i>NLP '00: Proceedings of the Second International Confer­ence on Natural Language Processing, </i>pages 106­117, London, UK.</p><p>David Chiang. 2007. Hierarchical phrase-based transla­tion. <i>Computational Linguistics, </i>33(2):201-228.</p><p>Matthias Eck and Chiori Hori. 2005. Overview of the IWSLT 2005 evaluation campaign. In <i>Proc. ofthe International Workshop on Spoken Language Trans­lation, </i>Pittsburgh, October.</p><p>Michel Galley, Jonathan Graehl, Kevin Knight, Daniel Marcu, Steve DeNeefe, Wei Wang, and Ignacio Thayer.   2006.   Scalable inference and training of context-rich syntactic translation models. In <i>Proc. of the 44th Annual Meeting of the ACL and 21st International Conference on Computational Linguis­tics (COLING/ACL-2006), </i>pages 961-968, Sydney, Australia, July.</p><p>Joshua T. Goodman. 1998. <i>Parsing inside-out. </i>Ph.D. thesis, Cambridge, MA, USA. Adviser-Stuart Shieber.</p><p>Liang Huang, Kevin Knight, and Aravind Joshi. 2006. Statistical syntax-directed translation with extended domain of locality. In <i>In Proceedings of the 7th Bien­nial Conference ofthe Association for Machine Trans­lation in the Americas (AMTA), </i>Boston, MA.</p><p>Abraham Ittycheriah and Salim Roukos. 2007. Direct translation model 2. In <i>Proc. ofthe 7th International Conference on Human Language Technology Research and 8th Annual Meeting ofthe NAACL (HLT-NAACL 2007), </i>pages 57-64, Rochester, USA.</p><p>Shankar Kumar and William Byrne. 2004. Minimum bayes-risk decoding for statistical machine translation.</p><p>In <i>Proc. ofthe 4th International Conference on Human Language Technology Research and 5th Annual Meet­ing ofthe NAACL (HLT-NAACL 2004), </i>pages 169­176.</p><p>Philip M. Lewis II and Richard E. Stearns. 1968. Syntax-directed transduction. <i>J. ACM, </i>15(3):465-488.</p><p>Percy Liang, Alexandre Bouchard-Cote, Dan Klein, and Ben Taskar. 2006. An end-to-end discriminative approach to machine translation. In <i>Proc. of the 44th Annual Meeting of the ACL and 21st Inter­national Conference on Computational Linguistics (COLING/ACL-2006), </i>pages 761-768, Sydney, Aus­tralia, July.</p><p>Haitao Mi, Liang Huang, and Qun Liu. 2008. Forest-based translation. In <i>Proc. ofthe 46th Annual Confer­ence ofthe Association for Computational Linguistics: Human Language Technologies (ACL-08:HLT), </i>pages 192-199, Columbus, Ohio, June.</p><p>Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In <i>Proc. </i><i>ofthe</i><i> 41st Annual Meeting of the ACL (ACL-2003), </i>pages 160­167, Sapporo, Japan.</p><p>Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic eval­uation of machine translation. In <i>Proc. </i><i>ofthe</i><i> 40th Annual Meeting of the ACL and 3rd Annual Meeting of the NAACL (ACL-2002), </i>pages 311-318, Philadelphia, Pennsylvania.</p><p>Taro Watanabe, Hajime Tsukada, and Hideki Isozaki. 2006. Left-to-right target generation for hierarchical phrase-based translation. In <i>Proc. </i><i>ofthe</i><i> 44th Annual Meeting of the ACL and 21st International Conference on Computational Linguistics (COLING/ACL-2006), </i>pages 777-784, Sydney, Australia.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">223</doubt></references></body></article>