<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="38"/><title>Context-Sensitive Syntactic Source-Reordering by Statistical Transduction</title><pubinfo>Proceedings of the 5th International Joint Conference on Natural Language Processing,pages 38-46, Chiang Mai, Thailand, November 8 - 13, 2011. ©2011 AFNLP</pubinfo><author surname="Khalilov" givenname="Maxim"><org  name="Union University" country="USA" city="Jackson"/></author><author surname="Sima'an" givenname="Khalil"><org  name="University of Amsterdam" country="The Netherlands" city="Amsterdam"/></author></firstpageheader><frontmatter><p><b>Context-Sensitive Syntactic Source-Reordering by Statistical Transduction</b></p><p><b>Maxim Khalilov* </b>and <b>Khalil Sima'an</b></p><p>Institute for Logic, Language and Computation University of Amsterdam P.O. Box 94242 1090 GE Amsterdam, The Netherlands (m.khalilov,k.simaan}@uva.nl</p></frontmatter><abstract>How well can a phrase translation model per­form if we permute the source words to fit tar­get word order as perfectly as word alignment might allow? And how well would it perform if we limit the allowed permutations to ITG-like tree-transduction operations on the source parse tree? First we contribute oracle results showing great potential for performance im-provementby source-reordering, ranging from 1.5 to 4 BLEU points depending on language pair. Although less outspoken, the potential of tree-based source-reordering is also signif­icant. Our second contribution is a source re­ordering model that works with two kinds of tree transductions: the one permutes the order of sibling subtrees under a node, and the other first deletes layers in the parse tree in order to exploit sibling permutation at the remaining levels.The statistical parameters of the model we introduce concern individual tree trans-ductions <i>conditioned on contextualfeatures </i>of the tree resulting from all preceding transduc-tions. Experiments in translating from En­glish to Spanish/Dutch/Chinese show signifi­cant improvements of respectively 0.6/1.2/2.0 BLEU points. </abstract></header><body><section number="1" title="Motivation"><p>Word order differences between languages are a major challenge in Machine Translation (MT). Phrase-based Statistical Machine Translation (PB-SMT) (Och and Ney, 2004; Zens et al., 2002; Koehn</p><p>^Currently, the first author is employed by TAUS B.V., Am­sterdam (The Netherlands).</p><p>et al., 2003) deals with word order differences in two subcomponents of a translation model. Firstly, us­ing the local word reordering implicitly encoded in phrase pairs. Secondly, using an explicit reorder­ing model which may reorder target phrases rela­tive to their source sides, e.g., as a monotone phrase sequence generation process with the possibility of swapping neighboring phrases (Tillman, 2004).</p><p>Arguably, local phrase reordering models cannot account for long-range reordering phenomena, e.g., (Chiang, 2005; Chiang, 2007). Hierarchical mod­els of phrase reordering employ synchronous gram­mars or tree transducers, e.g., (Wu and Wong, 1998; Chiang, 2005). These models explore a more var­ied range of reordering phenomena, e.g., defined by at most inverting the order of sibling subtrees un­der each node in binary source/target trees (akin to ITG (Wu and Wong, 1998)).</p><p>Undoubtedly, the word order of source and tar­get sentences is intertwined with the lexical choices on both side. Statistically speaking, however, one may first select a target word order given the source only, and then choose target words given the selected target word order and source words. One applica­tion of this idea is known as source reordering (or -permutation), e.g., (Collins et al., 2005; Xia and McCord, 2004; Wang et al., 2007; Li et al., 2007; Khalilov and Sima'an, 2010). Briefly, the words of the source string s are reordered to minimize word order differences with the target string t, leading to the source permuted string <i>s. </i>Presumably, a stan­dard PBSMT system trained to translate from <i>s </i>to t should have an easier task than translating directly from    to t.  The source reordering part,    to <i>s,</i> can be realized in various ways and may manipu­late morpho-syntactic parse trees of s, e.g., (Collins et al., 2005; Xia and McCord, 2004; Li et al., 2007).<page local="2" global="39"/></p><p>It may seem that source reordering should provide only limited improvement over the standard PB-SMT approach. The literature reports mixed perfor­mance improvements for different language pairs,</p><doubt alpha="50.5" length="99" tooSmall="False" monospace="0.0">e.g., (Collins et al., 2005; Xia and McCord, 2004; Wang et al., 2007; Li et al., 2007; Khalilov and</doubt><p>Sima'an, 2010). But what is the <i>potential improve­ment </i>of source reordering? We contribute experi­ments measuring oracle performance improvement for English to Dutch/Spanish/Chinese translations. Beside string-driven oracles, we report results using ITG-like transductions over a single syntactic parse tree of s. Our results confirm that reordering a sin­gle syntactic tree could be insufficient (e.g., (Huang et al., 2009)), yet they show substantial potential.</p><p>Our second contribution is a novel source reorder­ing model that manipulates the source parse tree with two kinds of tree transduction operators: the one permutes the order of sibling subtrees under a node, and the other first abolishes layers in the parse tree in order to exploit sibling permutation at the re­maining levels. The latter is the opposite of parse bi-narization using Expectation-Maximization (Huang et al., 2009). We use Maximum-Entropy training (Berger et al., 1996) to learn a sequence of tree transductions, each conditioned on contextual fea­tures <i>in tree resulting from outcome of the preced­ing transduction. </i>The conditioning on the outcome of preceding transductions is a departure from ear­lier approaches at learning independent source per­mutation steps, e.g., (Tromble and Eisner, 2009; Visweswariah et al., 2010).</p><p>The aim for the rest of this paper is firstly, to quantify the potential performance improvement of a standard PBSMT system if preceded by source reordering and secondly, to show that statistical Markov approach to tree transduction, where the probability of each transduction step is conditioned on the outcome of preceding steps, can improve the quality of PBSMT output significantly.</p></section><section number="2" title="Source-Reordering: Framework"><p>We start out from a word-aligned parallel corpus, consisting of triples (s, a, t), a source s, target t and word alignment a. Source reordering assumes that a permutation of s, called <i>s, </i>is first generated with a model Pr <i>(s </i>| s) followed by a phrase translation model <i>Pt(t </i>| <i>s). </i>The desired permutation <i>s </i>is one that has minimum word order divergence from t, i.e., when word-aligned again with t would have least number of crossing alignments.</p><p>Practically, the original parallel corpus {(s, a, t)} is split to two parallel corpora: (1) a source-to-permutation parallel corpus (consisting of (s, a, s<i>s)) </i>and (2) a permutation-to-target parallel corpus (con­sisting of <i>(gs,a,t)), </i>where <i>g s </i>is the output of a source reordering model (guessing at <i>s), </i>and a<i>s </i>re­sults from automatically word aligning (g <i>s, </i>t). The latter parallel corpus is used for training a phrase-based translation system <i>Pt(t </i>| g s), while the for­mer corpus is used for training a source reordering model <i>Pr</i><i> </i><i>(s</i><i> </i>| s ).  The problem of permuting the</p><doubt alpha="66.7" length="15" tooSmall="False" monospace="0.0">SBARIN    NP VP</doubt><p><b>that PRP VBP</b></p><p>Figure 1 : Example crossing alignments and long-distance reordering using a source parse tree.</p><p>source string to unfold the crossing alignments is computationally intractable (see (Tromble and Eis­ner, 2009)). However, various constraints can be made on unfolding the crossing alignments in a. A common approach is to assume a binary parse tree for the source string, and define a set of eligible per­mutations by binary ITG transductions. This defines permutations resulting from at most inverting pairs of children under nodes of the source tree. Fig­ure 1 exhibits a long-range reordering of the verb in English-to-Dutch translation: inverting the order of the children of the VP node would unfold the crossing alignment. However, crossing alignments represented as non-constituents cannot be resolved. This difficulty can be circumvented by employing multiple alternative parse trees, by applying heuris­tic transforms (e.g., binarization) to the tree to fit the alignments (Wang et al., June 2010), or by defining new local transductions, on top child permuta­tion (ITG) as we do next.<page local="3" global="40"/></p></section><section number="3" title="Existing work on source permutation"><p>Source reordering has been shown useful for PB-SMT for a wide variety of language pairs with high mutual word order disparity (Collins et al., 2005; Popovic' and Ney, 2006; Zwarts and Dras, 2007; Xia and McCord, 2004). In Costa-jussà and Fonol-losa (2006) statistical word classes as well as POS tags are used as patterns for reordering the input sen­tences and producing a new bilingual pair.</p><p>A rather popular class of source reordering al­gorithms involves syntactic information and aims at minimizing the need for reordering during trans­lation by permuting the source sentence (Collins</p><doubt alpha="51.1" length="45" tooSmall="False" monospace="0.0">et al., 2005; Wang et al., 2007; Khalilov and</doubt><p>Sima'an, 2010; Li et al., 2007). Some systems perform source permutation using a set of hand­crafted rules (Collins et al., 2005; Wang et al., 2007; Ramanathan et al., 2008), others make use of automatically learned reordering patterns extracted from the plain training data, the corresponding parse or dependency trees and the alignment ma­trix (Visweswariah et al., 2010).</p><p>Inspiring this work, source reordering as a pre-translation step is viewed as a word permutation learning problem in Tromble and Eisner (2009) and Li et al. (2007). The space of permutations is ap­proached efficiently using a binary ITG-like syn­chronous context-free grammar put on the parallel data. Similarly, a local ITG-based tree transducer with contextual conditioning is used in Khalilov and Sima'an (2010) and Li et al. (2007), and prelimi­nary experiments on a single language pair show im­proved performance.</p><p>Particularly, the model in (Li et al., 2007) is ex­plicitly aimed at long-distance reorderings (English-Chinese), prunes the alignment matrix gradually to fit the source syntactic parse and employs Maximum-Entropy modeling to choose the optimal local ITG-like permutation step ofsistersubtrees but interleaves that step with a translation step. The model which we present in Section 2 differs substan­tially from (Li et al., 2007) and other earlier work because it (1) incorporates other kinds of tree trans-duction operations than those promoted by ITG, and (2) works with the unmodified alignment matrix but learns reorderings only from those alignments that are consistent with the tree, thereby avoiding the ef­fects of heuristics for pruning alignments to fit the tree-structure, e.g., (Li et al., 2007).</p><p>In this paper we take the idea of learning source permutation one step further along a few dimen­sions. We show the utility of other kinds of tree transduction operations, besides those promoted by ITG, stress the importance of using a wide range of conditioning context features during learning, and report oracle and test results on <i>three </i>language pairs.</p><p>The majority of existing work reports encourag­ing performance improvements by source reorder­ing. Next we aim at quantifying the potential im­provement by oracle source reordering at the string level, if all permutations were to be allowed, and at the source syntactic tree level, by limiting the per­mutations with two kinds of local transductions.</p></section><section number="4" title="Oracle source reordering results"><p>Source reordering for PBSMT assumes that permut­ing the source words to minimize the order differ­ences with the target sentence could improve trans­lation performance. However, the question "how much?" is rarely asked. Here, we attempt answering this question with a set of oracle systems<footnote anchor="1"/>, in which we perform unfolding operations on the crossing links in alignment a (estimated between corpora s and t) that leads to a more monotone alignment <i>s</i>a (between s<i>s, </i>which is a permutation of s, and t). We introduce a set of tree-based constraints that control the unfolding of alignment crossings. We measure the impact of (un)folded alignment crossings on the performance of the PBSMT system (see Table 1).</p><p><b>Oracle String. </b>This method scans the alignment <i>a </i>from left-to-right and unfolds all the crossing links between bilingual phrases <i>(Oracle string). </i>Figure 2 shows an example of word reordering done on the string level. NULL aligned words do not move from their positions.</p><p><b>Oracle parse tree with permute siblings. </b>The or­acle system unfolds an alignment crossing if and only if the source side of the alignment crossing is covered by the same node in the syntactic source tree, and the alignment pair subject to crossing can be unfolded by permuting the order of the sibling nodes.<page local="4" global="41"/> NULL aligned words do not prevent unfold­ing crossings because we include them with the ad­jacent words that are involved in the crossings. We call this configuration <i>Oracle tree.</i></p><footnote label="1">All the source permutation methods presented in this Sec­tion are based on automatic alignments, which inevitably con­tain wrong links. In the future we plan to involve manual align­ments to the computation of oracle permutation</footnote><p>Figure 1 shows an example. According to the <i>Or­acle tree </i>constraint, the word "went" can be placed in the end of the sentence since the replacement can be done as a swapping of "VBP" and "PP" cate­gories. The same happens for the word "reflect" swapping with "S" constituent in Figure 1, but not for the chunks "the positions" and "not properly": this crossing cannot be resolved under the tree con­straint since they are not dominated by sibling syn­tactic categories.</p><p><b>Oracle tree with delete descendants, permute sib­lings. </b>This oracle implements an additional mech­anism of tree modification to increase the number of reordering permutations in comparison with <i>Ora­cle tree </i>algorithm. Here we allow for an additional tree transduction operation that deletes intervening layers before applying sibling permutation. This is illustrated in Figure 3. The word "must" can not be moved to the beginning of the sentence in Fig­ure 3a by <i>Oracle tree.</i><i> </i>Instead, this is done in two steps. Firstly, the VP dominating the words "must" and "apply" is deleted under the current node S, the transformed tree is shown in Figure 3b. Sub­sequently, the siblings under S in the resulting tree are permuted, "must" is reordered across the whole clause and placed to the first position (see Figure 3c). We call this system <i>Oracle mod. </i>Figure 4 shows an example in which crossing align­ment links cannot be unfolded without deleting 4 in­tervening layers.</p><p><b>Oracle results </b>Table 1 contrasts the oracle results with the performance shown by standard PBSMT systems. The experimental setup is detailed in Sec­tion 6. We consider the following baseline con­figurations: <i>PBSMT </i>- Moses-based PBSMT with distance-based reordering model; <i>PBSMT+MSD </i>-Moses-based PBSMT with distance-based reorder­ing model and MSD and <i>Moses-chart </i>- hierarchical</p><p>Moses-chart-based PBSMT.</p><p>Depending on number of parse tree levels allowed to be deleted, we consider three <i>Oracle mod </i>sys­tems: with two (2lt), three (3lt) and five (5lt) levels of descendants allowed to be deleted for a more flat parse tree structure before sibling permutation.</p><p>(a) Original bilingual phrase.</p><p>(b) Reordered bilingual phrase. Figure 2: Example of <i>Oracle string </i>unfolding.</p><p>(a) Original parse tree. (b) Parse tree with deleted VP category. (c) Reordered parse tree.</p><figure caption="Figure 3: Example of text monotonization with tree transformation."></figure><page local="5" global="42"/><p>The impact of corpus monotonization on transla­tion system performance is measured using the fi­nal point of weight optimization on the develop­ment set <i>(Dev BLEU), </i>as well as on the test set <i>(Test BLEU/NIST)</i>.</p><p>The major conclusion that can be drawn from the oracle results is that the source reordering de­fined in terms of parse tree transduction can poten­tially lead to increased translation quality (up to 1.2 BLEU points for English-Dutch, 0.5 for English-Spanish and 1.7 for English-Chinese). At the same time, a huge gap between performance shown by <i>Oracle string </i>and tree oracle systems («2.2 BLEU points for English-Dutch, «1.3 for English-Spanish and «2.5 for English-Chinese) shows that there are many crossing alignments which cannot be un­folded with simple, local transductions over a single source-side syntactic tree.</p></section><section number="5" title="Conditional Tree-Transductions"><p>Our model aims at learning from the source permuted parallel corpus (containing tu­ples (s, a, s)) a probabilistic optimization (argmax^(s) <i>Pr</i>(n(s) | <i>s,Ts)), </i>where Ts is the source parse and n(s) is some eligible permutation of s. We view the permutations leading from s to s as a sequence of local tree transductions <i>t</i><i>s</i><b><i>o </i></b><i>— </i>... — <i>nn</i>, where so = s and sn = s, and each transduction <i>Tsi-1 — </i>ts4 is defined using any of two kinds of local tree transduction operations used in Section 4 or alternatively NOP (No Operation).</p><p>The sequence ts0 — ... — Tsn is obtained by taking the next node in a top-down tree traversal, then statistically selecting the most likely of three transduction operations and applying the selected operation to the current node. If the current tree is ts4-1 , and the current node has address x, is syntacti­cally labeled Nx, directly dominates <i>ax </i>(the ordered sequence of node labels under x), we approximate the conditional probability P(ts4 | <i>Tsi-1 ) </i>with the transduction operation it employs:</p><p>• Permute the children of x in <i>Tsi-1 </i>with proba­bility</p><doubt alpha="47.6" length="21" tooSmall="False" monospace="0.0">«P(n(ax)|Nx—ax,CX)(1)</doubt><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">SBAR</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">S</doubt><doubt alpha="80.0" length="5" tooSmall="False" monospace="0.0">if NP</doubt><doubt alpha="52.9" length="17" tooSmall="False" monospace="0.0">PRP       VP ADVP</doubt><doubt alpha="54.2" length="24" tooSmall="False" monospace="0.0">they VBP    RB     RB VB</doubt><p><b>do     not properly reflect NP</b></p><figure caption="Figure 4: Example of unfoldable alignment crossings."></figure><p>where n(ax) is a permutation of <i>ax </i>(the or­dered sequence of node labels under x) and <i>Cx</i> is a local tree context of node x in tree t / .</p><doubt alpha="75.0" length="4" tooSmall="False" monospace="0.0">si-i</doubt><p>• Select a child of x to delete, pull its children up directly under x, effectively changing a<i>x </i>to some and then permute the children of the latter.</p><doubt alpha="46.9" length="32" tooSmall="False" monospace="0.0">« P(ax~»adx,n(ax) |Nx— ax,Cx)(2)</doubt><p>where (ax ~» ax symbolizes the result of deleting a subtree under a child of x. This op­eration applies also to subtrees of depth n &lt;E {1, 2, 3,5} under x, i.e., a child is depth 1, a child with its children is depth 2 and so on.</p><p>Obviously, the number of possible permutations of a<i>x </i>is factorial in the length of a<i>x. </i>Fortunately, the source permuted training data exhibits only a frac­tion of possible permutations even for longer a<i>x </i>se­quences. Furthermore, by conditioning the probabil­ity on local context C<i>x, </i>the number of permutations is limited to a handful set.</p><p>Theoretically, we could define the probabil­ity of the sequence of local tree transductions<page local="6" global="43"/></p><doubt alpha="53.8" length="13" tooSmall="False" monospace="0.0">TSo—...—Ts„as</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">n</doubt><doubt alpha="45.9" length="37" tooSmall="False" monospace="0.0">P (TSo—...—Ts„)=TJP(Tsi|TSi-i)(3) i=1</doubt><p>Baselines</p><p>However, unlike earlier work (e.g., (Tromble and Eisner, 2009)), we cannot afford to do so because every local transduction conditions on context Cx of an intermediate tree, which quickly risks becoming intractable (even when we use packed forests). Fur­thermore, the problem of calculating the most likely permutation under such a model is made difficult by the fact that different transduction sequences may lead to the same permutation, which demands sum­ming over these sequences (another intractable sum­mation). Earlier work has avoided conditioning con­text, effectively assuming that the each intermediate permutation is independent from the preceding ones.</p><p>Instead, we take a pragmatic approach and greed­ily select at every intermediate point ts<i>i-1 — </i>Ts<i>i </i>the single most likely local transduction that can be ap­plied to a node in the current intermediate tree ts <i>i-1</i>using an interpolation of the terms in Equations 1 and 2 with probability ratios of the language model s as follows:</p><doubt alpha="66.7" length="21" tooSmall="False" monospace="0.0">P(TRANSi\ Nx-+ax,Cx)x</doubt><doubt alpha="71.4" length="7" tooSmall="False" monospace="0.0">Plm(si)</doubt><p>where TRANSi is any of the two transduction oper­ations or NOP, and Pm is a language model trained on the <i>s </i>side of the corpus {(s, a, <i>s</i>)}. The ratio­nale behind this log-linear interpolation is that our source permutation approach aims at finding the op­timal permutation s of s that can serve as input for a subsequent translation model. Hence, we aim at tree transductions that are syntactically motivated that also lead to improved string permutation. In this sense, the tree transduction definitions can be seen as an efficient and syntactically informed way to define the space of possible permutations.</p><p><b>Estimates. </b>We estimate the string probabilities Plm ( ) using 3-gram language models trained on the s side of the source permuted parallel corpus {( s, a, s )} . We estimate the conditional probability <i>Pr</i>(TRANS | Nx — ax,Cx) using a Maximum-Entropy framework, where feature functions are de­fined to capture the permutation as a class, the node label Nx and its head POS tag, the child sequence ax together with the corresponding sequence of head POS tags and other features corresponding to dif­ferent contextual information.</p><p><b>Features in use. </b>We used a set of 15 features to capture reordering permutations from the syntac­tic and linguistic perspectives: <i>Local tree topology: </i>sub-tree instances that include parent node and the ordered sequence of child node labels (1); <i>Depen­dency features: </i>features that determine the POS tag of the head word of the current node (2), together with the sequence of POS tags ofthe head words of its child nodes (3) and the POS tag ofthe head word of the parent (4) and grandparent nodes (5); <i>Syntac­tic features: </i>apart from the whole path from the cur­rent node to the tree root node (6), we used three binary features from this class describe: (7) whether the parent node is a child of the node annotated with the same syntactic category, (8) whether the parent node is a descendant of the node annotated with the same syntactic category, and (9) if the current sub­tree is embedded into a <i>"S-SBAR" </i>sub-tree<footnote anchor="2"/>; <i>POS lexical features:</i><page local="7" global="44"/><i> </i>bi- and tri-grams of POS tags of the left- and right-hand side neighboring words (10­13); <i>Counters: </i>a number of words covered by a given constituent (14) and a number of children of the given node (15).</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>System</p></td><td class="cell"><p>EnNl</p></td><td class="cell"><p>EnEs</p></td><td class="cell"><p>EnZh</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Dev BLEU</p></td><td class="cell"><p>Test</p></td><td class="cell"><p>Dev BLEU</p></td><td class="cell"><p>Test</p></td><td class="cell"><p>Dev BLEU</p></td><td class="cell"><p>Test</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>BLEU NIST</p></td><td class="cell"><p>BLEU NIST</p></td><td class="cell"><p>BLEU NIST</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 1: Summary of oracle results." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PBSMT</p></td><td class="cell"><p>23.88</p></td><td class="cell"><p>24.04</p></td><td class="cell"><p>6.29</p></td><td class="cell"><p>32.31</p></td><td class="cell"><p>31.70</p></td><td class="cell"><p>7.48</p></td><td class="cell"><p>18.71</p></td><td class="cell"><p>22.21</p></td><td class="cell"><p>5.28</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PBSMT+MSD</p></td><td class="cell"><p>24.07</p></td><td class="cell"><p>24.04</p></td><td class="cell"><p>6.28</p></td><td class="cell"><p>32.45</p></td><td class="cell"><p>31.85</p></td><td class="cell"><p>7.47</p></td><td class="cell"><p>18.99</p></td><td class="cell"><p>21.18</p></td><td class="cell"><p>5.30</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Moses-chart</p></td><td class="cell"><p>23.94</p></td><td class="cell"><p>24.93</p></td><td class="cell"><p>6.39</p></td><td class="cell"><p>30.58</p></td><td class="cell"><p>31.80</p></td><td class="cell"><p>7.41</p></td><td class="cell"><p>19.93</p></td><td class="cell"><p>23.90</p></td><td class="cell"><p>5.41</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Oracle results</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Oracle tree</p></td><td class="cell"><p>24.70</p></td><td class="cell"><p>24.80</p></td><td class="cell"><p>6.32</p></td><td class="cell"><p>32.76</p></td><td class="cell"><p>32.21</p></td><td class="cell"><p>7.51</p></td><td class="cell"><p>20.23</p></td><td class="cell"><p>23.44</p></td><td class="cell"><p>5.35</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Oracle string</p></td><td class="cell"><p>26.28</p></td><td class="cell"><p>27.02</p></td><td class="cell"><p>6.50</p></td><td class="cell"><p>34.09</p></td><td class="cell"><p>33.52</p></td><td class="cell"><p>7.60</p></td><td class="cell"><p>23.01</p></td><td class="cell"><p>26.08</p></td><td class="cell"><p>5.52</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Oracle mod+21t</p></td><td class="cell"><p>25.05</p></td><td class="cell"><p>25.04</p></td><td class="cell"><p>6.36</p></td><td class="cell"><p>32.24</p></td><td class="cell"><p>32.18</p></td><td class="cell"><p>7.51</p></td><td class="cell"><p>20.64</p></td><td class="cell"><p>23.75</p></td><td class="cell"><p>5.37</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Oracle mod+31t</p></td><td class="cell"><p>25.11</p></td><td class="cell"><p>25.27</p></td><td class="cell"><p>6.37</p></td><td class="cell"><p>32.22</p></td><td class="cell"><p>32.34</p></td><td class="cell"><p>7.52</p></td><td class="cell"><p>20.71</p></td><td class="cell"><p>23.59</p></td><td class="cell"><p>5.37</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Oracle mod+51t</p></td><td class="cell"><p>25.07</p></td><td class="cell"><p>25.23</p></td><td class="cell"><p>6.37</p></td><td class="cell"><p>32.51</p></td><td class="cell"><p>32.37</p></td><td class="cell"><p>7.55</p></td><td class="cell"><p>20.93</p></td><td class="cell"><p>23.93</p></td><td class="cell"><p>5.39</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><section number="6" title="Translation and reordering experiments"><p><b>Data. </b>In our experiments we used English-Dutch and English-Spanish European Parliament data and an extraction from the English-Chinese Hong Kong Parallel Corpus. All the sets were provided with one reference translation. Basic statistics of the training data can be found in Table 2, development datasets contained 0.5K, 1.9K and 0.5K lines and test datasets contained 1K, 1.9K and 0.5K for Dutch, Spanish and English, respectively.</p><p><b>Experimental setup. </b>Word alignment was found using GIZA++<footnote anchor="3"/> (Och, 2003), supported by mk-cls<footnote anchor="4"/> (Och, 1999) tool. The PBSMT systems we con­sider in this study is based on Moses toolkit (Koehn et al., 2007). We followed the guidelines provided on the Moses web page<footnote anchor="5"/>.</p><p>Two phrase reordering methods are widely used in phrase-based systems. A distance-based reorder­ing model providing the decoder with a cost linear to the distance between words that are being re­ordered. This model constitutes the default for the Moses system. And, a lexicalized block-oriented data-driven reordering model (Tillman, 2004) con­siders three orientations: monotone (M), swap (S), and discontinuous (D), while the reordering proba­bilities are conditioned on the lexical context ofeach phrase pair.</p><p>All language models were trained with SRI LM toolkit (Stolcke, 2002). Language models for Dutch, Spanish and Chinese use 5 -grams, while the idealized English (s ) is modeled using 3-grams.</p><footnote label="2">The latter feature intends to model the divergence in word order in relative clauses between Dutch and English which is illustrated in Figure 1.</footnote><footnote label="3">code.google.com/p/giza-pp/</footnote><footnote label="4">http://www.fjoch.com/mkcls.html</footnote><footnote label="5">http://www.statmt.org/moses/</footnote><p>We use Stanford parser<footnote anchor="6"/> (Klein and Manning, 2003) as a source-side parsing engine. The parser was trained on the WSJ Penn treebank provided with 14 syntactic categories and 48 POS tags. The eval­uation conditions were case-sensitive and included punctuation marks. For Maximum Entropy model­ing we used the maxent toolkit<footnote anchor="7"/>.</p><p><b>Translation scores. </b>Table 3 shows the results of automatic evaluation using BLEU (Papineni et al., 2002) and NIST (Doddington, 2002) metrics.</p><p><i>MERrd </i>configuration corresponds to the PBSMT system with the source side of the parallel corpus re­ordered using our Maximum Entropy model, but the transduction operations are limited to permutation of the children only. <i>MERrd+xlt </i>configuration refers to the set of systems which, beside child permuta­tion, includes a deletion operation with the maxi­mum number of tree layers that can be deleted set to x. All reordered systems include a MSD model as a supporting reordering mechanism.</p><p>BLEU scores measured on the test data, which are statistically significant from the best PBSMT results are marked with bold. The statistical significance calculations have been done for a 95% confidence interval and 1000 resamples, following the guide­lines in Koehn (2004).</p><p><b>Analysis. </b>Our results show that source-reordering is beneficial for the language pairs with high mutual word order disparity. In contrast to English-Dutch and English-Chinese translation tasks, the statistical significance test reveals that all but the <i>MERrd+5lt </i>English-Spanish PBSMT systems with rearranged input are not different from the translation qual­ity delivered by Moses. This disappointing result for the English-to-Spanish translation task may be explained by the fact that many reordering differ­ences are resolved by standard reordering models (distance-based and MSD).</p><p>Table 3 shows the results of automatic transla­tion quality evaluation. A gap between the max­imum reachable performance shown by tree trans-duction systems and the translation quality delivered<page local="8" global="45"/></p><footnote label="6">http://nlp.stanford.edu/software/ lex- parser.shtml</footnote><footnote label="7">http://homepages.inf.ed.ac.uk/lzhang10/ maxent_toolkit.html</footnote><p>Baselines by our model is 0.05-0.29 BLEU points for English-Dutch, 0.01-0.09 for English-Spanish and 0.27­0.76 for English-Chinese. These numbers demon­strate that there are some potentially usable regu­larities not captured by our current conditional tree-transduction model.</p></section><section number="7" title="Conclusions and future work"><p>We present a source reordering system for PBSMT, in which the reordering decisions are conditioned on features from the source parse tree. Our system allows for two operations over the parse tree: per­muting the order of sibling nodes and deleting child nodes in order to make the tree flatter and exploit sibling permutations at the remaining layers.</p><p>Our contribution can be summarized as follows: (1) we report detailed results of maximum poten­tial performance that can be achieved with source reordering under different constraints, (2) we define a source-reordering process through an efficient se­quence of greedy, context-conditioned transduction operations over the source parse trees.</p><p>The method was tested on three different trans­lation tasks. The results show that our approach is more effective for language pairs with significant difference in word order. Another important ob­servation is that our model demonstrate translation quality comparable with the one delivered by SMT systems based on hierarchical phrases.</p><p>The introduced reordering algorithm and the re­sults obtained present many opportunities for future work. We plan to perform a detailed analysis of the structure of the extracted phrases to find out the par­ticular cases where the improvement comes from. We also propose to discover other possible trans-duction operations to better explore the reordering space.</p></section><section number="8" title="Acknowledgements"><p>This work is supported by The Netherlands Orga­nization for Scientific Research (NWO) under VIDI grant (nr. 639.022.604).</p><table caption="Table 2: Statistics of the training, development and test corpora." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Parameter</p></td><td class="cell"><p>Dutch</p></td><td class="cell"><p>English</p></td><td class="cell"><p>Spanish</p></td><td class="cell"><p>English</p></td><td class="cell"><p>Chinese</p></td><td class="cell"><p>English</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Training corpus</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Sentences</p></td><td class="cell"><p>1.2 M</p></td><td class="cell"><p>1.2 M</p></td><td class="cell"><p>1.4 M</p></td><td class="cell"><p>1.4 M</p></td><td class="cell"><p>1.5 M</p></td><td class="cell"><p>1.5 M</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Words</p></td><td class="cell"><p>32.9 M</p></td><td class="cell"><p>33.0 M</p></td><td class="cell"><p>40.1 M</p></td><td class="cell"><p>38.54 M</p></td><td class="cell"><p>35.35 M</p></td><td class="cell"><p>35.00 M</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Vocabulary</p></td><td class="cell"><p>228 K</p></td><td class="cell"><p>104 K</p></td><td class="cell"><p>168 K</p></td><td class="cell"><p>119K</p></td><td class="cell"><p>136 K</p></td><td class="cell"><p>245 K</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Average sentence length</p></td><td class="cell"><p>27.20</p></td><td class="cell"><p>27.28</p></td><td class="cell"><p>28.80</p></td><td class="cell"><p>27.67</p></td><td class="cell"><p>24.06</p></td><td class="cell"><p>23.83</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>System</p></td><td class="cell"><p>EnNl</p></td><td class="cell"><p>EnEs</p></td><td class="cell"><p>EnZh</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Dev BLEU</p></td><td class="cell"><p>Test</p></td><td class="cell"><p>Dev BLEU</p></td><td class="cell"><p>Test</p></td><td class="cell"><p>Dev BLEU</p></td><td class="cell"><p>Test</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>BLEU NIST</p></td><td class="cell"><p>BLEU NIST</p></td><td class="cell"><p>BLEU NIST</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 3: Experimental results." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PBSMT</p></td><td class="cell"><p>23.88</p></td><td class="cell"><p>24.04</p></td><td class="cell"><p>6.29</p></td><td class="cell"><p>32.31</p></td><td class="cell"><p>31.70</p></td><td class="cell"><p>7.48</p></td><td class="cell"><p>18.71</p></td><td class="cell"><p>22.21</p></td><td class="cell"><p>5.28</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PBSMT+MSD</p></td><td class="cell"><p>24.07</p></td><td class="cell"><p>24.04</p></td><td class="cell"><p>6.28</p></td><td class="cell"><p>32.45</p></td><td class="cell"><p>31.85</p></td><td class="cell"><p>7.47</p></td><td class="cell"><p>18.99</p></td><td class="cell"><p>21.18</p></td><td class="cell"><p>5.30</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Moses-chart</p></td><td class="cell"><p>23.94</p></td><td class="cell"><p>24.93</p></td><td class="cell"><p>6.39</p></td><td class="cell"><p>30.58</p></td><td class="cell"><p>31.80</p></td><td class="cell"><p>7.41</p></td><td class="cell"><p>19.93</p></td><td class="cell"><p>23.90</p></td><td class="cell"><p>5.41</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Reordering systems</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MERrd</p></td><td class="cell"><p>24.64</p></td><td class="cell"><p>24.72</p></td><td class="cell"><p>6.33</p></td><td class="cell"><p>31.97</p></td><td class="cell"><p>32.19</p></td><td class="cell"><p>7.52</p></td><td class="cell"><p>19.82</p></td><td class="cell"><p>23.17</p></td><td class="cell"><p>5.33</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MERrd+21t</p></td><td class="cell"><p>24.61</p></td><td class="cell"><p>24.99</p></td><td class="cell"><p>6.35</p></td><td class="cell"><p>31.70</p></td><td class="cell"><p>32.11</p></td><td class="cell"><p>7.50</p></td><td class="cell"><p>20.02</p></td><td class="cell"><p>23.01</p></td><td class="cell"><p>5.33</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MERrd+31t</p></td><td class="cell"><p>24.82</p></td><td class="cell"><p>24.98</p></td><td class="cell"><p>6.34</p></td><td class="cell"><p>31.65</p></td><td class="cell"><p>32.25</p></td><td class="cell"><p>7.52</p></td><td class="cell"><p>20.21</p></td><td class="cell"><p>23.14</p></td><td class="cell"><p>5.34</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MERrd+51t</p></td><td class="cell"><p>24.78</p></td><td class="cell"><p>25.12</p></td><td class="cell"><p>6.37</p></td><td class="cell"><p>31.99</p></td><td class="cell"><p>32.38</p></td><td class="cell"><p>7.52</p></td><td class="cell"><p>20.29</p></td><td class="cell"><p>23.17</p></td><td class="cell"><p>5.35</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="9" global="46"/></section><references><p>A. Berger, S. Della Pietra, and V. Della Pietra. 1996. A maximum entropy approach to natural language pro­cessing. <i>Computational Linguistics, </i>1(22):39-72. D. Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In <i>Proceedings of ACL '05, </i>pages 263-270, Ann Arbor, MI, USA. D. Chiang. 2007. Hierarchical phrase-based translation.</p><p><i>Computational Linguistics, </i>2(33):201-228. M. Collins, P. Koehn, and I. Ku5erova. 2005. Clause re­structuring for statistical machine translation. In <i>Pro­ceedings ofACL'05, </i>pages 531-540, Ann Arbor, MI, USA.</p><p>M. R. Costa-jussà and J. A. R. Fonollosa. 2006. Statistical machine reordering.   In <i>Proceedings of HLT/EMNLP '06, </i>pages 70-76, New York, NY, USA. G. Doddington. 2002. Automatic evaluation of machine translation quality using n-grams co-occurrence statis­tics. In <i>Proceedings of HLT'02, </i>pages 128-132, San Diego, CA, USA. L. Huang, H. Zhang, D. Gildea, and K. Knight. 2009. Binarization of synchronous context-free grammars. <i>Computational Linguistics, </i>35(4):559-595. M. Khalilov and K. Sima'an.  2010.  A discriminative syntactic model for source permutation via tree trans-duction. In <i>Proceedings of SSST-4 workshop at COL-ING '10, </i>pages 92-100, Beijing, China. D. Klein and C. Manning. 2003. Accurate unlexicalized parsing. In <i>Proceedings of ACL'03, </i>pages 423-430, Sapporo, Japan. Ph. Koehn, F. Och, and D. Marcu.   2003. Statistical phrase-based machine translation. In <i>Proceedings of the HLT-NAACL '03, </i>pages 48-54, Edmonton, Canada. Ph. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin, and E. Herbst.   2007.   Moses: open-source toolkit for statistical machine translation. In <i>Proceedings of ACL '07, </i>pages 177-180, Prague, Czech Republic. Ph. Koehn.    2004.    Statistical significance tests for machine translation evaluation.   In <i>Proceedings of</i></p><p><i>EMNLP'04, </i>pages 388-395, Barcelona, Spain. C. Li, L. Minghui, D. Zhang, M. Li, M. Zhou, and Y. Guan. 2007. A probabilistic approach to syntax-based reordering for statistical machine translation. In <i>Proceedings ofACL'07, </i>pages 720-727, Prague, Czech Republic. F. Och and H. Ney. 2004. The alignment template ap­proach to statistical machine translation. <i>Computa­tionalLinguistics, </i>30(4):417-449. F. Och. 1999. An efficient method for determining bilin­gual word classes. In <i>Proceedings of ACL '99, </i>pages 71-76, Maryland, MD, USA.</p><p>F. Och. 2003. Minimum error rate training in statistical machine translation. In <i>Proceedings ofACL'03, </i>pages 160-167, Sapporo, Japan.</p><p>K. Papineni, S. Roukos, T. Ward, and W. Zhu. 2002.</p><p>Bleu: a method for automatic evaluation of machine translation. In <i>Proceedings of ACL'02, </i>pages 311­318, Philadelphia, PA, USA. M. Popovic' and H. Ney. 2006. POS-based word re-orderings for statistical machine translation. In <i>Pro­ceedings ofLREC'06, </i>pages 1278-1283, Genoa, Italy, May.</p><p>A. Ramanathan, P. Bhattacharyya, J. Hegde, R.M. Shah, and M. Sasikumar. 2008. Simple syntactic and mor­phological processing can help english-hindi statistical machine translation. In <i>In Proceedings ofIJCNLP '08, </i>Hyderabad, India. A. Stolcke. 2002. SRILM: an extensible language mod­eling toolkit. In <i>Proceedings ofSLP'02, </i>pages 901­904.</p><p>C. Tillman. 2004. A unigram orientation model for sta­tistical machine translation. In <i>Proceedings ofHLT-</i> <i>NAACL'04, </i>pages 101-104, Boston, MA, USA.</p><p>R. Tromble and J. Eisner. 2009. Learning linear order­ing problems for better translation. In <i>Proceedings of</i> <i>EMNLP'09, </i>pages 1007-1016, Singapore.</p><p>K. Visweswariah, J. Navratil, J. Sorensen, V. Chenthama-rakshan, and N. Kambhatla. 2010. Syntax based re­ordering with automatically derived rules for improved statistical machine translation. In <i>ProceedingofCOL-ING '10, </i>pages 1119-1127, Beijing, China.</p><doubt alpha="61.2" length="49" tooSmall="False" monospace="0.0">C. Wang, M. Collins, and Ph. Koehn. 2007. Chinese</doubt><p>syntactic reordering for statistical machine translation. In <i>Proceedings ofEMNLP-CoNLL '07, </i>pages 737-745, Prague, Czech Republic.</p><p>W. Wang, J. May, K. Knight, and D. Marcu. June 2010.</p><p>Re-structuring, re-labeling, and re-aligning for syntax-based machine translation. <i>Computational Linguis­tics, </i>36:247-277.</p><p>D. Wu and H. Wong. 1998. Machine translation wih a stochastic grammatical channel. In <i>Proceedings of</i></p><doubt alpha="54.5" length="44" tooSmall="False" monospace="0.0">ACL-COLING'98,pages 1408-1415, Columbus, OH,</doubt><p>USA.</p><p>F. Xia and M. McCord. 2004. Improving a statistical MT system with automatically learned rewrite patterns. In <i>Proceedings ofCOLING'04, </i>pages 508-514, Geneva, Switzerland.</p><p>R. Zens, F. Och, and H. Ney. 2002. Phrase-based sta­tistical machine translation. In <i>Proceedings ofKI: Ad­vances in Artificial Intelligence</i>, pages 18-32. S. Zwarts and M. Dras. 2007. Syntax-based word re­ordering in phrase-based statistical machine transla­tion: Why does it work? In <i>Proceedings of the MT SummitXI, </i>Copenhagen, Denmark.</p></references></body></article>