<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="746"/><title>Binarizing Syntax Trees to Improve Syntax-Based Machine Translation Accuracy</title><pubinfo>Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning,pp. 746-754, Prague, June 2007. ©2007 Association for Computational Linguistics</pubinfo><author surname="Wang" givenname="Wei"><org  name="Language Weaver Inc." country="USA" city="Marina del Rey"/></author><author surname="Knight" givenname="Kevin"><org  name="Language Weaver Inc." country="USA" city="Marina del Rey"/></author><author surname="Marcu" givenname="Daniel"><org  name="Language Weaver Inc." country="USA" city="Marina del Rey"/></author></firstpageheader><frontmatter><p><b>Binarizing Syntax Trees to Improve Syntax-Based Machine Translation Accuracy</b></p><p><b>Wei Wang </b>and <b>Kevin Knight </b>and <b>Daniel Marcu</b></p><p>Language Weaver, Inc. 4640 Admiralty Way, Suite 1210 Marina del Rey, CA, 90292 {wwang,kknight,dmarcu}@languageweaver.com</p></frontmatter><abstract>We show that phrase structures in Penn Tree-bank style parses are not optimal for syntax-based machine translation. We exploit a se­ries of binarization methods to restructure the Penn Treebank style trees such that syn-tactified phrases smaller than Penn Treebank constituents can be acquired and exploited in translation. We find that by employing the EM algorithm for determining the binariza­tion of a parse tree among a set of alternative binarizations gives us the best translation re­sult. </abstract></header><body><section number="1" title="Introduction"><p>Syntax-based translation models (Eisner, 2003; Gal­ley et al., 2006; Marcu et al., 2006) are usually built directly from Penn Treebank (PTB) (Marcus et al., 1993) style parse trees by composing treebank gram­mar rules. As a result, often no substructures corre­sponding to partial PTB constituents are extracted to form translation rules.</p><p>Syntax translation models acquired by composing treebank grammar rules assume that long rewrites are not decomposable into smaller steps. This ef­fectively restricts the generalization power of the in­duced model. For example, suppose we have an xRs (Knight and Graehl, 2004) rule R1 in Figure 1 that translates the Chinese phrase russia minister viktor-chernomyrdin into an English NPB tree fragment yielding an English phrase. Also suppose that we want to translate a Chinese phrase</p><p>viktor-chernomyrdin and his colleague</p><p>into English. What we desire is that if we have another rule R2 as shown in Figure 1, we could somehow compose it with Ri to obtain the desir­able translation. We unfortunately cannot do this because R1 and R2 are not further decomposable and their substructures cannot be re-used. The re­quirement that all translation rules have exactly one root node does not enable us to use the translation of viktor-chernomyrdin in any other contexts than those seen in the training corpus.</p><p>A solution to overcome this problem is to right-binarize the left-hand side (LHS) (or the English-side) tree of R1 such that we can decompose R1 into R3 and R4 by factoring <b>NNP(viktor) NNP(chernomyrdin) </b>out as R4 according to the word alignments; and left-binarize the LHS ofR2 by introducing a new tree node that collapses the two NNP's, so as to generalize this rule, getting rule R5and rule R6. We also need to consistently syntact-ify the root labels of R4 and the new frontier label of <i>R6 </i>such that these two rules can be composed. Since labeling is not a concern of this paper, we sim­ply label new nodes with X-bar where X here is the parent label. With all these in place, we now can translate the foreign sentence by composing <i>R6 </i>and R4 in Figure 1.</p><p>Binarizing the syntax trees for syntax-based ma­chine translation is similar in spirit to generalizing parsing models via markovization (Collins, 1997; Charniak, 2000). But in translation modeling, it is unclear how to effectively markovize the translation rules, especially when the rules are complex like those proposed by Galley et al. (2006).</p><p>In this paper, we explore the generalization abil­ity of simple binarization methods like left-, right-, and head-binarization, and also their combinations. Simple binarization methods binarize syntax trees in a consistent fashion (left-, right-, or head-) and<page local="2" global="747"/></p><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">Ri</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">NPB</doubt><p>JJ      NNP    NNP NNP russia   minister   viktor chernomyrdin</p><p>RUSSIA MINISTER V-C</p><doubt alpha="50.0" length="2" tooSmall="False" monospace="0.0">R2</doubt><doubt alpha="64.0" length="25" tooSmall="False" monospace="0.0">x0:NNP x1:NNP CC PRB$ NNS</doubt><p>and   his colleague NNP      NNP    AND HIS COLLEAGUE</p><p>V-C AND HIS COLLEAGUE</p><doubt alpha="50.0" length="2" tooSmall="False" monospace="0.0">R6</doubt><doubt alpha="50.0" length="2" tooSmall="False" monospace="0.0">R4</doubt><p>Figure 1: Generalizing translation rules by binarizing trees.</p><p>thus cannot guarantee that all the substructures can be factored out. For example, right binarization on the LHS of R1 makes available R4, but misses R6on R2. We then introduce a <i>parallel restructuring </i>method, that is, one can binarize both to the left and right at the same time, resulting in a binarization for­est. We employ the EM (Dempster et al., 1977) algo­rithm to learn the binarization bias for each tree node from the parallel alternatives. The EM-binarization yields best translation performance.</p><p>The rest of the paper is organized as follows. Section 2 describes related research. Section 3 de­fines the concepts necessary for describing the bina-rizations methods. Section 4 describes the tree bina-rization methods in details. Section 5 describes the forest-based rule extraction algorithm, and section 6 explains how we restructure the trees using the EM algorithm. The last two sections are for experiments and conclusions.</p></section><section number="2" title="Related Research"><p>Several researchers (Melamed et al., 2004; Zhang et al., 2006) have already proposed methods for bi-narizing synchronous grammars in the context of machine translation. Grammar binarization usually maintains an equivalence to the original grammar such that binarized grammars generate the same language and assign the same probability to each string as the original grammar does. Grammar binarization is often employed to make the grammar fit in a CKY parser. In our work, we are focused on binarization of parse trees. Tree binarization generalizes the re­sulting grammar and changes its probability distri­bution. In tree binarization, synchronous grammars built from restructured (binarized) training trees still contain non-binary, multi-level rules and thus still require the binarization transformation so as to be employed by a CKY parser.</p><p>The translation model we are using in this paper belongs to the xRs formalism (Knight and Graehl, 2004), which has been proved successful for ma­chine translation in (Galley et al., 2004; Galley et al.,2006; Marcu etal., 2006).</p></section><section number="3" title="Concepts"><p>We focus on tree-to-string (in noisy-channel model sense) translation models. Translation models of this type are typically trained on tuples of a source-language sentence <b>f, </b>a target language (e.g., English) parse tree n that yields e and translates from <b>f, </b>and the word alignments <b>a </b>between <b>e </b>and <b>f</b>. Such a tuple is called an alignment graph in (Galley et al., 2004). The graph (1) in Figure 2 is such an alignment graph.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>R<b>3 </b>^NP§^</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>JJ NPB</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>russia minister</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>RUSSIA MINISTER</p></td><td class="cell"><p>^ R<b>4</b></p><p>NNP</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>viktor chernomyrdin</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>V-C</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>R<b>6</b></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>PRB$</p></td><td class="cell"><p>colleague</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>R<b>5</b></p></td><td class="cell"><p>NPB</p></td><td class="cell"><p>CC his</p></td><td class="cell"><p>COLLEAGUE</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>^ \ x0:NNP x1:NNP</p></td><td class="cell"><p>and HIS</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NNP NNP</p></td><td class="cell"><p>AND</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="3" global="748"/><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">NPB</doubt><p>(1) unbinarized tree (2) left-binarization (3) right-/head-binarization (4) left-binarization (5) right-binarization    (6) left-binarization    (7) right-/head-binarization</p><doubt alpha="50.0" length="24" tooSmall="True" monospace="0.0">NNP1    NNP2  NNP3 NNP4*</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">ii</doubt><doubt alpha="94.7" length="19" tooSmall="True" monospace="0.0">viktor Chernomyrdin</doubt><doubt alpha="94.7" length="19" tooSmall="True" monospace="0.0">VIKTOR-CHERNOMYRDIN</doubt><doubt alpha="91.7" length="24" tooSmall="True" monospace="0.0">NNPiNNP2NNP3Chernomyrdin</doubt><doubt alpha="75.0" length="8" tooSmall="True" monospace="0.0">NNP2NNP3</doubt><doubt alpha="93.8" length="16" tooSmall="True" monospace="0.0">NNP3Chernomyrdin</doubt><doubt alpha="92.9" length="14" tooSmall="True" monospace="0.0">NNPiNNP2viktor</doubt><p>NNP5 NPB*</p><doubt alpha="66.7" length="9" tooSmall="False" monospace="0.0">NNP3NNP4*</doubt><p>jor chernomyrdin</p><p>Figure 2: Left, right, and head binarizations. Heads are marked with *'s. New nonterminals introduced by binarization are denoted by X-bars.</p><p>A tree node in n is <i>admissible </i>if the <b>f </b>string cov­ered by the node is contiguous but not empty, and if the <b>f </b>string does not align to any <b>e </b>string that is not covered by n. An xRs rule can be extracted only from an admissible tree node, so that we do not have to deal with dis-contiguous <b>f </b>spans in decoding (or synchronous parsing). For example, in tree (2) in Figure 2, node NPB is not admissible because the <b>f </b>string that the node covers also aligns to <b>NNP4, </b>which is not covered by the NPB. Node NPB in tree (3), on the other hand, is admissible.</p><p>A set of sibling tree nodes is called <i>factorizable </i>if we can form an admissible new node dominating them. For example, in tree (1) of Figure 2, sibling nodes NNP2 NNP3 and NNP4 are factorizable be­cause we can factorize them out and form a new node NPB, resulting in tree (3). Sibling tree nodes NNPi NNP2 and NNP3 are not factorizable. In syn­chronous parse trees, not all sibling nodes are fac-torizable, thus not all sub-phrases can be acquired and syntactified. The main purpose of our paper is to restructure parse trees by factorization such that syntactified sub-phrases can be employed in transla­tion.</p></section><section number="4" title="Binarizing Syntax Trees"><p>We are going to binarize a tree node n that domi­nates r children n1, <i>nr</i>. Restructuring will be performed by introducing new tree nodes to domi­nate a subset of the children nodes. To avoid over-generalization, we allow ourselves to form only one new node at a time. For example, in Figure 2, we can binarize tree (1) into tree (2), but we are not allowed to form two new nodes, one dominating NNP1 NNP2 and the other dominating NNP3 NNP4. Since labeling is not the concern ofthis paper, we re­label the newly formed nodes as <i>n.</i></p><subsection number="4.1" title="Simple binarization methods"><p>The <i>left binarization </i>of node n (i.e., the NPB in tree (1) of Figure 2) factorizes the leftmost r — 1 children by forming a new node <i>n </i>(i.e., NPB in tree (2)) to dominate them, leaving the last child <i>nr </i>untouched; and then makes the new node <i>n </i>the left child of n. The method then recursively left-binarizes the newly formed node <i>n </i>until two leaves are reached. In Figure 2, we left-binarize tree (1) into (2) and then into (4).</p><p>The <i>right binarization </i>of node n factorizes the rightmost <i>r — </i>1 children by forming a new node <i>n </i>(i.e., NPB in tree (3)) to dominate them, leaving the first child <i>ri\ </i>untouched; and then makes the new node <i>n </i>the right child of <i>n.</i><page local="4" global="749"/><i> </i>The method then recur­sively right-binarizes the newly formed node <i>n. </i>In Figure 2, we right-binarize tree (1) into (3) and then into (7).</p><p>The <i>head binarization </i>of node n left-binarizes n if the head is the first child; otherwise, right-binarizes n. We prefer right-binarization to left-binarization when both are applicable under the head restriction because our initial motivation was to gen­eralize the <b>NPB</b>-rooted translation rules. As we will show in the experiments, binarization of other types of phrases contribute to the translation accuracy im­provement as well.</p><p>Any of these simple binarization methods is easy to implement, but is incapable of giving us all the factorizable sub-phrases. Binarizing all the way to the left, for example, from tree (1) to tree (2) and to tree (4) in Figure 2, does not enable us to acquire a substructure that yields NNP3 NNP4 and their trans-lational equivalences. To obtain more factorizable sub-phrases, we need to <i>parallel-binarize </i>in both di­rections.</p></subsection><subsection number="4.2" title="Parallel binarization"><p>Simple binarizations transform a parse tree into an­other single parse tree. Parallel binarization will transform a parse tree into a binarization forest, desirably packed to enable dynamic programming when extracting translation rules from it.</p><p>Borrowing terms from parsing semirings (Good­man, 1999), a packed forest is composed of addi­tive forest nodes (©-nodes) and multiplicative forest nodes (&lt;g)-nodes). In the binarization forest, a &lt;g&gt;-node corresponds to a tree node in the unbinarized tree; and this &lt;g)-node composes several ©-nodes, forming a one-level substructure that is observed in the unbinarized tree. A ©-node corresponds to al­ternative ways of binarizing the same tree node in the unbinarized tree and it contains one or more &lt; -nodes. The same ©-node can appear in more than one place in the packed forest, enabling sharing. Figure 3 shows a packed forest obtained by pack­ing trees (4) and (7) in Figure 2 via the following parallel binarization algorithm.</p><p>To parallel-binarize a tree node n that has children ni,<i>nr</i>, we employ the following steps:</p><doubt alpha="57.1" length="7" tooSmall="True" monospace="0.0">©i(NPB)</doubt><doubt alpha="53.3" length="15" tooSmall="True" monospace="0.0">®2(NPB)®ii(NPB)</doubt><doubt alpha="39.5" length="43" tooSmall="True" monospace="0.0">©3(NPB)         ©io(NNP4) ©r(NNPi) ©i2(NPB)</doubt><doubt alpha="43.8" length="16" tooSmall="True" monospace="0.0">®4(NPB) ®i3(NPB)</doubt><doubt alpha="34.9" length="43" tooSmall="True" monospace="0.0">©s(NPB)          ©9(NNP3) ©s(NNP2) ©i4(NPB)</doubt><doubt alpha="56.2" length="16" tooSmall="True" monospace="0.0">®e(NPB) ®is(NPB)</doubt><doubt alpha="39.5" length="43" tooSmall="True" monospace="0.0">©7(NNPi)         ©e(NNP2) ©g(NNP3)©io(NNP4)</doubt><p>Figure 3: Packed forest obtained by packing trees (4) and (7) in Figure 2</p><p>• We recursively parallel-binarize children nodes n1,       nr, producing binarization ©-nodes</p><p>ffi(ni),<i>©(n</i><i>r</i>), respectively.</p><p>• We right-binarize n, if any contiguous<footnote anchor="1"/> subset of children n2 ,...,nr is factorizable, by intro­ducing an intermediate tree node labeled as <i>n. </i>We recursively parallel-binarize <i>n </i>to generate a binarization forest node ffi(n). We form a multiplicative forest node &lt;g)R as the parent of e(ni) and <i>®(n).</i></p><p>• We left-binarize n if any contiguous subset of n1 ,...,nr-1 is factorizable and if this sub­set contains n1. Similar to the above right-binarization, we introduce an intermediate tree node labeled as <i>n, </i>recursively parallel-binarize <i>n</i><i> </i>to generate a binarization forest node <i>®(n),</i><i> </i>form a multiplicative forest node &lt;8&gt; L as the par­ent of ©(n) ande(ni).</p><p>• We form an additive node ©(n) as the parent of the two already formed multiplicative nodes</p><p>&lt;8&gt; <i>l </i>and <i>(&amp;r </i>.</p><p>The (left and right) binarization conditions con­sider any subset to enable the factorization of small constituents. For example, in tree (1) of Figure 2, although NNP1 NNP2 NNP3 of NPB are not factor-izable, the subset NNP1 NNP2 is factorizable. The binarization from tree (1) to tree (2) serves as a re­laying step for us to factorize NNP1 NNP2 in tree (4). The left-binarization condition is stricter than</p><p>'We factorize only subsets that cover contiguous spans to avoid introducing dis-contiguous constituents for practical pur­pose. In principle, the algorithm works fine without this bina-rization condition.</p><page local="5" global="750"/><p>the right-binarization condition to avoid spurious bi-narization; i.e., to avoid the same subconstituent be­ing reached via both binarizations. We could trans­form tree (1) directly into tree (4) without bother­ing to generate tree (3). However, skipping tree (3) will create us difficulty in applying the EM algo­rithm to choose a better binarization for each tree node, since tree (4) can neither be classified as left binarization nor as right binarization of the original tree (1) — it is the result of the composition of two left-binarizations.</p><p>In parallel binarization, nodes are not always bi-narizable in both directions. For example, we do not need to right-binarize tree (2) because NNP2 NNP3 are not factorizable, and thus cannot be used to form sub-phrases. It is still possible to right-binarize tree (2) without affecting the correctness of the parallel binarization algorithm, but that will spuriously in­crease the branching factor of the search for the rule extraction, because we will have to expand more tree nodes.</p><p>A restricted version of parallel binarization is the <i>headed parallel binarization, </i>where both the left and the right binarization must respect the head propaga­tion property at the same time.</p><p>A nice property of parallel binarization is that for any factorizable substructure in the unbinarized tree, we can always find a corresponding admissi­ble ©-node in the parallel-binarized packed forest. A leftmost substructure like the lowest NPB-subtree in tree (4) of Figure 2 can be made factorizable by several successive left binarizations, resulting in ©5 (NPB)-node in the packed forest in Figure 3. A substructure in the middle can be factorized by the composition of several left- and right-binarizations. Therefore, after a tree is parallel-binarized, to make the sub-phrases available to the MT system, all we need to do is to extract rules from the admissible nodes in the packed forest. Rules that can be ex­tracted from the original unrestructured tree can be extracted from the packed forest as well.</p><p>Parallel binarization results in parse forests. Thus translation rules need to be extracted from training data consisting of (e-forest, <b>f, a</b>)-tuples.</p></subsection></section><section number="5" title="Extracting translation rules from (e-forest, f, a)-tuples"><p>The algorithm to extract rules from (e-forest, <b>f, a)-</b>tuples is a natural generalization of the (e-parse, <b>f, a</b>)-based rule extraction algorithm in (Galley et al., 2006). The input to the forest-based algorithm is a (e-forest, <b>f, a</b>)-triple. The output of the algorithm is a derivation forest (Galley et al., 2006) composed of xRs rules. The algorithm recursively traverses the e-forest top-down and extracts rules only at admissible forest nodes.</p><p>The following procedure transforms the packed e-forest in Figure 3 into a packed synchronous deriva­tion in Figure 4.</p><p><b>Condition 1: </b>Suppose we reach an additive e-forest node, e.g. ©1(NPB) in Figure 3. For each of ©1(NPB)'s children, e-forest nodes &lt;g)2(NPB) and &lt;g)n(NPB), we go to condi­tion 2 to recursively extract rules on these two e-forest nodes, generating multiplicative derivation forest nodes, i.e., ig)(NPB(NPB : <i>x0 </i>NNP3(viktor) NNP4 (Chernomyrdin) 4) —► <i>x0 </i>V-C) and ®(NPB(NNPi NPB(NNP2 : <i>x0 </i>NPB : x1)) — x0 x1 x2) in Figure 4. We make these new &lt;g) nodes children of ©(NPB) in the derivation forest.</p><p><b>Condition 2: </b>Suppose we reach a multiplicative parse forest node, i.e., &lt;g)ii (NPB) in Figure 3. We extract rules rooted at it using the procedure in (Galley et al., 2006), forming multiplicative deriva­tion forest nodes, i.e., ®(NPB(NNPi NPB(NNP2 : <i>Xq </i>NPB : <i>x\))</i><i> </i><b>—&gt; </b><i>Xq X\ </i>£2) We then go to condition 1 to form the derivation forest on the additive frontier e-forest nodes of the newly extracted rules, generating additive derivation for­est nodes, i.e., ffi(NNPi), ©(NNP2) and ©(NPB). We make these © nodes the children of node ®(NPB(NNPi NPB(NNP2 : <i>x0 </i>NPB : <i>xi)) </i><i>-»■</i><i> </i>x0 x1 x2) in the derivation forest.</p><p>This algorithm is a natural extension ofthe extrac­tion algorithm in (Galley et al., 2006) in the sense that we have an extra condition (1) to relay rule ex­traction on additive e-forest nodes.</p><p>It is worthwhile to eliminate the spuriously am­biguous rules that are introduced by the parallel bi-<page local="6" global="751"/></p><doubt alpha="50.0" length="6" tooSmall="False" monospace="0.0">©(npb)</doubt><doubt alpha="53.8" length="104" tooSmall="False" monospace="0.0">® (npb(npb :x0nnp(viktor)nnp(chemomyrdin)) -» x0v-c)      ® (npb(nnp :x0npb(nnp :xinpb :x2)) -» x0xi x2)</doubt><doubt alpha="40.0" length="30" tooSmall="False" monospace="0.0">©(npb) ©(nnp)    ©(nnp) ©(Npb)</doubt><doubt alpha="58.9" length="73" tooSmall="False" monospace="0.0">® (npb(nnp :x0nnp :xi -» x0xi))® (npb(nnp(viktor)nnp(chemomyrdin))-» v-c)</doubt><p>Figure 4: Derivation forest.</p><p>narization. For example, we may extract the follow­ing two rules:</p><doubt alpha="45.2" length="31" tooSmall="False" monospace="0.0">- a(a(B::CoC:Xi)D:X2)—&gt;X\ Xq x2</doubt><doubt alpha="40.0" length="30" tooSmall="False" monospace="0.0">- a(B:a;oa(C:X\D:^))—&gt;x\ Xq x2</doubt><p>These two rules, however, are not really distinct. They both converge to the following rules if we delete the auxiliary nodes <i>A.</i></p><doubt alpha="41.4" length="29" tooSmall="False" monospace="0.0">- a(B::Coc:x\ d:x2)—&gt;x\ Xq x2</doubt><p>The forest-base rule extraction algorithm pro­duces much larger grammars than the tree-based one, making it difficult to scale to very large training data. From a 50M-word Chinese-to-English parallel corpus, we can extract more than 300 million trans­lation rules, while the tree-based rule extraction al­gorithm gives approximately 100 million. However, the restructured trees from the simple binarization methods are not guaranteed to give the best trees for syntax-based machine translation. What we desire is a binarization method that still produces single parse trees, but is able to mix left binarization and right binarization in the same tree. In the following, we shall use the EM algorithm to learn the desirable bi-narization on the forest of binarization alternatives proposed by the parallel binarization algorithm.</p></section><section number="6" title="Learning how to binarize via the EM algorithm"><p>The basic idea of applying the EM algorithm to choose a restructuring is as follows. We perform a set of binarization operations on a parse tree t. Each binarization // is the sequence of binarizations on the necessary (i.e., factorizable) nodes in t in pre-order. Each binarization / results in a restructured tree r<b>g</b>. We extract rules from (r<b>g</b>, <b>f, a), </b>generating a translation model consisting of parameters (i.e., rule</p><doubt alpha="85.7" length="7" tooSmall="False" monospace="0.0">e-parse</doubt><p>syntax translation       synchronous derivation forests model - viterbi derivations</p><p>Figure 5: Using the EM algorithm to choose restructuring.</p><p>probabilities) 0. Our aim is to obtain the binarization ///* that gives the best likelihood of the restructured training data consisting of (r<b>g</b>, <b>f</b>, <b>a</b>)-tuples. That is</p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">/*</doubt><doubt alpha="57.9" length="19" tooSmall="False" monospace="0.0">arg maxp(rg,f,a|0*)</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">g</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(1)</doubt><p>In practice, we cannot enumerate all the exponen­tial number of binarized trees for a given e-parse. We therefore use the packed forest to store all the binarizations that operate on an e-parse in a com­pact way, and then use the inside-outside algorithm (Lari and Young, 1990; Knight and Graehl, 2004) for model estimation.</p><p>The probability p(r<b>g</b>, f, a) of a (r<b>g</b>, <b>f, a</b>)-tuple is what the basic syntax-based translation model is concerned with. It can be further computed by ag­gregating the rule probabilities p(r) in each deriva­tion w in the set of all derivations Q (Galley et al., 2004; Marcu et al., 2006). That is</p><doubt alpha="55.6" length="9" tooSmall="False" monospace="0.0">P(rg,f,a)</doubt><doubt alpha="71.4" length="7" tooSmall="False" monospace="0.0">yinp(r)</doubt><doubt alpha="100.0" length="6" tooSmall="False" monospace="0.0">wenrew</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(2)</doubt><p>Since it has been well-known that applying EM with tree fragments of different sizes causes over-fitting (Johnson, 1998), and since it is also known that syntax MT models with larger composed rules in the mix significantly outperform rules that min­imally explain the training data (minimal rules) in translation accuracy (Galley et al., 2006), we decom­pose p(rb, f, a) using minimal rules during running of the EM algorithm, but, after the EM restructuring is finished, we build the final translation model using composed rules for evaluation.<page local="7" global="752"/></p><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">4</doubt><p>Figure 5 is the actual pipeline that we use for EM binarization. We first generate a packed e-forest via parallel binarization. We then extract minimal translation rules from the (e-forest, <b>f, a</b>)-tuples, pro­ducing synchronous derivation forests. We run the inside-outside algorithm on the derivation forests until convergence. We obtain the Viterbi derivations and project the English parses from the derivations. Finally, we extract composed rules using Galley et al. (2006)'s (e-tree, <b>f, a</b>)-based rule extraction algo­rithm. This procedure corresponds to the path 13*42 in the pipeline.</p></section><section number="7" title="Experiments"><p>We carried out a series of experiments to compare the performance of different binarization methods in terms of BLEU on Chinese-to-English translation tasks.</p><subsection number="7.1" title="Experimental setup"><p>Our bitext consists of 16M words, all in the mainland-news domain. Our development set is a 925-line subset of the 993-line NIST02 evaluation set. We removed long sentences from the NIST02 evaluation set to speed up discriminative training.</p><p>The test set is the full 919-line NIST03 evaluation set.</p><p>We used a bottom-up, CKY-style decoder that works with binary xRs rules obtained via a syn­chronous binarization procedure (Zhang et al., 2006). The decoder prunes hypotheses using strate­gies described in (Chiang, 2007).</p><p>The parse trees on the English side of the bitexts were generated using a parser (Soricut, 2004) imple­menting the Collins parsing models (Collins, 1997).</p><p>We used the EM procedure described in (Knight and Graehl, 2004) to perform the inside-outside al­gorithm on synchronous derivation forests and to generate the Viterbi derivation forest.</p><p>We used the rule extractor described in (Galley et al., 2006) to extract rules from (e-parse, <b>f, a</b>)-tuples, but we made an important modification: new nodes introduced by binarization will not be counted when computing the rule size limit unless they appear as the rule roots. The motivation is that binarization deepens the parses and increases the number of tree nodes. In (Galley et al., 2006), a composed rule is extracted only if the number of internal nodes it contains does not exceed a limit (i.e., 4), similar to the phrase length limit in phrase-based systems. This means that rules extracted from the restructured trees will be smaller than those from the unrestruc-tured trees, if the <i>X </i>nodes are deleted. As shown in (Galley et al., 2006), smaller rules lose context, and thus give lower translation performance. Ignoring <i>X </i>nodes when computing the rule sizes preserves the unstructured rules in the resulting translation model and adds substructures as bonuses.</p></subsection><subsection number="7.2" title="Experiment results"><p>Table 1 shows the BLEU scores of mixed-cased and detokenized translations of different systems. We see that all the binarization methods improve the baseline system that does not apply any binarization algorithm. The EM-binarization performs the best among all the restructuring methods, leading to 1.0 BLEU point improvement. We also computed the bootstrap p-values (Riezler and Maxwell, 2005) for the pairwise BLEU comparison between the base­line system and any of the system trained from bina-rized trees. The significance test shows that the EM binarization result is statistically significant better than the baseline system (p &gt; 0.005), even though the baseline is already quite strong. To our best knowledge, 37.94 is the highest BLEU score on this test set to date.</p><p>Also as shown in Table 1, the grammars trained from the binarized training trees are almost two times of the grammar size with no binarization. The extra rules are substructures factored out by these bi-narization methods.</p><p>How many more substructures (or translation rules) can be acquired is partially determined by how many more admissible nodes each binariza-tion method can factorize, since rules are extractable only from admissible tree nodes. According to Table 1, binarization methods significantly increase the number of admissible nodes in the training trees. The EM binarization makes available the largest<page local="8" global="753"/></p><p>Table 1: Translation performance, grammar size and # admissible nodes versus binarization algorithms. BLEU scores are for mixed-cased and detokenized translations, as we usually do for NIST MT evaluations.</p><p>Table 2: Binarization bias learned by EM.</p><p>number of admissible nodes, and thus results in the most rules.</p><p>The EM binarization factorizes more admissible nodes because it mixes both left and right binariza-tions in the same tree. We computed the binarization biases learned by the EM algorithm for each nonter­minal from the binarization forest ofheaded-parallel binarizations of the training trees, getting the statis­tics in Table 2. Of course, the binarization bias chosen by left-/right-binarization methods would be 100% deterministic. One noticeable message from Table 2 is that most of the categories are actually bi­ased toward left-binarization, although our motivat­ing example in our introduction section is for NPB, which needed right binarization. The main reason might be that the head sub-constituents of most cat­egories tend to be on the left, but according to the performance comparison between head binarization and EM binarization, head binarization does not suf­fice because we still need to choose the binarization between left and right if they both are head binariza-tions.</p></subsection></section><section number="8" title="Conclusions"><p>In this paper, we not only studied the impact of simple tree binarization algorithms on the perfor­mance of end-to-end syntax-based MT, but also pro­posed binarization methods that mix more than one simple binarization in the binarization of the same parse tree. Binarizing a tree node whether to the left or to the right was learned by employing the EM algorithm on a set of alternative binarizations and by choosing the Viterbi one. The EM binarization method is informed by word alignments such that unnecessary new tree nodes will not be "blindly" in­troduced.</p><p>To our best knowledge, our research is the first work that aims to generalize a syntax-based trans­lation model by restructuring and achieves signifi­cant improvement on a strong baseline. Our work differs from traditional work on binarization of syn­chronous grammars in that we are not concerned with the equivalence of the binarized grammar to the original grammar, but intend to generalize the orig­inal grammar via restructuring of the training parse trees to improve translation performance.</p></section><section title="Acknowledgments"><p>The authors would like to thank David Chiang, Bryant Huang, and the anonymous reviewers for their valuable feedbacks.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>EXPERIMENT</p></td><td class="cell"><p>NIST03-BLEU</p></td><td class="cell"><p># RULES</p></td><td class="cell"><p># ADMISSIBLE NODES IN TRAINING</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>no-bin</p></td><td class="cell"><p>36.94</p></td><td class="cell"><p>63.4M</p></td><td class="cell"><p>7,995,569</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>left binarization right binarization head binarization EM binarization</p></td><td class="cell"><p>37.47 <i>(p</i><i> </i><i>=</i><i> </i>0.047) 37.49 <i>(p</i><i> </i><i>=</i><i> </i>0.044) 37.54 <i>(p</i><i> </i><i>=</i><i> </i>0.086) <b>37.94 </b>(p = 0.0047)</p></td><td class="cell"><p>114.0M 113.0M 113.8M 115.6M</p></td><td class="cell"><p>10,463,148 10,413,194 10,534,339 10,658,859</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>nonterminal</b></p></td><td class="cell"><p><b>left-binariz ation</b></p></td><td class="cell"><p><b>right-binariz ation</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NP</p></td><td class="cell"><p><b>96.97%</b></p></td><td class="cell"><p>3.03%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NP-C</p></td><td class="cell"><p><b>97.49%</b></p></td><td class="cell"><p>2.51%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NPB</p></td><td class="cell"><p>0.25%</p></td><td class="cell"><p><b>99.75%</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>VP</p></td><td class="cell"><p><b>93.90%</b></p></td><td class="cell"><p>6.10%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PP</p></td><td class="cell"><p><b>83.75%</b></p></td><td class="cell"><p>16.25%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ADJP</p></td><td class="cell"><p><b>87.83%</b></p></td><td class="cell"><p>12.17%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>AD VP</p></td><td class="cell"><p><b>82.74%</b></p></td><td class="cell"><p>17.26%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>S</p></td><td class="cell"><p><b>85.91%</b></p></td><td class="cell"><p>14.09%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>s-c</p></td><td class="cell"><p>18.88%</p></td><td class="cell"><p><b>81.12%</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SBAR</p></td><td class="cell"><p><b>96.69%</b></p></td><td class="cell"><p>3.31%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>QP</p></td><td class="cell"><p><b>86.40%</b></p></td><td class="cell"><p>13.60%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PRN</p></td><td class="cell"><p><b>85.18%</b></p></td><td class="cell"><p>14.82%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>WHNP</p></td><td class="cell"><p><b>97.93%</b></p></td><td class="cell"><p>2.07%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NX</p></td><td class="cell"><p><b>100%</b></p></td><td class="cell"><p>0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SINV</p></td><td class="cell"><p><b>87.78%</b></p></td><td class="cell"><p>12.22%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PRT</p></td><td class="cell"><p><b>100%</b></p></td><td class="cell"><p>0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SQ</p></td><td class="cell"><p><b>93.53%</b></p></td><td class="cell"><p>6.47%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CONJP</p></td><td class="cell"><p>18.08%</p></td><td class="cell"><p><b>81.92%</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><references><p>E. Charniak. 2000. A maximum-entropy-inspired parser.</p><p>In <i>Proceedings of the Human Language Technology Conference ofthe North American Chapter ofthe As­sociation for Computational Linguistics, </i>Seattle, May.</p><p>David Chiang. 2007. Hierarchical phrase-based transla­tion. <i>Computational Linguistics, </i>33(2).</p><p>Michael Collins.   1997.  Three generative, lexicalized models for statistical parsing. In <i>Proceedings ofthe</i> <i>35 th Annual Meeting of the Association for Computa­tional Linguistics (ACL), </i>pages 16-23, Madrid, Spain, July.<page local="9" global="754"/></p><p>A. P. Dempster, N. M. Laird, and D. B. Rubin. 1977. Maximum likelihood from incomplete data via the EM algorithm. <i>Journal ofthe Royal Statistical Society, </i>39(1):1-38.</p><p>Jason Eisner. 2003. Learning non-isomorphic tree map­pings for machine translation. In <i>Proceedings ofthe 40th Annual Meeting of the Association for Compu­tational Linguistics (ACL), </i>pages 205-208, Sapporo,</p><p>July.</p><p>M. Galley, M. Hopkins, K. Knight, and D. Marcu. 2004. What's in a Translation Rule? In <i>Proceedings of the Human Language TechnologyConference and the North American Association for Computational Lin­guistics (HLT-NAACL), </i>Boston, Massachusetts.</p><p>M. Galley, J. Graehl, K. Knight, D. Marcu, S. DeNeefe, W. Wang, and I. Thayer. 2006. Scalable Inference and Training of Context-Rich Syntactic Models. In <i>Pro­ceedings ofthe 44th Annual Meetingofthe Association for Computational Linguistics (ACL).</i></p><p>Joshua Goodman. 1999. Semiring parsing. <i>Computa­tionalLinguistics, </i>25(4):573-605.</p><p>M. Johnson. 1998. The DOP estimation method is biased and inconsistent. <i>Computational Linguistics, </i>28(1):71-76.</p><p>K. Knight and J. Graehl. 2004. Training Tree Transduc­ers. In <i>Proceedings ofNAACL-HLT.</i></p><p>K. Lari and S. Young. 1990. The estimation of stochastic context-free grammars using the inside-outside algo­rithm. <i>Computer Speech and Language, </i>pages 35-56.</p><p>Daniel Marcu, Wei Wang, Abdessamad Echihabi, and Kevin Knight. 2006. SPMT: Statistical machine translation with syntactified target language phraases.</p><p>In <i>Proceedings of EMNLP-2006, pp. 44-52, </i>Sydney, Australia.</p><p>M. Marcus, B. Santorini, and M. Marcinkiewicz. 1993. Building a large annotated corpus of En­glish: The Penn Treebank. <i>Computational Linguistics, </i>19(2):313-330.</p><p>I. Dan Melamed, Giorgio Satta, and Benjamin Welling­ton. 2004. Generalized multitext grammars. In <i>Pro­ceedings ofthe 42nd Annual Meeting ofthe Associa­tion for Computational Linguistics (ACL), </i>Barcelona, Spain.</p><p>Stefan Riezler and John T. Maxwell. 2005. On some pitfalls in automatic evaluation and significance test­ing for MT. In <i>Proc. ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summa­rization.</i></p><p>Radu Soricut. 2004. A reimplementation of Collins's parsing models. Technical report, Information Sci­ences Institute, Department of Computer Science Uni­versity of Southern California.</p><p>Hao Zhang, Liang Huang, Daniel Gildea, and Kevin Knight. 2006. Synchronous binarization for machine translation. In <i>Proceedings ofthe HLT-NAACL.</i></p></references></body></article>