<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="1351"/><title>Clause-Based Reordering Constraints to Improve Statistical Machine Translation</title><pubinfo>Proceedings of the 5th International Joint Conference on Natural Language Processing,pages 1351-1355, Chiang Mai, Thailand, November 8 - 13, 2011. ©2011 AFNLP</pubinfo><author surname="Ramanathan" givenname="Ananthakrishnan"><org  name="IBM Research India" country="India" city="New Delhi"/></author><author surname="Bhattacharyya" givenname="Pushpak"><org  name="IBM Research India" country="India" city="New Delhi"/></author><author surname="Visweswariah" givenname="Karthik"><org  name="IBM Research India" country="India" city="New Delhi"/></author><author surname="Ladha" givenname="Kushal"><org  name="IBM Research India" country="India" city="New Delhi"/></author><author surname="Gandhe" givenname="Ankur"><org  name="IBM Research India" country="India" city="New Delhi"/></author></firstpageheader><frontmatter><p><b>Clause-Based Reordering Constraints to Improve Statistical Machine</b></p><p><b>Translation</b></p><p><b>Ananthakrishnan Ramanathan<footnote anchor="1"/>, Pushpak Bhattacharyya<footnote anchor="2"/>, Karthik Visweswariah<footnote anchor="1"/>,</b></p><p><b>Kushal Ladha<footnote anchor="2"/> and Ankur Gandhe<footnote anchor="1"/></b></p><p><footnote anchor="1"/>IBM Research India <footnote anchor="2"/>Department of CSE, IIT Bombay</p><p>|aramana2,v-karthik,ankugand} @in.ibm.com {pb,kush} @cse.iitb.ac.in</p></frontmatter><abstract>We demonstrate that statistical machine translation (SMT) can be improved sub­stantially by imposing clause-based re­ordering constraints during decoding. Our analysis of clause-wise translation of dif­ferent types of clauses shows that it is ben­eficial to apply these constraints for finite clauses, but not for non-finite clauses. In our experiments in English-Hindi trans­lation with an SMT system (DTM2), on a test corpus containing around 850 sen­tences with manually annotated clause boundaries, BLEU improves to 20.4 from the baseline score of 19.4. This statisti­cally significant improvement is also con­firmed by subjective (human) evaluation. We also report preliminary work on au­tomatically identifying the kind of clause boundaries appropriate for enforcing re­ordering constraints. </abstract></header><body><section number="1" title="Introduction"><p>It has been recognized widely that reordering is the Achilles' heel of most SMT models (Birch et al., 2009; Hoang and Koehn, 2009), espe­cially when applied to languages with large syn­tactic differences. In our experiments in English-Hindi SMT, we observe that it is quite frequent in multi-clause sentences for phrases to move out of their respective clauses due to incorrect re­ordering. While such mistakes can be avoided by restricting reordering over clause boundaries, this paper demonstrates that such a strategy works well only for finite clauses, and not for non-finite clauses.</p><p>Clause-wise or part-by-part translation has been a standard approach in traditional transfer-based systems. In Systran, as described by Hutchins and Somers (1992), conjunct and relative clauses were segmented in a preprocessing step. Simi­lar methods were used in the the Stanford Ma­chine Translation project (Wilks, 1973). Chan-drasekar (1994) applies a sentence simplification method to machine translation, where sentences are split at conjunctions, relative pronouns, etc., before translation. Rao <i>et al. </i>(2000) describe a clause-wise translation strategy within an English-Hindi transfer-based MT system.</p><p>In the context of SMT, Koehn and Knight (2003) use a dedicated noun phrase (NP) translation subsystem to obtain significant improvements in German-English translation. Other similar work includes (Watanabe et al., 2003) for Japanese-English SMT and (Hewavitha-rana et al., 2007) for Arabic-English. Sudoh et al. (2010) propose methods to perform clause-level alignment of the parallel corpus, and to translate clauses (all clauses identified by a syn­tactic parser) as a unit to improve long-distance reordering in a specialized domain - English-Japanese translation of research paper abstracts in the medical domain.</p><p>While our approach draws from many of the above, it is novel in the following ways: (i) We provide an analysis of different clause types in translation - we show that only some kinds of clauses benefit from the use of reordering con­straints, and (ii) We demonstrate significant im­provements using this strategy for English-Hindi SMT in a general domain.</p></section><section number="2" title="Problem Statement"><p>We analyzed a set of 225 sentences translated through the baseline system. Of the 120 sentences in this set which had more than one clause, 45 sentences were found to have inter-clause reorder­ing problems; that is, some words or phrases are wrongly placed in a clause where they do not be­long. Such long-range reordering problems obvi­ously have a serious detrimental effect on translation quality, and it is quite likely that even limiting the reordering problems to the respective clauses will aid comprehensibility.<page local="2" global="1352"/></p></section><section number="3" title="Analysis of Clause Types in Translation"><p>We will now look at how finite and non-finite clauses behave in translation. Finite clauses are basically tensed clauses, while non-finite clauses are untensed.</p><p><b>• Finite clauses: </b>Finite clauses appear most commonly in conjunct or relative construc­tions. In such cases, each finite clause can be translated separately and glued together us­ing the correct translation of the conjunction or the relative pronoun. While this strategy works very well for conjunct clauses, in rela­tive clauses the resulting translation is some­times not completely natural, but it is never­theless always clearly understandable. Con­sider the following example:</p><p><b>Input: </b>The boy, who stays in Delhi, won the match.</p><p>Using a clause-wise translation approach, this would be translated as,</p><p>H^«hi, sit <b><i>fkwr   </i></b><b>^ttît </b>ft, #w <i>ladakaa, jo dillii men rehtaa hai, match jiita gayaa</i> <i>Gloss:</i><i>boy, </i>who Delhi in stays, match won</p><p>However, a more natural translation would be using correlatives:</p><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">,</doubt><p><i>jo ladakaa dillii men rehtaa hai, vo match ji-ita gayaa</i></p><p>Gloss:which boy Delhi in stays, he match won</p><p>Certain kinds of nominal clauses also result in similar disfluencies. For example, in the sentence "The playground is where the statue is situated", putting a reordering constraint around the clause "where the statue is situ­ated" results in an unnatural translation.</p><p><b>• Non-finite clauses: </b>Compared to finite clauses, the translation of a non-finite clause is much more dependent on its role within the</p><p>sentence. There are two main issues: (i) all or part of the non-finite clause could get re­ordered with the surrounding clause, or (ii) the overall meaning is conveyed by a phrase or group of words from the non-finite clause and a surrounding or neighbouring clause.</p><p>Taking <i>to-infinitives </i>as the example category, consider the following constructions:</p><p>a. <i>To-infinitive clause in raising construc­tions: </i>Here the embedded clause with the raised element is translated as a finite clause with <i>that </i>as a complementizer. For example, <b>Input:</b><b> </b>John is certain to win, which is translated as "It is certain that John will win":</p><p>TIT <i>WW </i>I" <b>f% </b><i>WM </i><b>^IcMll </b><b>I </b><i>yaha tay hai ki John jiitega Gloss</i>:it certain is that John will-win</p><p>It is the combination of "certain" (or similar word) and the to-infinitive that results in this translation.</p><p>b. <i>To-infinitive with a copular verb: </i>In such cases, the infinitive is inflected with a WÏT <i>(kara) </i>ending to indicate sequentiality:</p><p><b>Input: </b>John was happy to see him <i>John use dekhakara khusha huaa</i> <i>Gloss</i>:John him-to seen-having happy was/became</p><doubt alpha="59.1" length="22" tooSmall="False" monospace="0.0">STFT 3#im.4f3T JT3TT I</doubt><p>Here, the non-finite clause "to see him", gets reordered into the main clause, something that will not be possible if reordering con­straints are applied.</p><p>Similarly, there are differences in the way other non-finite clauses (with -ed and -ing participles) are treated. The point being made here is that handling these differences is crucial to the correct meaning being con­veyed. In other words, simply translating non-finite clauses separately with reordering constraints around them, will not lead to good translation, because the translation of these clauses is often dependent on the superor­dinate clause, and also there is reordering between these clauses and the superordinate clause.</p><page local="3" global="1353"/><p>Table 1: Manually identified clauses. 5: 99% statistical significance; pp: 95% statistical significance</p><p>Based on the above analysis, we are encouraged to test the hypothesis that it is helpful to put re­ordering constraints around finite clauses, but not around non-finite clauses.</p><p>Finiteness is one broad dimension along which clauses may be categorized. Subordinate clauses can be further classified based on the position in which they occur in the sentence - that is, whether they occur in the complement position, the speci­fier position, or the adjunct position. We may also classify clauses based on whether they play an ad­jectival, nominal, or adverbial role in the sentence. The focus of the present work, however, is only on the finiteness aspect.</p></section><section number="4" title="Experiments"><p>In this section, we first describe briefly our base­line system and our approach to handling clauses within this system. We then summarize the datasets used and our evaluation methodology, be­fore describing the results of various experiments.</p><subsection number="4.1" title="Approach"><p>The baseline system we use is <i>DTM2 </i>(a direct translation model) (Ittycheriah and Roukos, 2007). The word-alignments were done using an HMM aligner. We used the best-performing parameter setting in the decoder <footnote anchor="1"/>.</p><p>A beam search decoder (Ittycheriah and Roukos, 2007) similar to other phrase-based de­coders (Tillmann and Ney, 2003) is used for trans­lation. The reordering restriction is applied by treating the relevant clause-boundaries as <i>barriers, </i>and putting a hard constraint on reordering across barriers - that is, during decoding, if a new hy­pothesis requires reordering over a barrier, the hy­pothesis is discarded.</p><footnote label="1">Specifically, the skip-length (distortion limit) was set to 8. Lower skip-lengths led to much poorer performance. For example, with a skip-length of 4, the BLEU score dropped by around 2 points compared to a skip-length of 8</footnote><p>We experiment with clause boundaries identi­fied in three ways: (i) manually (section 4.3.1), (ii) automatically using a constituency parser (sec­tion 4.3.2), and (iii) automatically using a CRF-based clause-boundary classifer using part-of-speech and parser features (section 4.3.3).</p></subsection><subsection number="4.2" title="Data and Evaluation"><p>The system was trained on a parallel corpus with 289k sentences (combining the LDC English-Hindi parallel texts with internal datasets) consist­ing of various domains including news, and tested on 844 sentences. The language model was trained on around 1.5 million sentences.</p><p>Automatic evaluation was done using BLEU (Papineni et al., 2001) with a single reference translation per sentence. Statistical significance of the test results with BLEU was computed using paired bootstrap resam­pling (Koehn, 2004). Subjective evaluation was performed on hundred randomly selected multi-clause sentences, using a five-point scale.</p></subsection><subsection number="4.3" title="Results"><subsubsection number="4.3.1" title="Manual annotation: finite vs. non-finite clauses"><p>For this experiment, the sentences in the test sets were manually annotated with finite and non-finite clause boundaries. The results in table 1 indicate that reordering constraints around finite clauses work much better than around non-finite clauses.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>BLEU</p></td><td class="cell"><p>Adequacy</p></td><td class="cell"><p>Fluency</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>baseline</p></td><td class="cell"><p>19.4</p></td><td class="cell"><p>2.04</p></td><td class="cell"><p>2.41</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>finite</p></td><td class="cell"><p>20.4"<footnote anchor="5"/></p></td><td class="cell"><p>2.32<footnote anchor="0"/></p></td><td class="cell"><p>2.67<footnote anchor="0"/></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>non-finite</p></td><td class="cell"><p>19.6</p></td><td class="cell"><p></p></td><td class="cell"><p>2.5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>finite + non-finite</p></td><td class="cell"><p>19.8^</p></td><td class="cell"><p>2.17</p></td><td class="cell"><p>2.51^</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 2: Automatically identified finite clauses" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Method</p></td><td class="cell"><p>ACI accuracy</p></td><td class="cell"><p>BLEU</p></td><td class="cell"><p>Adequacy</p></td><td class="cell"><p>Fluency</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>parser</p></td><td class="cell"><p>0.42</p></td><td class="cell"><p>19.3</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CRF - word and pos</p></td><td class="cell"><p>0.69</p></td><td class="cell"><p>19.8^</p></td><td class="cell"><p>2.27<footnote anchor="0"/></p></td><td class="cell"><p>2.59<footnote anchor="0"/></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="4" global="1354"/></subsubsection><subsubsection number="4.3.2" title="Automatic clause identification using a parser"><p>The goal of this experiment was to determine whether clauses obtained from a parser could be used for the purpose of imposing clause-based constraints. We performed the experiment by us­ing a state-of-the-art maxent parser (Ratnaparkhi, 1999) to mark all clause boundaries (clause-level nodes based on the Penn Treebank II Style brack­eting guidelines - S, S-BAR, SQ, SBARQ, and SINV). The results were negative (table 2 - row titled <i>parser), </i>indicating that straightforward use of a parser is not sufficient to help in identifying clauses suitable for reordering constraints. The column titled <i>ACI accuracy </i>has the F-measure for automatic clause-boundary identification mea­sured over the entire test corpus.</p></subsubsection><subsubsection number="4.3.3" title="Automatic clause identification using a CRF classifier"><p>We annotated a set of 1500 English sentences with finite-clause boundaries, and used this to train a CRF-based clause-boundary classifier (Ram and Devi, 2008; Tjong et al., 2001). Unigram and bi­gram word features, unigram, bigram and trigram POS features, and the POS of the following verb group were used in the model (Kashioka et al., 2003) <footnote anchor="2"/>. We see (table 2) that a reasonable gain is obtained using the classifier, though not nearly as much as with manual annotation.</p></subsubsection></subsection></section><section number="5" title="Discussion"><p>The subjective evaluation scores reveal that only a few translations degrade in quality when reorder­ing constraints are used with finite clauses (largely due to fluency issues of the kind described in sec­tion 3). Table 3 shows the number of translations that improved (i.e., the average adequacy and flu­ency score increased) due to clause-based transla­tion, and the number which degraded, among the hundred sentences taken up for subjective evaluation. The following is an illustration of the kind of improvements clause-based translation brings:</p><footnote label="2">Adding clause-boundary information from the parser as features in the classifier resulted in a lower F-measure of 0.65</footnote><p><b>Input: </b>America claims that Iran wants to con­tinue its nuclear programme, and secretly builds atomic weapons.</p><p><b>Baseline translation:</b></p><p><i>mnm </i>flwrr <i>fwwtw</i><i> </i>wrw I" <b>i</b></p><p><i>amerika kaa daavaa hai ki usake paramaanu kaaryakrama rahanaa caahate hain aur iraana paramaanu hathiyaara nirmaana karataa hai</i></p><p><i>Gloss: </i>America's claim is that their nuclear pro­gramme continue want and Iran nuclear weapons builds <b>Clause-based translation:</b><b></b></p><p><b>I" Pft </b><b>fur </b><i>mnm </i><b>*i4^ </b><b>snft </b><b>wïït        </b>I" <b>sflr </b><i>mm% ^fkmr </i><i>fawtw</i></p><p><i>amerika kaa daavaa hai ki iran apane para-maanu kaaryakrama ko jaarii rakhanaa caahataa hai aura paramaanu hathiyaara nirmaana kartaa hai</i></p><p><i>Gloss: </i>America's claim is that Iran its nu­clear programme to continue wants and nuclear weapons builds</p></section><section number="6" title="Conclusion and Future Work"><p>We have shown that the quality of English-Hindi statistical machine translation can be improved by performing clause-wise translation. An impor­tant finding in our work is that, in general, finite clauses are more directly suited for such transla­tion compared to non-finite clauses.</p><p>Possible directions of future work include: (i) using clause-based constraints also in training, (ii) learning from word-alignments, the kinds of clauses that can benefit from reordering con­straints, (iii) analyzing the impact of finer-grained clause types (as mentioned in section 3) on transla­tion, (iv) comparing with other SMT models (Gal­ley and Manning, 2010; Chiang, 2005), and (v) using soft constraints instead of hard constraints for reordering (Marton and Resnik, 2008).</p><table caption="Table 3: Number of translations improved/degraded" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>improved</p></td><td class="cell"><p>degraded</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>finite (manual)</p></td><td class="cell"><p>36</p></td><td class="cell"><p>8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>finite (auto)</p></td><td class="cell"><p>35</p></td><td class="cell"><p>17</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>non-finite (manual)</p></td><td class="cell"><p>17</p></td><td class="cell"><p>10</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>finite + non-finite (manual)</p></td><td class="cell"><p>19</p></td><td class="cell"><p>11</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="5" global="1355"/></section><references><p>Adwait Ratnaparkhi, Learning to parse natu­ral language with maximum entropy models, <i>Machine Learning, </i>34(1-3), 1999.</p><p>Birch, A., Blunsom, P., and Osborne, M., A quantitative analysis of reordering phenom­ena, <i>Proceedings of the Fourth Workshop on Statistical Machine Translation, </i>2009.</p><p>Chandrasekar, R., <i>A Hybrid Approach to Machine Translation using Man Machine Communi­cation, </i>Ph.D. thesis, Tata Institute of Fun­damental Research, Mumbai, 1994.</p><p>Chiang, D., A hierarchical phrase-based model for statistical machine translation, <i>Proceedings</i> <i>ofACL, </i>2005.</p><p>Galley, M. and Manning, C., Accurate Non-Hierarchical Phrase-Based Translation, <i>Pro­ceedings ofHLT-NAACL, </i>2010.</p><p>Hewavitharana, S., Lavie, A., and Vogel, S., Ex­periments with a noun-phrase driven statisti­cal machine translation system, <i>Proceedings ofthe MT Summit XI, </i>2007.</p><p>Hoang, H. and Koehn, P., Improving mid-range re-ordering using templates of factors, <i>Pro­ceedings ofEACL, </i>2009.</p><p>Hutchins, J., and Somers, H., <i>An Introduction to Machine Translation, </i>pages 175-189, Aca­demic Press, 1992.</p><p>Ittycheriah, A., and Roukos, S., Direct translation model 2, <i>Proceedings ofNAACL-HLT, </i>2007.</p><p>Kashioka, H., Maruyama, T., and Tanaka, H., Building a parallel corpus for monologue with clause alignment,  <i>Proceedings ofthe</i> <i>MT Summit IX, </i>2003.</p><p>Koehn, P., and Knight, K., Feature-rich statistical translation of noun phrases, <i>Proceedings of</i> <i>ACL, </i>2003.</p><p>Koehn, P., Statistical significance tests for ma­chine translation evaluation, <i>Proceedings of</i> <i>EMNLP, </i>2004.</p><p>Marton, Y. and Resnik, P., Soft syntactic con­straints for hierarchical phrase-based transla­tion, <i>Proceedings ofACL-HLT, </i>2008.</p><p>Papineni, K., Roukos, S., Ward, T., and Zhu, W., BLEU: a method for automatic evaluation of machine translation, <i>IBM Research Report, </i>Thomas J. Watson Research Center, 2001.</p><p>Ram, V., S. and Devi, S., L., Clause bound­ary identification using conditional random fields, <i>Proceedings of CICLing, </i>2008.</p><p>Rao, D., Mohanraj, K., Hegde, J., Mehta, V., and Mahadane, P., A practical framework for syntactic transfer of compound-complex sen­tences for English-Hindi machine translation, <i>Proceedings ofKBCS, </i>2000.</p><p>Sudoh, K., Duh, K., Tsukada, H., Hirao, T., and Nagata, M., Divide and translate: improv­ing long distance reordering in statistical ma­chine translation, <i>Workshop on Statistical Machine Translation and Metrics, </i>2010.</p><p>Tillmann, C. and Ney, H., Word reordering and a dynamic programming beam search algo­rithm for statistical machine translation <i>Com­putational Linguistics, </i>29(1), 2003.</p><p>Tjong, E., F., Sang, K., and, Dejean, H., Introduc­tion to the CoNLL-2001 shared task: clause identification <i>Proceedings of CoNLL, </i>2001.</p><p>Watanabe, T., Sumita, E., and Okuno, H., Chunk-based statistical translation, <i>Proceedings of</i></p><p>Wilks, Y., The Stanford Machine Translation project, <i>Natural Language Processing, </i>Al-gorithmics Press, pages 243-290, 1973.</p></references></body></article>