<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="1124"/><title>Bidirectional Phrase-based Statistical Machine Translation</title><author surname="Finch" givenname="Andrew"><org  name="Keihanna Science City" country="Japan"/></author><author surname="Sumita" givenname="Eiichiro"><org  name="Keihanna Science City" country="Japan"/></author></firstpageheader><frontmatter><p><b>Bidirectional Phrase-based Statistical Machine Translation</b></p><p><b>Andrew Finch</b></p><p>NICT, Keihanna Science City, Kyoto, 619-0288, Japan andrew.finch@nict.go.jp</p><p><b>Eiichiro Sumita</b></p><p>NICT, Keihanna Science City, Kyoto, 619-0288, Japan</p><p>eiichiro.sumita@nict.go.jp</p></frontmatter><abstract>This paper investigates the effect of di­rection in phrase-based statistial machine translation decoding. We compare a typ­ical phrase-based machine translation de­coder using a left-to-right decoding strat­egy to a right-to-left decoder. We also investigate the effectiveness of a bidirec­tional decoding strategy that integrates both mono-directional approaches, with the aim of reducing the effects due to lan­guage specificity. Our experimental eval­uation was extensive, based on 272 differ­ent language pairs, and gave the surprising result that for most of the language pairs, it was better decode from right-to-left than from left-to-right. As expected the rela­tive performance of left-to-right and right-to-left strategies proved to be highly lan­guage dependent. The bidirectional ap­proach outperformed the both the left-to-right strategy and the right-to-left strategy, showing consistent improvements that ap­peared to be unrelated to the specific lan­guages used for translation. Bidirectional decoding gave rise to an improvement in performance over a left-to-right decoding strategy in terms of the BLEU score in 99% of our experiments. </abstract></header><body><section number="1" title="Introduction"><p>Human language production by its very nature is an ordered process. That is to say, words are writ­ten/uttered in a sequence. The current genera­tion of phrase-based statistical machine translation (SMT) systems also generate their target word se­quences according to an order. Since the gener­ation process is symmetrical, there are two pos­sible strategies that could be used to generate the target: from beginning to end; or from end to beginning. Generating the target in the 'wrong' di­rection (the opposite direction to the way in which humans do) is counter intuitive, and possibly as a result of this, SMT systems typically generate the target word sequence in the same order as human language production. However it is not necessar­ily the case that this is most effective strategy for all language pairs. In this paper we investigate the effect of direction in phrase-based SMT decoding.</p><p>For the purposes of this paper, we will refer to target word sequence generation that follows the same order as human language production as <i>forward </i>generation, and generation in the oppo­site direction to human language production as <i>re­verse </i>generation. These are often referred "left-to-right" and "right-to-left" respectively in the litera­ture, but we avoid this notation as many languages are naturally written from right-to-left.</p><p>In earlier work (Watanabe and Sumita, 2002), it was hypothesized that the optimal direction for decoding was dependent on the characteristics of the target language. Their results show that for Japanese to English translation a reverse decod­ing strategy was the most effective, whereas for English to Japanese translation, a forward decod­ing strategy proved superior. In addition they im­plemented a bidirectional decoder, but their re­sults were mixed. For English to Japanese transla­tion, decoding bidirectionally gives higher perfor­mance, but for Japanese to English translation they were unable to improve performance by decod­ing bidirectionally. Their experiments were per­formed using a decoder based on IBM Model 4 using the translation techniques developed at IBM (Brownetal., 1993).</p><p>This work is closely related to the techniques proposed in (Watanabe and Sumita, 2002), but in our case we decode within the framework of a phrase-based SMT system, rather than the IBM model. Our intention was to explore the effect of direction in decoding within the context of a more</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1124</doubt><p><i>Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, </i>pages 1124—1132, Singapore, 6-7 August 2009. ©2009 ACL and AFNLP<page local="2" global="1125"/></p><p>contemporary machine translation paradigm, and to experiment with a broader range of languages. The underlying motivation for our studies however remains the same. Languages have considerably different structure, and certain grammatical con­structs tend to occupy particular positions within sentences of the same language, but different po­sitions across languages. These differences may make it easier to tackle the automatic translation of a sentence in a given language from a partic­ular direction. Our approach differs in that the decoding process of a phrased-based decoder is quite different from that used by (Watanabe and Sumita, 2002) since decoding is done using larger units making the re-ordering process much sim­pler. In (Watanabe and Sumita, 2002) only one language pair is considered, for our experiments we extended this to include translation among 17 different languages including the Japanese and En­glish pair used in (Watanabe and Sumita, 2002). We felt that it was important to consider as many languages as possible in this study, as intuition and evidence from the original study suggests that the effect of direction in decoding is likely to be strongly language dependent.</p><p>The next section briefly describes the mecha­nisms underlying phrase-based decoding. Then we explain the principles behind the forward, re­verse and bidirectional decoding strategies used in our experiments. Section 3 presents the experi­ments we performed. Section 4 gives the results and some analysis. Finally in Section 5, we con­clude and offer possible directions for future re­search.</p></section><section number="2" title="Phrase-based Translation"><p>For our experiments we use the phrase-based ma­chine translation techniques described in (Koehn, 2004) and (Koehn et al., 2007), integrating our models within a log-linear framework (Och and Ney, 2002).</p><p>One of the advantages of a log-linear model is that it is possible to integrate a diverse set of fea­tures into the model. For the decoders used in the experiments in this paper, we included the follow­ing feature functions:</p><p>• An n-gram language model over the target word sequence</p><p>- Ensures the target word sequence is a likely sequence of words in the target language• A phrase translation model</p><p>- Effects the segmentation of the source word sequence, and is also responsible for the transformation of source phrases into target phrases.</p><p>• A target word sequence length model</p><p>- Controls the length of the target word sequence. This is usually a constant term added for each word in the trans­lation hypothesis.</p><p>• A lexicalized distortion model</p><p>- Influences the reordering of the trans­lated source phrases in the target word sequence using lexical context on the boundaries of the phrases being re­ordered.</p><subsection number="2.1" title="Decoding"><p>In a phrase-based SMT decoder, the word se­quence of the target language is typically gener­ated in order in a forward manner. The words at the start of the translation are generated first, then the subsequent words, in order until the fi­nal word of the target word sequence is gener­ated. As the process is phrase-based, the trans­lation is generated in a phrase-by-phrase manner, rather word-by-word. The basic idea is to seg­ment the source word sequence into subsequences (phrases), then translate each phrase individually, and finally compose the target word sequence by reordering the translations of the source phrases. This composition must occur in a particular order, such that target words are generated sequentially from the start (or end in the case of reverse de­coding) of the sentence. The reason that the target needs to be generated sequentially is to allow an n-gram language model to be applied to the partial target word sequence at each step of the decoding process.</p><p>This process is illustrated in Figure 1. In the decoding for both forward and reverse decoders the source sentence is segmented into 2 phrases: "where is" and "the station" (although in this ex­ample the segmentation is the same for both de­coding strategies, it is not necessarily the case since the search processes are different). In the forward decoding process, first the English phrase "the station" is translated into the Japanese phrase "eki wa".  Initially the target sequence consists<page local="3" global="1126"/></p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1125</doubt><p>Left to right</p><p>Right to left</p><p>where is the station</p><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">&lt;s&gt;</doubt><doubt alpha="51.5" length="33" tooSmall="False" monospace="0.0">&lt;s&gt;(eki wa)' &lt;s&gt; eki wa(doko)&lt;/s&gt;</doubt><doubt alpha="47.4" length="19" tooSmall="False" monospace="0.0">Plm = P(eki I &lt;s&gt; )</doubt><doubt alpha="50.0" length="66" tooSmall="False" monospace="0.0">P(wa I eki, &lt;s&gt;)P(doko I wa, eki, &lt;s&gt;)P(&lt;/s&gt; I doko, wa, eki, &lt;s&gt;)</doubt><doubt alpha="25.0" length="4" tooSmall="False" monospace="0.0">&lt;/s&gt;</doubt><doubt alpha="55.6" length="9" tooSmall="False" monospace="0.0">doko)&lt;/s&gt;</doubt><doubt alpha="55.0" length="20" tooSmall="False" monospace="0.0">&lt;s&gt;(eki wa)doko &lt;/s&gt;</doubt><doubt alpha="42.9" length="21" tooSmall="False" monospace="0.0">Plm = P(doko | &lt;/s&gt; )</doubt><doubt alpha="49.3" length="69" tooSmall="False" monospace="0.0">P(wa I doko, &lt;/s&gt;)P(eki I wa, doko, &lt;/s&gt;)P(&lt;s&gt; I eki, wa, doko, &lt;/s&gt;)</doubt><p>Figure 1: The phrase-based decoding process for an English to Japanese translation, in both forward and reverse directions. The n-gram language model probability calculation for the completed translation hypotheses are also shown on the bottom of the figure. See Section 2.1 for a description of the decoding process.</p><p>of only the start of sentence marker "(s)". This marker only serves as context to indicate the start of the sequence for the benefit of the language model. The first target phrase is separated into its component words and each word is added in order to the target word sequence. Each addition causes an application of the language model, hence in Figure 1 the first term of <i>Plm </i>is P(eki|(s)), the second is P(wa|(s)) and so on. For reverse de­coding, the target sentence is generated starting from the end of sentence marker (/s) with the lan­guage model context being to the right of the cur­rent word. For the case of bidirectional decoding, the model probability for the hypothesis is a linear interpolation of the scores for both forward and re­verse hypotheses.</p></subsection><subsection number="2.2" title="Direction in Decoding"><p>Direction in decoding influences both the models used by the decoder and the search process itself. The direction of decoding determines the order in which <i>target </i>words are generated, the source phrases being translated in any order, therefore it is likely to be features of the target language rather than those of the the source language that deter­mine the effect that the decoding direction has on decoder performance.</p></subsection><subsection number="2.2" title=".1 The Language Model"><p>The fundamental difference between the language models of a forward decoder and that of a reverse decoder is the direction in which the model looks for its context. The forward model looks back to the start of the sentence, whereas the reverse model looks forward to the end of the sentence.</p><subsubsection number="2.2.2" title="The Search"><p>Assuming a full search, a unigram language model and no limitations on reordering, the forward and reverse decoding processes are equivalent. When these constraints are lifted, as is the case in the experiments in this paper, the two search processes diverge and can give rise to hypotheses that are different in character.</p><p>The partial hypotheses from early in the search process for forward decoding represent hypothe­ses for the first few words of the target word se­quence, whereas the early partial hypotheses of a reverse decoder hold the last few words. This has two consequences for the search. The first is that (assuming a beam search as used in our ex­periments), certain candidate word sequences in the early stages of the search might be outside the beam and be pruned. The consequence of this is that sentences that start with (or end with in the case of reverse decoding) the pruned word se­quence will not be considered during the remain­der of the search. The second is that word sequences in the partial hypotheses are used in the context of the models used in the subsequent de­coding.<page local="4" global="1127"/> Thus, correctly decoding the start (or end for reverse decoding) of the sentence will benefit the subsequent decoding process.</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1126</doubt></subsubsection></subsection></section><section number="3" title="Experiments"><subsection number="3.1" title="Experimental Data"><p>The experiments were conducted on all possi­ble pairings among 17 languages. A key to the acronyms used for languages together with in­formation about their respective characteristics is given in Table 1.</p><p>We used all of the first ATR Basic Travel Ex­pression Corpus (BTEC1) (Kikui et al., 2003) for these experiments. This corpus contains the kind of expressions that one might expect to find in a phrase-book for travelers. The corpus is similar in character to the IWSLT06 Evaluation Campaign on Spoken Language Translation (Paul, 2006) J-E open track. The sentences are relatively short (see Table 1) with a simple structure and a fairly narrow range of vocabulary due to the limited domain.</p><p>The experiments were conducted on data that contained no case information, and also no punc­tuation (this was an arbitrary decision that we be­lieve had no impact on the results).</p><p>We used a 1000 sentence development corpus for all experiments, and the corpus used for eval­uation consisted of 5000 sentences with a single reference for each sentence.</p></subsection><subsection number="3.2" title="Training"><p>Each instance of the decoder is a standard phrase-based machine translation decoder that operates according to the same principles as the publicly available PHARAOH (Koehn, 2004) and MOSES (Koehn et al., 2007) SMT decoders. In these experiments 5-gram language models built with Witten-Bell smoothing were used along with alex-icalized distortion model. The system was trained in a standard manner, using a minimum error-rate training (MERT) procedure (Och, 2003) with re­spect to the BLEU score (Papineni et al., 2001) on held-out development data to optimize the log-linear model weights. For simplicity, the MERT procedure was performed on independently on the forward and reverse decoders for the bidirectional system, rather them attempting to tune the param­eters for the full system.</p></subsection><subsection number="3.3" title="Translation Engines"><subsubsection number="3.3.1" title="Forward"><p>The forward decoding translation systems used in these experiments represent the baseline of our ex­periments. They consist of phrase-based, multi-stack, beam search decoders commonly used in the field.</p></subsubsection><subsubsection number="3.3.2" title="Reverse"><p>The reverse decoding translation systems used in these experiments were exactly the same as the forward decoding systems. The difference being the that word sequences in the training, develop­ment, and source side of the test corpora were re­versed prior to training the systems. The final out­put of the reverse decoders was reordered in a post processing step before evaluation.</p></subsubsection><subsubsection number="3.3.3" title="Bidirectional"><p>The decoder used for the bidirectional decoding experiments was modified in order to be able to decode both forward and reverse in separate in­stances of the decoder. Models for decoding in forward and reverse directions are loaded, and two decoding instances created. Scores for hypotheses that share the same target word sequence from the two decoders were combined at the end of the de­coding process linearly using equal interpolation weights. Hypotheses that were generated by only one of the component decoders were not pruned. The scores from these hypotheses only had a con­tribution from the decoder that was able to gener­ate them, the contribution from the other decoder being zero.</p></subsubsection></subsection><subsection number="3.4" title="Decoding Constraints"><p>The experiments reported in this paper were con­ducted with loose constraints on the decoding as overconstraining the decoding process could lead to differences between unidirectional and bidirec­tional strategies. More specificially, the decod­ing was done with a beam width of 100, no beam thresholding and no constraints on the reordering process. Figure 2 shows the effect of varying the beam width (stack size) in the search for forward decoder of the English to Japanese translation ex­periment. At the beam width of 100 used in our experiments, the gains from doubling the beam with are small (0.07 BLEU percentage points).</p><p>It is also important to note that a future cost identical to that used in the MOSES decoder<page local="5" global="1128"/></p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1127</doubt><p>Table 1: Key to the languages, corpus statistics and word order. SVO denotes a language that predomi­nantly has subject-verb-object order, and SOV denotes a language that predominantly has subject-object-verb order</p><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.422</doubt><doubt alpha="25.0" length="8" tooSmall="False" monospace="0.0">~_-*-M-X</doubt><doubt alpha="14.3" length="7" tooSmall="False" monospace="0.0">0.414f—</doubt><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">03 !</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">o</doubt><doubt alpha="0.0" length="9" tooSmall="False" monospace="0.0">^ 0.406 —</doubt><doubt alpha="62.5" length="8" tooSmall="True" monospace="0.0">LUIf_l "</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">m</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.398</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">X</doubt><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">0.39 -</doubt><doubt alpha="0.0" length="18" tooSmall="False" monospace="0.0">0 256 512 768 1024</doubt><p>Stack size</p><figure caption="Figure 2: The performance of a forward decoder (En-Ja) with increasing stack size."></figure><p>(Koehn et al., 2007) was also included in the scores for partial hypothesis during the decoding.</p></subsection><subsection number="3.5" title="Computational Overhead"><p>In the current implementation, bidirectional de­coding takes twice as long as a mono-directional system. However, in a multi-threaded environ­ment, each instance of the decoder is able to run on its own thread in parallel, and so this slowdown can be mitigated in some circumstances. Future generations of the bidirectional decoder will more tightly couple the two decoders, and we believe this will lead to faster and more effective search. <b>3.6 Evaluation</b></p><p>The results presented in this paper are given in terms of the BLEU score (Papineni et al., 2001). This metric measures the geometric mean of <i>n-</i>gram precision of n-grams drawn from the output translation and a set of reference translations for that translation.</p><p>There are large number of proposed methods for carrying out machine translation evaluation. Methods differ in their focus of characteristics of the translation (for example fluency or adequacy), and moreover anomolous results can occur if a single metric is relied on. Therefore, we also carried out evaluations using the NIST (Dodding-ton, 2002), METEOR (Banerjee and Lavie, 2005), WER (Hunt, 1989), PER (Tillmann et al., 1997) and TER (Snover et al., 2005) machine translation evaluation techniques.</p></subsection></section><section number="4" title="Results"><p>The results of the experiments in terms of the BLEU score are given in Tables ??, 5, 3 and 3. These results show the performance of the re­verse and bidirectional decoding strategies relative to the usual forward decoding strategy. The cells in the tables that represent experiments in which<page local="6" global="1129"/></p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1128</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Abbreviation</p></td><td class="cell"><p>Language</p></td><td class="cell"><p>#Words</p></td><td class="cell"><p>Avg. sent length</p></td><td class="cell"><p>Vocabulary</p></td><td class="cell"><p>Order</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ar</p></td><td class="cell"><p>Arabie</p></td><td class="cell"><p>806853</p></td><td class="cell"><p>5.16</p></td><td class="cell"><p>47093</p></td><td class="cell"><p>SVO</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>da</p></td><td class="cell"><p>Danish</p></td><td class="cell"><p>806853</p></td><td class="cell"><p>5.16</p></td><td class="cell"><p>47093</p></td><td class="cell"><p>SVO</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>de</p></td><td class="cell"><p>German</p></td><td class="cell"><p>907354</p></td><td class="cell"><p>5.80</p></td><td class="cell"><p>23443</p></td><td class="cell"><p>SVO</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>en</p></td><td class="cell"><p>English</p></td><td class="cell"><p>970252</p></td><td class="cell"><p>6.21</p></td><td class="cell"><p>12900</p></td><td class="cell"><p>SVO</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>es</p></td><td class="cell"><p>Spanish</p></td><td class="cell"><p>881709</p></td><td class="cell"><p>5.64</p></td><td class="cell"><p>18128</p></td><td class="cell"><p>SVO</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>fr</p></td><td class="cell"><p>French</p></td><td class="cell"><p>983402</p></td><td class="cell"><p>6.29</p></td><td class="cell"><p>17311</p></td><td class="cell"><p>SVO</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>id</p></td><td class="cell"><p>Indonesian (Malay)</p></td><td class="cell"><p>865572</p></td><td class="cell"><p>5.54</p></td><td class="cell"><p>15527</p></td><td class="cell"><p>SVO</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>it</p></td><td class="cell"><p>Italian</p></td><td class="cell"><p>865572</p></td><td class="cell"><p>5.54</p></td><td class="cell"><p>15527</p></td><td class="cell"><p>SVO</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ja</p></td><td class="cell"><p>Japanese</p></td><td class="cell"><p>1149065</p></td><td class="cell"><p>7.35</p></td><td class="cell"><p>15405</p></td><td class="cell"><p>SOV</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ko</p></td><td class="cell"><p>Korean</p></td><td class="cell"><p>1091874</p></td><td class="cell"><p>6.98</p></td><td class="cell"><p>17015</p></td><td class="cell"><p>sov</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ms</p></td><td class="cell"><p>Malaysian (Malay)</p></td><td class="cell"><p>873959</p></td><td class="cell"><p>5.59</p></td><td class="cell"><p>16182</p></td><td class="cell"><p>SVO</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ni</p></td><td class="cell"><p>Dutch</p></td><td class="cell"><p>927861</p></td><td class="cell"><p>5.94</p></td><td class="cell"><p>19775</p></td><td class="cell"><p>SVO</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>pt</p></td><td class="cell"><p>Portuguese</p></td><td class="cell"><p>881428</p></td><td class="cell"><p>5.64</p></td><td class="cell"><p>18217</p></td><td class="cell"><p>SVO</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ru</p></td><td class="cell"><p>Russian</p></td><td class="cell"><p>781848</p></td><td class="cell"><p>5.00</p></td><td class="cell"><p>32199</p></td><td class="cell"><p>SVO</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>th</p></td><td class="cell"><p>Thai</p></td><td class="cell"><p>1211690</p></td><td class="cell"><p>7.75</p></td><td class="cell"><p>6921</p></td><td class="cell"><p>SVO</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>vi</p></td><td class="cell"><p>Vietnamese</p></td><td class="cell"><p>1223341</p></td><td class="cell"><p>7.83</p></td><td class="cell"><p>8055</p></td><td class="cell"><p>SVO</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>zh</p></td><td class="cell"><p>Chinese</p></td><td class="cell"><p>873375</p></td><td class="cell"><p>5.59</p></td><td class="cell"><p>14854</p></td><td class="cell"><p>SVO</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Table 2: Baseline BLEU scores for all systems. The figures represent the scores in BLEU percentage points of the baseline left-to-right decoding systems. Source languages are indicated by the column headers, the row headers denoting the target languages.</p><p>the forward strategy outperformed the contrasting strategy are shaded in gray. The numbers in the cells represent the difference in BLEU percentage points for the systems being compared in that cell.</p><p>It is clear from Table 3 that for most of the lan­guage pairs (67% of them for BLEU, and a simi­lar percentage for all the other metrics except ME­TEOR), better evaluation scores were achieved by using a reverse decoding strategy than a forward strategy. This is a surprising result because lan­guage is produced naturally in a forward manner (by definition), and therefore one might expect this to also be the optimal direction for word sequence generation in decoding.</p><subsection number="4.1" title="Word Order Typography"><p>Following (Watanabe and Sumita, 2002), to ex­plain the effects we observe in our results we look to the word order typography of the target lan­guage (Comrie and Vogel, 2000). The word or­der of a language is defined in terms of the order in which you would expect to encounter the finite verb (V) and its arguments, subject (S) and ob­ject (O). In most languages S precedes O and V. Whether or not O precedes or follows V defines the two most prevalent word order types SOV and SVO (Comrie and Vogel, 2000). Two of the target languages in this study (Japanese and Korean) have the SOV word type, the remainder having the SVO word order type. In Table 3 looking at the rows for <i>ja </i>and <i>ko </i>we can see that for both of these languages reverse decoding outperformed forward decoding in only 4 out of 12 experiments. Furthermore these two languages were the two languages that benefited the most (in terms of the number of experimental cases) from forward decoding. The two languages also agree on the best decoding direction for 12 of the 16 language pairs. This apparent correlation may reflect similarities between the two languages (word order type, or other common features of the languages).</p><p>Given this evidence, it seems plausible that word order does account in part for the differences in performance when decoding in differing direc­tions, but this can only be part of the explanation since there are 4 source languages for which re­verse decoding yielded higher performance.</p><p>It should be noted that our results differ from those of (Watanabe and Sumita, 2002) for En­glish to Japanese translation, who observed gains when decoding in the reverse direction for this lan­guage pair. It is hard to compare our results di­rectly with theirs however, due to the differences in the decoders used in the experiments (ours be­ing phrase-based, and theirs based on the IBM ap-<page local="7" global="1130"/></p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1129</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>ar</p></td><td class="cell"><p>da</p></td><td class="cell"><p>de</p></td><td class="cell"><p>en</p></td><td class="cell"><p>es</p></td><td class="cell"><p>fr</p></td><td class="cell"><p>id</p></td><td class="cell"><p>it</p></td><td class="cell"><p>ja</p></td><td class="cell"><p>ko</p></td><td class="cell"><p>ms</p></td><td class="cell"><p>ni</p></td><td class="cell"><p>pt</p></td><td class="cell"><p>ru</p></td><td class="cell"><p>th</p></td><td class="cell"><p>vi</p></td><td class="cell"><p>zh</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ar</p></td><td class="cell"><p>-</p></td><td class="cell"><p>47.8</p></td><td class="cell"><p>48.8</p></td><td class="cell"><p>51.7</p></td><td class="cell"><p>48.8</p></td><td class="cell"><p>47.3</p></td><td class="cell"><p>46.5</p></td><td class="cell"><p>49.2</p></td><td class="cell"><p>29.8</p></td><td class="cell"><p>27.8</p></td><td class="cell"><p>46.9</p></td><td class="cell"><p>49.0</p></td><td class="cell"><p>49.0</p></td><td class="cell"><p>47.8</p></td><td class="cell"><p>39.7</p></td><td class="cell"><p>43.0</p></td><td class="cell"><p>27.8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>da</p></td><td class="cell"><p>58.3</p></td><td class="cell"><p>-</p></td><td class="cell"><p>58.7</p></td><td class="cell"><p>63.0</p></td><td class="cell"><p>58.6</p></td><td class="cell"><p>55.7</p></td><td class="cell"><p>53.5</p></td><td class="cell"><p>58.5</p></td><td class="cell"><p>37.5</p></td><td class="cell"><p>35.1</p></td><td class="cell"><p>54.4</p></td><td class="cell"><p>59.6</p></td><td class="cell"><p>59.0</p></td><td class="cell"><p>55.4</p></td><td class="cell"><p>48.1</p></td><td class="cell"><p>51.7</p></td><td class="cell"><p>35.2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>de</p></td><td class="cell"><p>53.8</p></td><td class="cell"><p>55.5</p></td><td class="cell"><p>-</p></td><td class="cell"><p>59.4</p></td><td class="cell"><p>55.9</p></td><td class="cell"><p>51.9</p></td><td class="cell"><p>50.3</p></td><td class="cell"><p>55.3</p></td><td class="cell"><p>34.2</p></td><td class="cell"><p>32.0</p></td><td class="cell"><p>50.8</p></td><td class="cell"><p>57.0</p></td><td class="cell"><p>55.9</p></td><td class="cell"><p>51.2</p></td><td class="cell"><p>45.7</p></td><td class="cell"><p>48.9</p></td><td class="cell"><p>32.7</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>en</p></td><td class="cell"><p>63.6</p></td><td class="cell"><p>65.8</p></td><td class="cell"><p>64.8</p></td><td class="cell"><p>-</p></td><td class="cell"><p>67.0</p></td><td class="cell"><p>61.0</p></td><td class="cell"><p>58.4</p></td><td class="cell"><p>65.8</p></td><td class="cell"><p>41.1</p></td><td class="cell"><p>38.7</p></td><td class="cell"><p>59.1</p></td><td class="cell"><p>67.6</p></td><td class="cell"><p>66.7</p></td><td class="cell"><p>58.7</p></td><td class="cell"><p>52.8</p></td><td class="cell"><p>57.7</p></td><td class="cell"><p>38.6</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>es</p></td><td class="cell"><p>57.6</p></td><td class="cell"><p>58.2</p></td><td class="cell"><p>58.0</p></td><td class="cell"><p>65.6</p></td><td class="cell"><p>-</p></td><td class="cell"><p>56.6</p></td><td class="cell"><p>54.2</p></td><td class="cell"><p>61.1</p></td><td class="cell"><p>38.3</p></td><td class="cell"><p>36.4</p></td><td class="cell"><p>54.3</p></td><td class="cell"><p>59.6</p></td><td class="cell"><p>62.6</p></td><td class="cell"><p>55.1</p></td><td class="cell"><p>47.6</p></td><td class="cell"><p>51.3</p></td><td class="cell"><p>36.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>fr</p></td><td class="cell"><p>57.8</p></td><td class="cell"><p>58.3</p></td><td class="cell"><p>58.0</p></td><td class="cell"><p>62.3</p></td><td class="cell"><p>58.9</p></td><td class="cell"><p>-</p></td><td class="cell"><p>52.7</p></td><td class="cell"><p>57.4</p></td><td class="cell"><p>39.1</p></td><td class="cell"><p>37.7</p></td><td class="cell"><p>53.8</p></td><td class="cell"><p>58.3</p></td><td class="cell"><p>57.9</p></td><td class="cell"><p>54.8</p></td><td class="cell"><p>47.7</p></td><td class="cell"><p>50.4</p></td><td class="cell"><p>37.6</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>id</p></td><td class="cell"><p>54.7</p></td><td class="cell"><p>52.8</p></td><td class="cell"><p>52.8</p></td><td class="cell"><p>56.6</p></td><td class="cell"><p>53.7</p></td><td class="cell"><p>51.0</p></td><td class="cell"><p>-</p></td><td class="cell"><p>53.1</p></td><td class="cell"><p>37.2</p></td><td class="cell"><p>35.6</p></td><td class="cell"><p>86.4</p></td><td class="cell"><p>53.8</p></td><td class="cell"><p>53.0</p></td><td class="cell"><p>51.3</p></td><td class="cell"><p>46.4</p></td><td class="cell"><p>48.4</p></td><td class="cell"><p>34.9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>it</p></td><td class="cell"><p>54.1</p></td><td class="cell"><p>53.4</p></td><td class="cell"><p>54.4</p></td><td class="cell"><p>59.4</p></td><td class="cell"><p>56.4</p></td><td class="cell"><p>51.8</p></td><td class="cell"><p>49.2</p></td><td class="cell"><p>-</p></td><td class="cell"><p>34.4</p></td><td class="cell"><p>32.8</p></td><td class="cell"><p>49.9</p></td><td class="cell"><p>55.1</p></td><td class="cell"><p>56.2</p></td><td class="cell"><p>50.5</p></td><td class="cell"><p>44.0</p></td><td class="cell"><p>47.0</p></td><td class="cell"><p>33.6</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ja</p></td><td class="cell"><p>38.2</p></td><td class="cell"><p>39.2</p></td><td class="cell"><p>38.6</p></td><td class="cell"><p>41.9</p></td><td class="cell"><p>39.9</p></td><td class="cell"><p>40.2</p></td><td class="cell"><p>40.7</p></td><td class="cell"><p>39.5</p></td><td class="cell"><p>-</p></td><td class="cell"><p>69.4</p></td><td class="cell"><p>40.4</p></td><td class="cell"><p>39.5</p></td><td class="cell"><p>39.7</p></td><td class="cell"><p>37.8</p></td><td class="cell"><p>37.3</p></td><td class="cell"><p>37.2</p></td><td class="cell"><p>52.1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ko</p></td><td class="cell"><p>34.4</p></td><td class="cell"><p>35.3</p></td><td class="cell"><p>34.6</p></td><td class="cell"><p>38.2</p></td><td class="cell"><p>36.3</p></td><td class="cell"><p>36.2</p></td><td class="cell"><p>36.8</p></td><td class="cell"><p>35.6</p></td><td class="cell"><p>66.4</p></td><td class="cell"><p>-</p></td><td class="cell"><p>36.6</p></td><td class="cell"><p>35.6</p></td><td class="cell"><p>36.3</p></td><td class="cell"><p>34.5</p></td><td class="cell"><p>34.2</p></td><td class="cell"><p>34.1</p></td><td class="cell"><p>46.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ms</p></td><td class="cell"><p>54.5</p></td><td class="cell"><p>52.7</p></td><td class="cell"><p>52.6</p></td><td class="cell"><p>56.2</p></td><td class="cell"><p>53.4</p></td><td class="cell"><p>50.6</p></td><td class="cell"><p>82.5</p></td><td class="cell"><p>53.2</p></td><td class="cell"><p>36.8</p></td><td class="cell"><p>34.9</p></td><td class="cell"><p>-</p></td><td class="cell"><p>53.6</p></td><td class="cell"><p>53.4</p></td><td class="cell"><p>51.3</p></td><td class="cell"><p>46.7</p></td><td class="cell"><p>49.2</p></td><td class="cell"><p>34.8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ni</p></td><td class="cell"><p>55.1</p></td><td class="cell"><p>57.3</p></td><td class="cell"><p>58.8</p></td><td class="cell"><p>63.2</p></td><td class="cell"><p>58.5</p></td><td class="cell"><p>54.5</p></td><td class="cell"><p>52.4</p></td><td class="cell"><p>57.1</p></td><td class="cell"><p>36.7</p></td><td class="cell"><p>34.1</p></td><td class="cell"><p>53.4</p></td><td class="cell"><p>-</p></td><td class="cell"><p>58.3</p></td><td class="cell"><p>53.5</p></td><td class="cell"><p>48.7</p></td><td class="cell"><p>50.7</p></td><td class="cell"><p>35.2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>pt</p></td><td class="cell"><p>56.8</p></td><td class="cell"><p>57.7</p></td><td class="cell"><p>57.6</p></td><td class="cell"><p>63.8</p></td><td class="cell"><p>62.0</p></td><td class="cell"><p>55.5</p></td><td class="cell"><p>52.7</p></td><td class="cell"><p>59.7</p></td><td class="cell"><p>37.8</p></td><td class="cell"><p>36.4</p></td><td class="cell"><p>53.4</p></td><td class="cell"><p>58.7</p></td><td class="cell"><p>-</p></td><td class="cell"><p>54.2</p></td><td class="cell"><p>47.1</p></td><td class="cell"><p>50.6</p></td><td class="cell"><p>35.8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ru</p></td><td class="cell"><p>51.4</p></td><td class="cell"><p>49.1</p></td><td class="cell"><p>50.2</p></td><td class="cell"><p>53.3</p></td><td class="cell"><p>52.0</p></td><td class="cell"><p>48.7</p></td><td class="cell"><p>48.6</p></td><td class="cell"><p>51.6</p></td><td class="cell"><p>31.9</p></td><td class="cell"><p>29.5</p></td><td class="cell"><p>49.1</p></td><td class="cell"><p>50.9</p></td><td class="cell"><p>50.5</p></td><td class="cell"><p>-</p></td><td class="cell"><p>41.8</p></td><td class="cell"><p>43.7</p></td><td class="cell"><p>30.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>th</p></td><td class="cell"><p>53.8</p></td><td class="cell"><p>55.0</p></td><td class="cell"><p>54.8</p></td><td class="cell"><p>58.2</p></td><td class="cell"><p>55.8</p></td><td class="cell"><p>53.3</p></td><td class="cell"><p>55.0</p></td><td class="cell"><p>54.8</p></td><td class="cell"><p>41.4</p></td><td class="cell"><p>39.2</p></td><td class="cell"><p>55.4</p></td><td class="cell"><p>55.9</p></td><td class="cell"><p>55.5</p></td><td class="cell"><p>53.0</p></td><td class="cell"><p>-</p></td><td class="cell"><p>56.0</p></td><td class="cell"><p>40.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>vi</p></td><td class="cell"><p>53.6</p></td><td class="cell"><p>53.6</p></td><td class="cell"><p>54.2</p></td><td class="cell"><p>57.4</p></td><td class="cell"><p>54.2</p></td><td class="cell"><p>51.4</p></td><td class="cell"><p>52.3</p></td><td class="cell"><p>53.3</p></td><td class="cell"><p>37.6</p></td><td class="cell"><p>35.8</p></td><td class="cell"><p>53.3</p></td><td class="cell"><p>54.6</p></td><td class="cell"><p>54.4</p></td><td class="cell"><p>51.7</p></td><td class="cell"><p>50.3</p></td><td class="cell"><p>-</p></td><td class="cell"><p>36.2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>zh</p></td><td class="cell"><p>32.0</p></td><td class="cell"><p>33.0</p></td><td class="cell"><p>32.6</p></td><td class="cell"><p>34.6</p></td><td class="cell"><p>33.2</p></td><td class="cell"><p>33.7</p></td><td class="cell"><p>34.2</p></td><td class="cell"><p>33.2</p></td><td class="cell"><p>47.8</p></td><td class="cell"><p>43.5</p></td><td class="cell"><p>33.9</p></td><td class="cell"><p>33.4</p></td><td class="cell"><p>32.6</p></td><td class="cell"><p>32.2</p></td><td class="cell"><p>31.1</p></td><td class="cell"><p>29.7</p></td><td class="cell"><p>-</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Table 3: Gains in BLEU score from reverse decoding over a forward decoding strategy The numbers in the cells are the differences in BLEU percentage points between the systems. Shaded cells indicate the cases where forward decoding give a higher score. Source languages are indicated by the column headers, the row headers denoting the target languages.</p><p>Table 4: Summary of the results using several au­tomatic metrics for evaluation. Numbers in the ta­ble correspond to the percentage of experiments in which the condition at the head of the column was true (for example figure in the first row and first column means that for 98.9 percent of the lan­guage pairs the BLEU score for the bidirectional decoder was better than that of the forward de­coder) proach (Brown et al., 1993)).</p><p>The results were the similar in character when other MT evaluation methods were used. These results are summarized in Table 3.</p></subsection></section><section number="4" title=".2 Bidirectional Decoding"><table caption="Table 5 shows the performance of the bidirectional decoder relative to a forward decoder. As can be"></table><p>seen from the table, in 269 out of the 272 experi­ments the bidirectional decoder outperformed the unidirectional decoder. The gains ranged from a maximum of 1.81 BLEU (translating from Thai to Arabic) points, to a minimum of -0.04 BLEU points (translating from Indonesian to Japanese) with the average gain over all experiments being 0.56 BLEU points. It is clear from our experi­ments that there is much to be gained from decod­ing bidirectionally. Our results were almost unani­mously positive, and in all three negative cases the drop in performance was small.</p></section><section number="5" title="Conclusion"><p>In this paper we have investigated the effects on phrase-based machine translation performance of three different decoding strategies: forward, re­verse and bidirectional. The experiments were conducted on a large set of source and target lan­guages consisting of 272 experiments representing all possible pairings from a set of 17 languages. These languages were very diverse in character and included a broad selection of European and Asian languages. The experimental results re­vealed that for SVO word order languages it is usually better to decode in a reverse manner, and in contrast, for SOV word order languages it is usu-<page local="8" global="1131"/></p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1130</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>ar</p></td><td class="cell"><p>da</p></td><td class="cell"><p>de</p></td><td class="cell"><p>en</p></td><td class="cell"><p>es</p></td><td class="cell"><p>fr</p></td><td class="cell"><p>id</p></td><td class="cell"><p>it</p></td><td class="cell"><p>ja</p></td><td class="cell"><p>ko</p></td><td class="cell"><p>ms</p></td><td class="cell"><p>ni</p></td><td class="cell"><p>pt</p></td><td class="cell"><p>ru</p></td><td class="cell"><p>th</p></td><td class="cell"><p>vi</p></td><td class="cell"><p>zh</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ar</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.87</p></td><td class="cell"><p>0.34</p></td><td class="cell"><p>1.30</p></td><td class="cell"><p>0.93</p></td><td class="cell"><p>1.63</p></td><td class="cell"><p>0.66</p></td><td class="cell"><p>0.58</p></td><td class="cell"><p>0.12</p></td><td class="cell"><p>0.36</p></td><td class="cell"><p>0.85</p></td><td class="cell"><p>0.33</p></td><td class="cell"><p>0.88</p></td><td class="cell"><p>0.22</p></td><td class="cell"><p>1.33</p></td><td class="cell"><p>1.04</p></td><td class="cell"><p>0.88</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>da</p></td><td class="cell"><p></p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.41</p></td><td class="cell"><p>0.71</p></td><td class="cell"><p>0.56</p></td><td class="cell"><p>0.70</p></td><td class="cell"><p>1.10</p></td><td class="cell"><p>0.31</p></td><td class="cell"><p>0.46</p></td><td class="cell"><p>0.07</p></td><td class="cell"><p>0.96</p></td><td class="cell"><p>0.13</p></td><td class="cell"><p>0.62</p></td><td class="cell"><p>0.17</p></td><td class="cell"><p>1.28</p></td><td class="cell"><p>0.71</p></td><td class="cell"><p>0.29</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>de</p></td><td class="cell"><p>0.41</p></td><td class="cell"><p>0.04</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.38</p></td><td class="cell"><p>0.52</p></td><td class="cell"><p>0.15</p></td><td class="cell"><p>0.80</p></td><td class="cell"><p>0.01</p></td><td class="cell"><p>0.47</p></td><td class="cell"><p>0.72</p></td><td class="cell"><p>0.60</p></td><td class="cell"><p>0.25</p></td><td class="cell"><p>0.21</p></td><td class="cell"><p>0.05</p></td><td class="cell"><p>0.47</p></td><td class="cell"><p>0.68</p></td><td class="cell"><p>0.20</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>en</p></td><td class="cell"><p>0.04</p></td><td class="cell"><p>0.05</p></td><td class="cell"><p>0.21</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.05</p></td><td class="cell"><p>0.13</p></td><td class="cell"><p>0.58</p></td><td class="cell"><p>0.02</p></td><td class="cell"><p>0.73</p></td><td class="cell"><p>0.35</p></td><td class="cell"><p>0.39</p></td><td class="cell"><p>0.07</p></td><td class="cell"><p>0.52</p></td><td class="cell"><p>0.05</p></td><td class="cell"><p>0.67</p></td><td class="cell"><p>0.63</p></td><td class="cell"><p>0.29</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>es</p></td><td class="cell"><p>0.14</p></td><td class="cell"><p>0.19</p></td><td class="cell"><p>0.05</p></td><td class="cell"><p>0.35</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.68</p></td><td class="cell"><p>0.01</p></td><td class="cell"><p>0.08</p></td><td class="cell"><p>0.25</p></td><td class="cell"><p>0.31</p></td><td class="cell"><p>0.25</p></td><td class="cell"><p>0.25</p></td><td class="cell"><p>0.17</p></td><td class="cell"><p>0.07</p></td><td class="cell"><p>0.43</p></td><td class="cell"><p>0.44</p></td><td class="cell"><p>0.78</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>fr</p></td><td class="cell"><p>0.37</p></td><td class="cell"><p>0.57</p></td><td class="cell"><p>0.38</p></td><td class="cell"><p>0.66</p></td><td class="cell"><p>0.21</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.36</p></td><td class="cell"><p>0.28</p></td><td class="cell"><p>0.15</p></td><td class="cell"><p>0.45</p></td><td class="cell"><p>0.22</p></td><td class="cell"><p>0.46</p></td><td class="cell"><p>0.64</p></td><td class="cell"><p>0.10</p></td><td class="cell"><p>0.25</p></td><td class="cell"><p>0.58</p></td><td class="cell"><p>0.31</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>id</p></td><td class="cell"><p>0.16</p></td><td class="cell"><p>0.02</p></td><td class="cell"><p>0.31</p></td><td class="cell"><p>1.45</p></td><td class="cell"><p>0.58</p></td><td class="cell"><p>0.50</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.34</p></td><td class="cell"><p>0.03</p></td><td class="cell"><p>0.27</p></td><td class="cell"><p>0.00</p></td><td class="cell"><p>0.42</p></td><td class="cell"><p>0.57</p></td><td class="cell"><p>0.36</p></td><td class="cell"><p>0.53</p></td><td class="cell"><p>1.04</p></td><td class="cell"><p>0.59</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>it</p></td><td class="cell"><p>0.28</p></td><td class="cell"><p>0.72</p></td><td class="cell"><p>0.36</p></td><td class="cell"><p>0.27</p></td><td class="cell"><p>0.08</p></td><td class="cell"><p>0.30</p></td><td class="cell"><p>0.11</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.07</p></td><td class="cell"><p>0.12</p></td><td class="cell"><p>0.37</p></td><td class="cell"><p>0.23</p></td><td class="cell"><p>0.05</p></td><td class="cell"><p>0.37</p></td><td class="cell"><p>0.04</p></td><td class="cell"><p>0.63</p></td><td class="cell"><p>0.37</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ja</p></td><td class="cell"><p>0.36</p></td><td class="cell"><p>0.22</p></td><td class="cell"><p>0.03</p></td><td class="cell"><p>0.03</p></td><td class="cell"><p>0.22</p></td><td class="cell"><p>0.13</p></td><td class="cell"><p>0.64</p></td><td class="cell"><p>0.36</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.21</p></td><td class="cell"><p>0.57</p></td><td class="cell"><p>0.46</p></td><td class="cell"><p>0.08</p></td><td class="cell"><p>0.33</p></td><td class="cell"><p>0.08</p></td><td class="cell"><p>0.83</p></td><td class="cell"><p>0.70</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ko</p></td><td class="cell"><p>0.35</p></td><td class="cell"><p>0.01</p></td><td class="cell"><p>0.31</p></td><td class="cell"><p>0.03</p></td><td class="cell"><p>0.12</p></td><td class="cell"><p>0.07</p></td><td class="cell"><p>0.13</p></td><td class="cell"><p>0.21</p></td><td class="cell"><p>0.42</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.29</p></td><td class="cell"><p>0.07</p></td><td class="cell"><p>0.42</p></td><td class="cell"><p>0.40</p></td><td class="cell"><p>0.44</p></td><td class="cell"><p>0.62</p></td><td class="cell"><p>0.05</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ms</p></td><td class="cell"><p>0.06</p></td><td class="cell"><p>0.49</p></td><td class="cell"><p>0.53</p></td><td class="cell"><p>1.38</p></td><td class="cell"><p>0.99</p></td><td class="cell"><p>0.71</p></td><td class="cell"><p>0.47</p></td><td class="cell"><p>0.34</p></td><td class="cell"><p>0.11</p></td><td class="cell"><p>0.32</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.62</p></td><td class="cell"><p>0.27</p></td><td class="cell"><p>0.10</p></td><td class="cell"><p>0.83</p></td><td class="cell"><p>0.99</p></td><td class="cell"><p>0.11</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ni</p></td><td class="cell"><p>0.26</p></td><td class="cell"><p>0.03</p></td><td class="cell"><p>0.26</p></td><td class="cell"><p>0.30</p></td><td class="cell"><p>0.20</p></td><td class="cell"><p>0.19</p></td><td class="cell"><p>0.47</p></td><td class="cell"><p>0.23</p></td><td class="cell"><p>0.13</p></td><td class="cell"><p>0.06</p></td><td class="cell"><p>0.06</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.08</p></td><td class="cell"><p>0.09</p></td><td class="cell"><p>0.06</p></td><td class="cell"><p>1.00</p></td><td class="cell"><p>0.15</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>pt</p></td><td class="cell"><p>0.03</p></td><td class="cell"><p>0.34</p></td><td class="cell"><p>0.06</p></td><td class="cell"><p>0.51</p></td><td class="cell"><p>0.07</p></td><td class="cell"><p>0.17</p></td><td class="cell"><p>0.06</p></td><td class="cell"><p>0.18</p></td><td class="cell"><p>0.13</p></td><td class="cell"><p>0.65</p></td><td class="cell"><p>0.08</p></td><td class="cell"><p>0.10</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.06</p></td><td class="cell"><p>0.09</p></td><td class="cell"><p>0.85</p></td><td class="cell"><p>0.35</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ru</p></td><td class="cell"><p>0.25</p></td><td class="cell"><p>0.58</p></td><td class="cell"><p>0.67</p></td><td class="cell"><p>0.74</p></td><td class="cell"><p>0.01</p></td><td class="cell"><p>0.48</p></td><td class="cell"><p>0.50</p></td><td class="cell"><p>0.27</p></td><td class="cell"><p>0.41</p></td><td class="cell"><p>0.38</p></td><td class="cell"><p>0.13</p></td><td class="cell"><p>0.38</p></td><td class="cell"><p>0.46</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.88</p></td><td class="cell"><p>0.56</p></td><td class="cell"><p>0.49</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>th</p></td><td class="cell"><p>0.19</p></td><td class="cell"><p>0.28</p></td><td class="cell"><p>0.21</p></td><td class="cell"><p>0.41</p></td><td class="cell"><p>0.05</p></td><td class="cell"><p>0.23</p></td><td class="cell"><p>0.30</p></td><td class="cell"><p>0.00</p></td><td class="cell"><p>0.34</p></td><td class="cell"><p>0.04</p></td><td class="cell"><p>0.25</p></td><td class="cell"><p>0.07</p></td><td class="cell"><p>0.21</p></td><td class="cell"><p>0.08</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.46</p></td><td class="cell"><p>0.25</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>vi</p></td><td class="cell"><p>0.21</p></td><td class="cell"><p>0.34</p></td><td class="cell"><p>0.24</p></td><td class="cell"><p>0.65</p></td><td class="cell"><p>0.72</p></td><td class="cell"><p>0.34</p></td><td class="cell"><p>0.06</p></td><td class="cell"><p>0.59</p></td><td class="cell"><p>0.24</p></td><td class="cell"><p>0.22</p></td><td class="cell"><p>0.19</p></td><td class="cell"><p>0.12</p></td><td class="cell"><p>0.11</p></td><td class="cell"><p>0.18</p></td><td class="cell"><p>0.63</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.15</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>zh</p></td><td class="cell"><p>0.43</p></td><td class="cell"><p>0.26</p></td><td class="cell"><p>0.42</p></td><td class="cell"><p>0.05</p></td><td class="cell"><p>0.15</p></td><td class="cell"><p>0.31</p></td><td class="cell"><p>0.16</p></td><td class="cell"><p>0.28</p></td><td class="cell"><p>0.00</p></td><td class="cell"><p>0.31</p></td><td class="cell"><p>0.40</p></td><td class="cell"><p>0.14</p></td><td class="cell"><p>0.67</p></td><td class="cell"><p>0.18</p></td><td class="cell"><p>0.39</p></td><td class="cell"><p>0.21</p></td><td class="cell"><p>-</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Metric</p></td><td class="cell"><p>Bi&gt;For</p></td><td class="cell"><p>Bi&gt;Rev</p></td><td class="cell"><p>Rev&gt;For</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>BLEU</p></td><td class="cell"><p>98.90</p></td><td class="cell"><p>84.93</p></td><td class="cell"><p>67.65</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NIST</p></td><td class="cell"><p>98.53</p></td><td class="cell"><p>78.31</p></td><td class="cell"><p>75.00</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>METEOR</p></td><td class="cell"><p>99.63</p></td><td class="cell"><p>95.96</p></td><td class="cell"><p>50.74</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>WER</p></td><td class="cell"><p>99.26</p></td><td class="cell"><p>92.85</p></td><td class="cell"><p>66.18</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PER</p></td><td class="cell"><p>98.53</p></td><td class="cell"><p>84.97</p></td><td class="cell"><p>70.59</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>TER</p></td><td class="cell"><p>99.63</p></td><td class="cell"><p>91.18</p></td><td class="cell"><p>68.75</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Table 5: Gains in BLEU score from decoding bidirectionally over a forward decoding strategy. The numbers in the cells are the differences in BLEU percentage points between the systems. Shaded cells indicate the cases where forward decoding gave a higher score. Source languages are indicated by the column headers, the row headers denoting the target languages.</p><p>ally better to decode in a forward direction. Our main contribution has been to show that a bidirec­tional decoding strategy is superior to both mono-directional decoding strategies. It might be argued that the gains arise simply from system combina­tion. However, our systems are combined in a sim­ple linear fashion, and gains will only arise when the second system contributes novel and useful in­formation to into the combination. Furthermore, our systems are trained on two copies of the same data, no additional data is required. The gains from decoding bidirectionally were obtained very consistently, with only loose constraints on the de­coding. This can be seen clearly in Table 5 where the results are almost unanimously positive. More­over, these gains appear to be independent of the linguistic characteristics of the source and target languages.</p><p>In the future we would like to explore the pos­sibilities created by more tightly coupling the for­ward and reverse components of the bidirectional decoder. Scores from partial hypotheses of both processes could be combined and used at each step of the decoding, making the search more in­formed. Furthermore, forward partial hypotheses and reverse hypotheses would 'meet' during de­coding (when one decoding direction has covered words in the source that the other has yet to cover), and provide paths for each other to a final state in the search.</p><p><b>Acknowledgment</b></p><p>This work is partly supported by the Grant-in-Aid for Scientific Research (C) Number 19500137 and "Construction of speech translation founda­tion aiming to overcome the barrier between Asian languages", the Special Coordination Funds for Promoting Science and Technology of the Min­istry of Education, Culture, Sports, Science and Technology, Japan.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>ar</p></td><td class="cell"><p>da</p></td><td class="cell"><p>de</p></td><td class="cell"><p>en</p></td><td class="cell"><p>es</p></td><td class="cell"><p>fr</p></td><td class="cell"><p>id</p></td><td class="cell"><p>it</p></td><td class="cell"><p>ja</p></td><td class="cell"><p>ko</p></td><td class="cell"><p>ms</p></td><td class="cell"><p>ni</p></td><td class="cell"><p>pt</p></td><td class="cell"><p>ru</p></td><td class="cell"><p>th</p></td><td class="cell"><p>vi</p></td><td class="cell"><p>zh</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ar</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.66</p></td><td class="cell"><p>0.51</p></td><td class="cell"><p>1.03</p></td><td class="cell"><p>0.65</p></td><td class="cell"><p>0.75</p></td><td class="cell"><p>0.59</p></td><td class="cell"><p>0.47</p></td><td class="cell"><p>0.46</p></td><td class="cell"><p>0.85</p></td><td class="cell"><p>0.59</p></td><td class="cell"><p>0.69</p></td><td class="cell"><p>0.39</p></td><td class="cell"><p>0.30</p></td><td class="cell"><p>1.81</p></td><td class="cell"><p>1.30</p></td><td class="cell"><p>0.85</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>da</p></td><td class="cell"><p>0.27</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.61</p></td><td class="cell"><p>0.63</p></td><td class="cell"><p>0.38</p></td><td class="cell"><p>0.60</p></td><td class="cell"><p>0.59</p></td><td class="cell"><p>0.29</p></td><td class="cell"><p>1.04</p></td><td class="cell"><p>0.79</p></td><td class="cell"><p>0.69</p></td><td class="cell"><p>0.45</p></td><td class="cell"><p>0.89</p></td><td class="cell"><p>0.27</p></td><td class="cell"><p>1.28</p></td><td class="cell"><p>0.87</p></td><td class="cell"><p>0.47</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>de</p></td><td class="cell"><p>0.52</p></td><td class="cell"><p>0.51</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.54</p></td><td class="cell"><p>0.44</p></td><td class="cell"><p>0.42</p></td><td class="cell"><p>0.70</p></td><td class="cell"><p>0.40</p></td><td class="cell"><p>0.74</p></td><td class="cell"><p>0.45</p></td><td class="cell"><p>0.83</p></td><td class="cell"><p>0.37</p></td><td class="cell"><p>0.28</p></td><td class="cell"><p>0.34</p></td><td class="cell"><p>0.77</p></td><td class="cell"><p>0.90</p></td><td class="cell"><p>0.84</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>en</p></td><td class="cell"><p>0.53</p></td><td class="cell"><p>0.01</p></td><td class="cell"><p>0.32</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.23</p></td><td class="cell"><p>0.25</p></td><td class="cell"><p>0.56</p></td><td class="cell"><p>0.19</p></td><td class="cell"><p>1.11</p></td><td class="cell"><p>0.59</p></td><td class="cell"><p>0.28</p></td><td class="cell"><p>0.27</p></td><td class="cell"><p>0.45</p></td><td class="cell"><p>0.60</p></td><td class="cell"><p>0.89</p></td><td class="cell"><p>0.61</p></td><td class="cell"><p>0.58</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>es</p></td><td class="cell"><p>0.28</p></td><td class="cell"><p>0.48</p></td><td class="cell"><p>0.45</p></td><td class="cell"><p>0.56</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.43</p></td><td class="cell"><p>0.12</p></td><td class="cell"><p>0.26</p></td><td class="cell"><p>0.57</p></td><td class="cell"><p>0.64</p></td><td class="cell"><p>0.56</p></td><td class="cell"><p>0.06</p></td><td class="cell"><p>0.04</p></td><td class="cell"><p>0.24</p></td><td class="cell"><p>1.16</p></td><td class="cell"><p>1.23</p></td><td class="cell"><p>0.68</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>fr</p></td><td class="cell"><p>0.70</p></td><td class="cell"><p>0.33</p></td><td class="cell"><p>0.54</p></td><td class="cell"><p>0.66</p></td><td class="cell"><p>0.46</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.49</p></td><td class="cell"><p>0.57</p></td><td class="cell"><p>0.24</p></td><td class="cell"><p>0.13</p></td><td class="cell"><p>0.11</p></td><td class="cell"><p>0.43</p></td><td class="cell"><p>0.33</p></td><td class="cell"><p>0.55</p></td><td class="cell"><p>0.91</p></td><td class="cell"><p>1.09</p></td><td class="cell"><p>0.57</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>id</p></td><td class="cell"><p>0.24</p></td><td class="cell"><p>0.32</p></td><td class="cell"><p>0.36</p></td><td class="cell"><p>0.93</p></td><td class="cell"><p>0.70</p></td><td class="cell"><p>0.65</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.35</p></td><td class="cell"><p>0.75</p></td><td class="cell"><p>0.77</p></td><td class="cell"><p>0.11</p></td><td class="cell"><p>0.46</p></td><td class="cell"><p>0.69</p></td><td class="cell"><p>0.57</p></td><td class="cell"><p>0.99</p></td><td class="cell"><p>0.85</p></td><td class="cell"><p>0.47</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>it</p></td><td class="cell"><p>0.13</p></td><td class="cell"><p>0.55</p></td><td class="cell"><p>0.32</p></td><td class="cell"><p>0.43</p></td><td class="cell"><p>0.47</p></td><td class="cell"><p>0.51</p></td><td class="cell"><p>0.64</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.65</p></td><td class="cell"><p>0.42</p></td><td class="cell"><p>0.77</p></td><td class="cell"><p>0.51</p></td><td class="cell"><p>0.51</p></td><td class="cell"><p>0.69</p></td><td class="cell"><p>0.85</p></td><td class="cell"><p>0.98</p></td><td class="cell"><p>0.58</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ja</p></td><td class="cell"><p>0.38</p></td><td class="cell"><p>0.62</p></td><td class="cell"><p>0.60</p></td><td class="cell"><p>0.61</p></td><td class="cell"><p>0.38</p></td><td class="cell"><p>0.73</p></td><td class="cell"><p>0.04</p></td><td class="cell"><p>0.43</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.35</p></td><td class="cell"><p>0.05</p></td><td class="cell"><p>0.70</p></td><td class="cell"><p>0.30</p></td><td class="cell"><p>0.38</p></td><td class="cell"><p>0.53</p></td><td class="cell"><p>0.17</p></td><td class="cell"><p>0.02</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ko</p></td><td class="cell"><p>0.49</p></td><td class="cell"><p>0.62</p></td><td class="cell"><p>0.90</p></td><td class="cell"><p>0.40</p></td><td class="cell"><p>0.34</p></td><td class="cell"><p>0.57</p></td><td class="cell"><p>0.47</p></td><td class="cell"><p>0.47</p></td><td class="cell"><p>0.02</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.23</p></td><td class="cell"><p>0.52</p></td><td class="cell"><p>0.20</p></td><td class="cell"><p>0.83</p></td><td class="cell"><p>0.70</p></td><td class="cell"><p>0.44</p></td><td class="cell"><p>0.83</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ms</p></td><td class="cell"><p>0.37</p></td><td class="cell"><p>0.57</p></td><td class="cell"><p>0.63</p></td><td class="cell"><p>0.92</p></td><td class="cell"><p>0.81</p></td><td class="cell"><p>0.75</p></td><td class="cell"><p>0.36</p></td><td class="cell"><p>0.54</p></td><td class="cell"><p>0.70</p></td><td class="cell"><p>1.31</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.76</p></td><td class="cell"><p>0.35</p></td><td class="cell"><p>0.51</p></td><td class="cell"><p>1.14</p></td><td class="cell"><p>0.70</p></td><td class="cell"><p>0.35</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ni</p></td><td class="cell"><p>0.35</p></td><td class="cell"><p>0.14</p></td><td class="cell"><p>0.54</p></td><td class="cell"><p>0.33</p></td><td class="cell"><p>0.30</p></td><td class="cell"><p>0.46</p></td><td class="cell"><p>0.68</p></td><td class="cell"><p>0.69</p></td><td class="cell"><p>0.77</p></td><td class="cell"><p>0.63</p></td><td class="cell"><p>0.44</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.42</p></td><td class="cell"><p>0.67</p></td><td class="cell"><p>0.71</p></td><td class="cell"><p>1.13</p></td><td class="cell"><p>0.55</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>pt</p></td><td class="cell"><p>0.46</p></td><td class="cell"><p>0.21</p></td><td class="cell"><p>0.37</p></td><td class="cell"><p>0.21</p></td><td class="cell"><p>0.17</p></td><td class="cell"><p>0.49</p></td><td class="cell"><p>0.47</p></td><td class="cell"><p>0.24</p></td><td class="cell"><p>0.88</p></td><td class="cell"><p>0.45</p></td><td class="cell"><p>0.54</p></td><td class="cell"><p>0.39</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.41</p></td><td class="cell"><p>0.94</p></td><td class="cell"><p>1.15</p></td><td class="cell"><p>0.90</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ru</p></td><td class="cell"><p>0.69</p></td><td class="cell"><p>0.63</p></td><td class="cell"><p>0.69</p></td><td class="cell"><p>0.77</p></td><td class="cell"><p>0.26</p></td><td class="cell"><p>0.50</p></td><td class="cell"><p>0.79</p></td><td class="cell"><p>0.52</p></td><td class="cell"><p>0.69</p></td><td class="cell"><p>0.90</p></td><td class="cell"><p>0.66</p></td><td class="cell"><p>0.69</p></td><td class="cell"><p>0.40</p></td><td class="cell"><p>-</p></td><td class="cell"><p>1.19</p></td><td class="cell"><p>1.23</p></td><td class="cell"><p>0.47</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>th</p></td><td class="cell"><p>0.90</p></td><td class="cell"><p>0.49</p></td><td class="cell"><p>0.53</p></td><td class="cell"><p>0.77</p></td><td class="cell"><p>0.64</p></td><td class="cell"><p>0.38</p></td><td class="cell"><p>0.21</p></td><td class="cell"><p>0.60</p></td><td class="cell"><p>0.37</p></td><td class="cell"><p>0.96</p></td><td class="cell"><p>0.38</p></td><td class="cell"><p>0.63</p></td><td class="cell"><p>0.68</p></td><td class="cell"><p>0.72</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.33</p></td><td class="cell"><p>0.45</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>vi</p></td><td class="cell"><p>0.64</p></td><td class="cell"><p>0.61</p></td><td class="cell"><p>0.42</p></td><td class="cell"><p>1.09</p></td><td class="cell"><p>0.84</p></td><td class="cell"><p>0.63</p></td><td class="cell"><p>0.34</p></td><td class="cell"><p>0.70</p></td><td class="cell"><p>0.59</p></td><td class="cell"><p>0.39</p></td><td class="cell"><p>0.16</p></td><td class="cell"><p>0.56</p></td><td class="cell"><p>0.36</p></td><td class="cell"><p>0.50</p></td><td class="cell"><p>0.77</p></td><td class="cell"><p>-</p></td><td class="cell"><p>0.53</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>zh</p></td><td class="cell"><p>0.23</p></td><td class="cell"><p>0.48</p></td><td class="cell"><p>0.96</p></td><td class="cell"><p>0.33</p></td><td class="cell"><p>0.49</p></td><td class="cell"><p>0.32</p></td><td class="cell"><p>0.27</p></td><td class="cell"><p>0.43</p></td><td class="cell"><p>0.43</p></td><td class="cell"><p>0.69</p></td><td class="cell"><p>0.31</p></td><td class="cell"><p>0.97</p></td><td class="cell"><p>0.85</p></td><td class="cell"><p>0.23</p></td><td class="cell"><p>0.40</p></td><td class="cell"><p>0.50</p></td><td class="cell"><p>-</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><references><p>Satanjeev Banerjee and Alon Lavie. 2005. Me­teor: an automatic metric for mt evaluation with im­proved correlation with human judgments. In <i>ACL-2005: Workshop on Intrinsic and Extrinsic Evalua­tion Measures for Machine Translation and/or Sum­marization, </i>pages 65-72.</p><p>P. Brown, S. Deila Pietra, V. Delia Pietra, and R.J. Mer­cer. 1993. The Mathematics of Statistical Machine Translation: Parameter Estimation. <i>Computational Linguistics, </i>19(2):263-311.</p><p>Bernard Comrie and Petra M Vogel, editors. 2000. <i>Ap­proaches to the Typography of Word Classes. </i>Mou­ton de Gruyter, Berlin.</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1131</doubt><page local="9" global="1132"/><p>G. Doddington. 2002. Automatic Evaluation of Machine Translation Quality Using N-gram Co­occurrence Statistics. In <i>Proceedings of the HLT Conference, </i>San Diego, California.</p><p>Melvyn J. Hunt. 1989. Figures of merit for assess­ing connected-word recognisers. In <i>In Proceed­ings of the ESCA Tutorial and Research Workshop on Speech Input/Output Assessment and Speech Databases, </i>pages 127-131.</p><p>G. Kikui, E. Sumita, T. Takezawa, and S. Yamamoto. 2003. Creating corpora for speech-to-speech trans­lation. In <i>Proceedings of EUROSPEECH-03, </i>pages 381-384.</p><p>Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowa, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexan­dra Constantin, and Evan Herbst. 2007. Moses: open source toolkit for statistical machine transla­tion. In <i>ACL 2007: proceedings of demo and poster sessions, </i>pages 177-180, Prague, Czeck Republic, June.</p><p>Philipp Koehn. 2004. Pharaoh: a beam search de­coder for phrase-based statistical machine transla­tion models. In <i>Machine translation: from real users to research: 6th conference of AMT A, </i>pages 115-124, Washington, DC.</p><p>Franz Josef Och and Hermann Ney. 2002. Discrimina­tive training and maximum entropy models for sta­tistical machine translation. In <i>In Proceedings of the 40th Annual Meeting of the Association for Compu­tational Linguistics (ACL 2002), </i>pages 295-302.</p><p>Franz J. Och. 2003. Minimum error rate training for statistical machine translation. In <i>Proceedings of the ACL.</i></p><p>Kishore Papineni, Salim Roukos, Todd Ward, and Wei-JingZhu. 2001. Bleu: a method for automatic eval­uation of machine translation. In <i>ACL '02: Proceed­ings of the 40th Annual Meeting on Association for Computational Linguistics, </i>pages 311-318, Morris-town, NJ, USA. Association for Computational Lin­guistics.</p><p>Michael Paul. 2006. Overview of the iwslt 2006 eval­uation campaign. In <i>Proceedings of the IWLST.</i></p><p>Mathew Snover, Bonnie Dorr, Richard Schwartz, John Makhoul, Linnea Micciula, and Ralph Weischedel. 2005. A study of translation error rate with tar­geted human annotation. Technical report, Univer­sity of Maryland, College Park and BBN Technolo­gies, July.</p><p>C. Tillmann, S. Vogel, H. Ney, A. Zubiaga, and H. Sawaf. 1997. Accelerated dp based search for statistical translation. In <i>In European Conf. on Speech Communication and Technology, </i>pages 2667-2670.</p><p>Taro Watanabe and Eiichiro Sumita. 2002. Bidirec­tional decoding for statistical machine translation. In <i>Proceedings of the 19th international conference on Computational linguistics, </i>pages 1-7, Morris-town, NJ, USA. Association for Computational Lin­guistics.</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1132</doubt></references></body></article>