<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="551"/><title>Training Data in Statistical Machine Translation - the More, the Better?</title><pubinfo>Proceedings of Recent Advances in Natural Language Processing,pages 551-556, Hissar, Bulgaria, 12-14 September 2011.</pubinfo><author surname="Gavrila" givenname="Monica"><org  name="University of Hamburg" country="Germany" city="Hamburg"/></author><author surname="Vertan" givenname="Cristina"><org  name="University of Hamburg" country="Germany" city="Hamburg"/></author></firstpageheader><frontmatter><p><b>Training Data in Statistical Machine Translation - The More, the Better? -</b></p><p><b>Monica Gavrila Cristina Vertan</b></p><p>University of Hamburg, Germany University of Hamburg, Germany</p><p>gavrila@informatik. cristina.vertan@uni-hamburg.de uni-hamburg.de</p></frontmatter><abstract>Current statistical machine translation (SMT) systems are stated to be dependent on the availability of a very large training data for producing the language and trans­lation models. Unfortunately, large paral­lel corpora are available for a limited set of language pairs and for an even more lim­ited set of domains. In this paper we investigate the behavior of an SMT system exposed to training data of different sizes and types. Our experimen­tal results show that even parallel corpora of modest sizes can be used for training purposes without lowering too much the evaluation scores. We consider two lan­guage pairs in both translation directions for the experiments: English-Romanian and German-Romanian. </abstract></header><body><section number="1" title="Introduction"><p>Statistical machine translation (SMT) is the most frequently used paradigm, especially when a trans­lation system has to be implemented for a new (less researched) language pair. The pure statisti­cal approach has the advantage that no additional bilingual linguistic expertise is required. Once the training data is available, open-source, language independent systems can be reused. However, the quality of the results is strongly influenced by the size and type of the available training data.</p><p>State-of-the-art literature tends to share the opinion that the larger the data, the better the re­sults. (Suresh, 2010) shows that a larger corpus size for training increases the quality of a Moses-based SMT system, for the Europarl corpus for English-French. The same conclusion appears also in (Koehn et al., 2003), for German-English. In (Brants et al., 2007) experiments for Arabic-English data with billions of tokens are presented and a dependency between the output quality and the size of the training data is also demonstrated.</p><p>Unfortunately, large amount of parallel train­ing data is available only for a restricted num­ber of language pairs and domains. Addition­ally, the training step on large corpora is time and (computing-) resources consuming. On the other hand, smaller corpora can be more easily achieved and have the advantage of requiring less time for training. They also offer the possibility of manu­ally correcting and creating the data.</p><p>Experiments with smaller data for Serbian-English (approx. 2.6K sentences) are presented in (Popovic and Ney, 2006). In the same paper also experimental results for Spanish-English, with dif­ferent data sizes are reported. The systems trained on smaller data give acceptable results. However, the trend remains the same: larger data provides better results.</p><p>For English-Romanian, SMT systems are pre­sented in (Cristea, 2009) and (Ignat, 2009), with BLEU results of 0.5464 and 0.3208, respec­tively. Although both systems use as training and test data parts of the JRC-Acquis corpus, the architecture described in (Cristea, 2009) in­volves the use of linguistic resources and the sys­tem implemented in (Ignat, 2009) uses pivot lan­guages. As long as comparisons are not made on identical training and test data, it is difficult to estimate if, overall, the inclusion of linguis­tic tools increases significantly the performance. The SMT results for Romanian-English, German-Romanian and Romanian-German reported in (Ig-nat, 2009) are 0.3840, 0.2373 and 0.2415, respec­tively. For Romanian-English the BLEU score ob­tained in (Cristea, 2009) is 0.4604.</p><p>Especially for MT systems embedded in on­line applications, which face a dynamic domain change and involve several language pairs, it is ex­tremely important to be aware of the small amount of training data which is available. Such a case is the ATLAS content management system, de­veloped within the EU-Project "Applied Language Technology for Content Management Systems"<footnote anchor="1"/>.<page local="2" global="552"/> In this project a machine translation (MT) engine should be available to translate abstracts from var­ious domains across twelve language pairs.</p><p>In this paper we present the results of a Moses-based SMT system, trained on different types of small size corpora (2.2K). For comparison reasons we additionally consider a larger corpus (330K). Especially with respect to the availability of paral­lel corpora and linguistic resources, Romanian can be considered a lesser resourced language<footnote anchor="2"/>.</p><p>We chose two language pairs (English-Romanian and German-Romanian) in both directions of translations and, in contrast to (Popovic and Ney, 2006), we use for all exper­iments the same language pairs. The language pair Romanian (ro)-German (ge) is particularly interesting as both languages present morphologi­cal and syntactical features which do not occur in English (en) and make the process of translation even more challenging.</p><p>In the following sections we present the Moses-based SMT system used and the data employed in our experiments (Section 2), the translation results and their interpretation (Section 3). Conclusions and further work are described in Section 4.</p></section><section number="2" title="Experimental Setting 2.1   The SMT System"><p>Our MT system follows the description of the baseline architecture provided at the Sixth Work­shop on SMT<footnote anchor="3"/> and uses Moses<footnote anchor="4"/>. Moses imple­ments the statistical paradigm and allows the user to train automatically translation models (TM) for the involved language pair. It is assumed that the user has the required training data. The target lan­guage model (LM) and the word alignment for the parallel corpus are obtained through external ap­plications. We used for our experiments SRILM<footnote anchor="5"/></p><footnote label="1">http://www.atlasproject.eu .</footnote><footnote label="2">While the interest for translation from or into German or English appeared in an early stage of MT, an increased demand for automatic translation from and into Romanian was noticed after the enlargement of the European Union in 2007.</footnote><footnote label="3">http://www.statmt.org/wmt11/baseline . html.</footnote><footnote label="4">http://www.statmt.org/moses/ , (Koehn et al.,</footnote><p>2007).</p><footnote label="5">http://www.speech.sri.com/projects/</footnote><p>srilm/ , (Stolcke, 2002).</p><p>and GIZA++ <footnote anchor="6"/>, respectively.</p><p>Two changes have been made to the specifica­tions of the Workshop on SMT: we left out the tun­ing step and considered the language model (LM) order 3 (instead of 5). Leaving out the tuning step is motivated by previous experiments we made, in which the tuned system did not always provide the best results. A reason for choosing the order three for the LM was provided by the results shown in the presentation of the SMART<footnote anchor="7"/> project (Rousu, 2008), in which it was stated that "3-grams work generally the best".</p><subsection number="2.2" title="Data Description"><p>We want to study the influence of the training data on the translation results. Therefore, we use for our experiments three corpora of different sizes, which have various compilation methods: <b>JRC-Acquis_L </b>(a large-size parallel corpus, automati­cally aligned at sentence level), <b>JRC-Acquis_S </b>(a small-size parallel corpus, automatically aligned at sentence level), and <b>RoGER_S </b>(a small-size technical manual, manually compiled and aligned at sentence level).</p><p>The first corpus <b>(JRC-Acquis_L) </b>is part of the JRC-Acquis<footnote anchor="8"/>, a freely available parallel corpus in 22 languages, which consists of European Union documents of legal nature. In order to reduce er­rors we considered only the one-to-one sentence alignments obtained with Vanilla<footnote anchor="9"/>. In fact, the alignment is realized at paragraph level<footnote anchor="10"/>, where a <i>paragraph </i>can be a simple or complex sen­tence, or a sub-sentential phrase (such as a noun phrase). More details on JRC-Acquis can be found in (Steinberger et al., 2006).</p><p>Filtering the sentence alignments had different influences on the data-size. For English - Roma­nian, from 391324 links <i>(&lt;</i><i> </i><i>p</i><i> </i>&gt;-alignments) in 6557 documents, only 336509 links were retained. Subsequently, the cleaning step<footnote anchor="11"/> of the SMT sys­tem reduced the translation model (TM) to 240219 links. This represents approx. 61.38% of the ini­tial corpus. For German - Romanian, from 391972</p><footnote label="6">Details on GIZA++ can be found in (Och and Ney, 2003).</footnote><footnote label="7">www.smart-project.eu - last accessed on June</footnote><p>27th, 2011.</p><footnote label="8">The JRC Collection of the Acquis Communautaire: http://wt.jrc.it/lt/Acquis/ .</footnote><footnote label="9">See http://nl.ijs.si/telri/Vanilla/ .</footnote><footnote label="10">The tag &lt;p&gt;  from the initial HTML files.</footnote><footnote label="11">The cleaning step is integrated in Moses and supposes the elimination of sentences longer than 40 words.</footnote><page local="3" global="553"/><p>links in 6558 documents, only 324448 links were considered for the LM. The TM was reduced to 238172 links (i.e 60.76% of the initial corpus).</p><p>The corpus is not manually corrected. There­fore, translation, alignment or spelling errors might influence negatively the output quality.</p><p>The tests were run on 897 (3 x 299) sentences, which were not used for training. Sentences were randomly removed from different parts of JRC-Acquis to ensure a relevant lexical, syntactic and semantic coverage. These test sets of 299 sen­tences represent in the following sections the data sets <b>Test 1, Test 2, </b>and <b>Test 3. Test 1+2+3 </b>is formed from all 897 sentences. The test data has no sentence length restriction. Some statistical information on JRC-Acquis_L are summarized in Table 1, in which an item represents a word, a number or a punctuation sign.</p><p>The second corpus we used is <b>JRC-Acquis_S, </b>a sub-corpus of JRC-Acquis_L, which consists of 2333 sentences. The sentences were extracted from the middle of JRC-Acquis_L. From these, 133 sentences were randomly selected as test data. The remaining 2200 sentences represent the training data. The statistics on this corpus are presented in Table 2.</p><p><b>RoGER_S, </b>the third corpus in this paper, is a parallel corpus, consisting of technical texts in four languages<footnote anchor="12"/>, which is manually aligned at sentence level. The text is preprocessed by re­placing concepts such as numbers or web pages with <i>'meta-notions': </i>numbers = NUM, websites = WWW etc. More about the RoGER corpus can be found in (Gavrila and Elita, 2006). RoGERS has the same number of training and test sentences as JRC-Acquis_S. The main difference to JRC-Acquis_S is the correctness of the translations and sentence alignments. The statistical information about this corpus is presented in Table 3.</p></subsection></section><section number="3" title="Evaluation and Interpretation of Translation Results"><subsection number="3.1" title="Automatic Evaluation"><p>The obtained translations have been evaluated us­ing two automatic metrics: BLEU and TER. The choice of the metrics is motivated by the available resources and, for comparison reason, by the re­sults reported in the literature.<page local="4" global="554"/> The comparison was done with only one reference translation, as we work in a realistic scenario with dynamic do­main change (see section 1.)</p><footnote label="12">Ro manian, G erman, E nglish, R ussian.</footnote><table caption="Table 2: Statistics for JRC-Acquis_S." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Data SL</b></p></td><td class="cell"><p><b>No. of items</b></p></td><td class="cell"><p><b>Voc.</b></p></td><td class="cell"><p><b>Average sent. length</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>en - ro</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Training</b></p></td><td class="cell"><p>75405</p></td><td class="cell"><p>3578</p></td><td class="cell"><p>34.27</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test</b></p></td><td class="cell"><p>4434</p></td><td class="cell"><p>992</p></td><td class="cell"><p>33.33</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ro - en</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Training</b></p></td><td class="cell"><p>72170</p></td><td class="cell"><p>5581</p></td><td class="cell"><p>32.80</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test</b></p></td><td class="cell"><p>4325</p></td><td class="cell"><p>1260</p></td><td class="cell"><p>32.51</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ge - ro</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Training</b></p></td><td class="cell"><p>69735</p></td><td class="cell"><p>5929</p></td><td class="cell"><p>31.69</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test</b></p></td><td class="cell"><p>3947</p></td><td class="cell"><p>1178</p></td><td class="cell"><p>29.67</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ro - ge</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Training</b></p></td><td class="cell"><p>75156</p></td><td class="cell"><p>6390</p></td><td class="cell"><p>34.16</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test</b></p></td><td class="cell"><p>4366</p></td><td class="cell"><p>1320</p></td><td class="cell"><p>32.82</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 1: Corpus statistics for JRC-Acquis_L (* voc = vocabulary, sent=sentence)." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Data</b></p></td><td class="cell"><p><b>No. of items</b></p></td><td class="cell"><p><b>Voc.* size</b></p></td><td class="cell"><p><b>Average sent.* length</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>en - ro</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Training (SL)</b></p></td><td class="cell"><p>3579856</p></td><td class="cell"><p>39784</p></td><td class="cell"><p>14.90</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>LM Romanian</b></p></td><td class="cell"><p>9572058</p></td><td class="cell"><p>81616</p></td><td class="cell"><p>28.45</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 1 (SL)</b></p></td><td class="cell"><p>6424</p></td><td class="cell"><p>1048</p></td><td class="cell"><p>21.48</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 2 (SL)</b></p></td><td class="cell"><p>7523</p></td><td class="cell"><p>735</p></td><td class="cell"><p>25.16</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 3 (SL)</b></p></td><td class="cell"><p>5609</p></td><td class="cell"><p>1111</p></td><td class="cell"><p>18.76</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 1+2+3 (SL)</b></p></td><td class="cell"><p>19556</p></td><td class="cell"><p>2345</p></td><td class="cell"><p>21.80</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ro - en</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Training (SL)</b></p></td><td class="cell"><p>3386495</p></td><td class="cell"><p>55871</p></td><td class="cell"><p>14.10</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>LM English</b></p></td><td class="cell"><p>9955983</p></td><td class="cell"><p>55856</p></td><td class="cell"><p>29.59</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 1 (SL)</b></p></td><td class="cell"><p>5672</p></td><td class="cell"><p>1245</p></td><td class="cell"><p>18.97</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 2 (SL)</b></p></td><td class="cell"><p>7194</p></td><td class="cell"><p>923</p></td><td class="cell"><p>24.06</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 3 (SL)</b></p></td><td class="cell"><p>5144</p></td><td class="cell"><p>1355</p></td><td class="cell"><p>17.20</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 1+2+3 (SL)</b></p></td><td class="cell"><p>18010</p></td><td class="cell"><p>2717</p></td><td class="cell"><p>20.08</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ge - ro</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Training (SL)</b></p></td><td class="cell"><p>3256047</p></td><td class="cell"><p>76600</p></td><td class="cell"><p>13.67</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>LM Romanian</b></p></td><td class="cell"><p>9122333</p></td><td class="cell"><p>80484</p></td><td class="cell"><p>28.12</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 1 (SL)</b></p></td><td class="cell"><p>5325</p></td><td class="cell"><p>1140</p></td><td class="cell"><p>17.81</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 2 (SL)</b></p></td><td class="cell"><p>10286</p></td><td class="cell"><p>1439</p></td><td class="cell"><p>34.40</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 3 (SL)</b></p></td><td class="cell"><p>5125</p></td><td class="cell"><p>1292</p></td><td class="cell"><p>17.23</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 1+2+3 (SL)</b></p></td><td class="cell"><p>20763</p></td><td class="cell"><p>3000</p></td><td class="cell"><p>23.15</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ro - ge</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Training (SL)</b></p></td><td class="cell"><p>3453586</p></td><td class="cell"><p>56219</p></td><td class="cell"><p>14.50</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>LM German</b></p></td><td class="cell"><p>8469146</p></td><td class="cell"><p>121969</p></td><td class="cell"><p>26.10</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 1 (SL)</b></p></td><td class="cell"><p>5432</p></td><td class="cell"><p>1294</p></td><td class="cell"><p>18.17</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 2 (SL)</b></p></td><td class="cell"><p>11488</p></td><td class="cell"><p>1663</p></td><td class="cell"><p>38.42</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 3 (SL)</b></p></td><td class="cell"><p>5317</p></td><td class="cell"><p>1388</p></td><td class="cell"><p>17.78</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 1+2+3 (SL)</b></p></td><td class="cell"><p>22237</p></td><td class="cell"><p>3336</p></td><td class="cell"><p>24.79</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 3: Statistics for RoGERS." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Data SL</b></p></td><td class="cell"><p><b>No. of items</b></p></td><td class="cell"><p><b>Voc.</b></p></td><td class="cell"><p><b>Average sent. length</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>en - ro</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Training</b></p></td><td class="cell"><p>27889</p></td><td class="cell"><p>2367</p></td><td class="cell"><p>12.68</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test</b></p></td><td class="cell"><p>1613</p></td><td class="cell"><p>522</p></td><td class="cell"><p>12.13</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ro - en</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Training</b></p></td><td class="cell"><p>28946</p></td><td class="cell"><p>3349</p></td><td class="cell"><p>13.16</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test</b></p></td><td class="cell"><p>1649</p></td><td class="cell"><p>659</p></td><td class="cell"><p>12.40</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ge - ro</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Training</b></p></td><td class="cell"><p>28361</p></td><td class="cell"><p>3230</p></td><td class="cell"><p>12.89</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test</b></p></td><td class="cell"><p>1657</p></td><td class="cell"><p>604</p></td><td class="cell"><p>12.46</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ro - ge</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Training</b></p></td><td class="cell"><p>28946</p></td><td class="cell"><p>3349</p></td><td class="cell"><p>13.16</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test</b></p></td><td class="cell"><p>1649</p></td><td class="cell"><p>659</p></td><td class="cell"><p>12.40</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Although criticized, <b>BLEU (b</b>i<b>l</b>ingual <b>e</b>valuation <b>u</b>nderstudy) is the score mostly used for MT evaluation in the last couple of years. It measures the number of n-grams, of different lengths, of the system output that appear in a set of reference translations. More details about BLEU<footnote anchor="13"/> can be found in (Papineni et al., 2002).</p><p><b>TER<footnote anchor="14"/> </b>calculates the minimum number of ed­its required to get from obtained translations to the reference translations, normalized by the av­erage length of the references. It considers in­sertions, deletions, substitutions of single words and an edit-operation which moves sequences of words. More information about TER can be found in (Snover et al., 2006).</p><p>In Table 4 we present the results we obtained for all three corpora. The boldface numbers rep­resent the highest scores for the specific language combination and evaluation metric.</p><p>The results from Table 4 for Romanian-English are overall similar with state-of-the art evaluation described in Section 1. For Romanian-German our result overtake the system presented in (Ignat, 2009). However, a truly one-to-one comparison is not possible, as we do not work with identical test and training data as the referred systems.</p><p>Even for same training data evaluation results may vary across test sets, as presented in Table 5. Here we show how dependent are the SMT re­sults on the test data. As the size and domain-type of the test data <b>(Test 1 </b>- <b>Test</b><b> </b><b>3)</b><b> </b>is identical, the differences in BLEU and TER score can be ex­plained only through lexical and syntactical varia­tion across test-sets. Some sources for these vari­ations are represented by out-of-vocabulary words (OOV-words) and the number of test sentences al­ready found in training data. An overview of these two aspects in all the three corpora can be seen in Tables 6 and 7. As expected, best results are obtained for the test data set which has less OOV-words and which contains most sentences in the training data: <b>Test</b><b> </b><b>2.</b><b> </b>As it is not the topic of this paper, we will not extend the explanation for these variations or present any possible solutions.</p><footnote label="13">We considered the NIST/BLEU implementation mte-vaLv12, as on http://www.itl.nist.gov/iad/ mig/tests/mt/2 00 8/scoring.html.</footnote><footnote label="14">TER (t ranslation e rror r ate.) as implemented on http://www.cs.umd.edu/- snover/tercom/ - last accessed on 12.01.2010.</footnote><table caption="Table 5: Evaluation results for JRC-Acquis_L" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Score</b></p></td><td class="cell"><p><b>Test 1</b></p></td><td class="cell"><p><b>Test 2</b></p></td><td class="cell"><p><b>Test 3</b></p></td><td class="cell"><p><b>Test 1+2+3</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>en - ro</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>BLEU</b></p></td><td class="cell"><p>0.3997</p></td><td class="cell"><p><b>0.4179</b></p></td><td class="cell"><p>0.3797</p></td><td class="cell"><p>0.4015</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>TER</b></p></td><td class="cell"><p>0.5007</p></td><td class="cell"><p><b>0.4898</b></p></td><td class="cell"><p>0.5208</p></td><td class="cell"><p>0.5023</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ro - en</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>BLEU</b></p></td><td class="cell"><p>0.2545</p></td><td class="cell"><p><b>0.5628</b></p></td><td class="cell"><p>0.4271</p></td><td class="cell"><p>0.4255</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>TER</b></p></td><td class="cell"><p>0.5020</p></td><td class="cell"><p><b>0.3756</b></p></td><td class="cell"><p>0.4684</p></td><td class="cell"><p>0.4457</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ge - ro</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>BLEU</b></p></td><td class="cell"><p>0.2955</p></td><td class="cell"><p><b>0.4244</b></p></td><td class="cell"><p>0.2884</p></td><td class="cell"><p>0.3644</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>TER</b></p></td><td class="cell"><p>0.6200</p></td><td class="cell"><p><b>0.5905</b></p></td><td class="cell"><p>0.6438</p></td><td class="cell"><p>0.6113</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ro - ge</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>BLEU</b></p></td><td class="cell"><p>0.2953</p></td><td class="cell"><p><b>0.4411</b></p></td><td class="cell"><p>0.2939</p></td><td class="cell"><p>0.3726</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>TER</b></p></td><td class="cell"><p>0.6437</p></td><td class="cell"><p><b>0.5588</b></p></td><td class="cell"><p>0.6791</p></td><td class="cell"><p>0.6112</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 4: Evaluation results (all three corpora)." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Score</b></p></td><td class="cell"><p><b>Rogers</b></p></td><td class="cell"><p><b>JRC-Acquis_S</b></p></td><td class="cell"><p><b>JRC-Acquis_L (Test 1+2+3)</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>en - ro</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>BLEU</b></p></td><td class="cell"><p>0.4386</p></td><td class="cell"><p><b>0.4801</b></p></td><td class="cell"><p>0.4015</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>TER</b></p></td><td class="cell"><p><b>0.3784</b></p></td><td class="cell"><p>0.5032</p></td><td class="cell"><p>0.5023</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ro - en</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>BLEU</b></p></td><td class="cell"><p>0.4765</p></td><td class="cell"><p><b>0.4904</b></p></td><td class="cell"><p>0.4255</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>TER</b></p></td><td class="cell"><p><b>0.3465</b></p></td><td class="cell"><p>0.4509</p></td><td class="cell"><p>0.4457</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ge - ro</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>BLEU</b></p></td><td class="cell"><p>0.3240</p></td><td class="cell"><p>0.2811</p></td><td class="cell"><p><b>0.3644</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>TER</b></p></td><td class="cell"><p><b>0.5239</b></p></td><td class="cell"><p>0.6658</p></td><td class="cell"><p>0.6113</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ro - ge</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>BLEU</b></p></td><td class="cell"><p>0.3405</p></td><td class="cell"><p>0.2926</p></td><td class="cell"><p><b>0.3726</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>TER</b></p></td><td class="cell"><p><b>0.5570</b></p></td><td class="cell"><p>0.6816</p></td><td class="cell"><p>0.6112</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 6: Analysis of the test data sets (JRC-Acquis_L)" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Corpus</b></p></td><td class="cell"><p><b>No. of</b></p><p><b>OOV-Words</b></p><p><b>(% from voc. size)</b></p></td><td class="cell"><p><b>Sentences in the corpus</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>JRC-Acquis_L en - ro</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 1</b></p></td><td class="cell"><p>33 (3.15%)</p></td><td class="cell"><p>69 (23.07%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 2</b></p></td><td class="cell"><p>2 (0.27%)</p></td><td class="cell"><p>134 (44.81%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 3</b></p></td><td class="cell"><p>96 (8.64%)</p></td><td class="cell"><p>85 (28.42%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 1+2+3</b></p></td><td class="cell"><p>131 (5.59%)</p></td><td class="cell"><p>288 (21.10%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ro - en</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 1</b></p></td><td class="cell"><p>51 (4.10%)</p></td><td class="cell"><p>69 (23.07%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 2</b></p></td><td class="cell"><p>7 (0.76%)</p></td><td class="cell"><p>117 (39.13%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 3</b></p></td><td class="cell"><p>111 (8.19%)</p></td><td class="cell"><p>81 (27.09%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 1+2+3</b></p></td><td class="cell"><p>169 (6.22%)</p></td><td class="cell"><p>267 (29.76%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ge - ro</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 1</b></p></td><td class="cell"><p>69 (6.05%)</p></td><td class="cell"><p>73 (24.41%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 2</b></p></td><td class="cell"><p>53 (3.68%)</p></td><td class="cell"><p>121 (40.46%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 3</b></p></td><td class="cell"><p>187 (14.47%)</p></td><td class="cell"><p>83 (27.75%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 1+2+3</b></p></td><td class="cell"><p>309 (10.30%)</p></td><td class="cell"><p>277 (30.88%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ro - ge</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 1</b></p></td><td class="cell"><p>44 (3.40%)</p></td><td class="cell"><p>76 (25.41%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 2</b></p></td><td class="cell"><p>97 (5.83%)</p></td><td class="cell"><p>109 (36.45%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 3</b></p></td><td class="cell"><p>105 (7.56%)</p></td><td class="cell"><p>79 (26.42%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test 1+2+3</b></p></td><td class="cell"><p>246 (7.37%)</p></td><td class="cell"><p>264 (29.43%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="5" global="555"/><p>In the next subsection we will show more de­tailed the sensitivity of SMT systems to training and test data size and type.</p></subsection><subsection number="3.2" title="Interpretation of the Results"><p>In Table 4 we presented the variation of BLEU and TER scores across the three corpora. In (Koehn et al., 2003) a log-linear dependency be­tween the size of the training corpora and the BLEU scores was observed. In contrast, our re­sults cannot confirm this dependency for all lan­guage pairs investigated<footnote anchor="15"/>. While for German-Romanian the log-linear dependency seem to be preserved, for English-Romanian the BLEU scores for JRC-Acquis_S are better than the ones for JRC-Acquis_L. Also worth to remark is that the BLEU scores for the other small corpus -ROGERS -, are in the case of English-Romanian between the other two BLEU scores, and in the case of Romanian-English closer to the BLEU score for JRC-Acquis_S. This leads us to the con­clusion that the hypothesis of log-linear depen­dency has to be tested before one decides to invest a lot of work in collecting large data sets. Giving the fact that in both of our experiments, as well as in (Koehn et al., 2003), the log-linear dependency was noticed in case of language pairs involving German, it could be an indication that the German specific morphological features, in special the dynamic word composition, could be a reason for this behavior. The high number of compounds in German may imply a higher data-sparseness, which can be compensated only through large amounts of training data.</p><footnote label="15">We also do not exclude the difference in the results due also to different evaluation methodology. However, this as­pect is not analyzed in this paper</footnote><p>Another interesting observation can be done re­garding the TER Scores. The best TER scores were obtained, independent of the chosen lan­guage pair, for the ROGERS corpus. One expla­nation is the particular syntax of this corpus: tech­nical short sentences, in which the translation usu­ally preserves the SL word order, as far as the syn­tax in both source and target languages allows. In contrast, in JRC-Acquis one finds often reformu­lations or shorter sentences. As TER measures the differences between output and reference transla­tion in number of insertions, deletions and replace­ments, this may be cause of alternation of the TER scores.</p><p>Given the fact that the BLEU scores for the ROGERS corpus are also in line with current state-of-the-art systems, we can conclude that for technical domains a small, manually corrected corpus can be successfully used for obtaining a reasonable translation output.</p><p>All the results we have presented reinforce the idea that SMT is fully dependent on the training and test data size and type and on the evaluation procedure. We will further show how dependent the results are to all the steps involved in the trans­lation and evaluation processes by presenting the results in Table 8. We evaluated the results for the JRC-Acquis_S corpus, when no detokeniza-tion or recasing in the post-processing has been done. In contrast to the information from Table 4, in this last case, the translation evaluation scores are better. This shows that, next to the training and test data itself, sometimes pre- or post-processing steps affect (negatively) the evaluation scores.</p></subsection></section><section number="4" title="Conclusions"><p>The results presented and discussed in this paper let us conclude that there is not always an a priori size which can be recommended for develop­ing a standard SMT systems independent of lan­guage pair and domain.<page local="6" global="556"/> The experiments we made showed (again) how dependent SMT results are on training and test data and on all processing steps. Especially for on-line applications which embed MT systems, where translation domain changes dynamically and a large number of language pairs is involved, a framework criteria for the training and test data is necessary. Our further work in­cludes more experiments with different data (type and size) and language pairs. Also the associated statistical confidence intervals need to be calcu­lated to have a better view on the evaluation re­sults.</p><table caption="Table 7: Analysis of the test data sets (RoGER and JRC-Acquis_S)" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Corpus</b></p></td><td class="cell"><p><b>No. of</b></p><p><b>OOV-Words</b></p><p><b>(% from voc. size)</b></p></td><td class="cell"><p><b>Sentences in the corpus</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>RoGERS</b></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>en - ro</b></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test</b></p></td><td class="cell"><p>60 (11.49%)</p></td><td class="cell"><p>37 (27.81%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ro - en</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test</b></p></td><td class="cell"><p>84(12.75%)</p></td><td class="cell"><p>34 (25.56%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ge - ro</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test</b></p></td><td class="cell"><p>101 (16.72%)</p></td><td class="cell"><p>31 (23.30%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ro - ge</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test</b></p></td><td class="cell"><p>84(12.75%)</p></td><td class="cell"><p>34 (25.56%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>JRC-Acquis_S</b></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>en - ro</b></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test</b></p></td><td class="cell"><p>72 (7.25%)</p></td><td class="cell"><p>38 (28.57%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ro - en</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test</b></p></td><td class="cell"><p>129 (10.23%)</p></td><td class="cell"><p>33 (24.81%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ge - ro</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test</b></p></td><td class="cell"><p>171 (14.51%)</p></td><td class="cell"><p>41 (30.82%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ro - ge</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Test</b></p></td><td class="cell"><p>160 (12.12%)</p></td><td class="cell"><p>40 (30.07%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 8: Results for JRC-Acquis_S (no recasing, no detokenization)" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Language Pair</b></p></td><td class="cell"><p><b>BLEU</b></p></td><td class="cell"><p><b>TER</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>en - ro</b></p></td><td class="cell"><p>0.5359</p></td><td class="cell"><p>0.3586</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ro - en</b></p></td><td class="cell"><p>0.5573</p></td><td class="cell"><p>0.3279</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ge - ro</b></p></td><td class="cell"><p>0.3051</p></td><td class="cell"><p>0.5808</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ro - ge</b></p></td><td class="cell"><p>0.3279</p></td><td class="cell"><p>0.5796</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><section title="Acknowledgments"><p>Ideas and results presented in this paper are part of Monica Gavrila's PhD research, conducted at the University of Hamburg, and of the AT­LAS EU-Project (www.atlasproject.eu), supported through the ICT-PSP-Programme ofthe EU-Commission.</p></section><references><p>Thorsten Brants, Ashok C. Popat, Peng Xu, Franz J. Och, and Jeffrey Dean. 2007. Large language models in machine translation. In <i>Proceedings of the 2007 Joint Conference on Empirical Meth­ods in Natural Language Processing and Computa­tional Natural Language Learning, </i>pages 858-867, Prague, Czech Republic.</p><p>Dan Cristea. 2009. Romanian language technol­ogy and resources go to europe. Presentated at the FP7 Language Technology Informa­tive Days, January, 20-11. To be found at: ftp://ftp.cordis.europe.eu/pub/fp7/ict/docs/language-technologies/cristea_en.pdf - last accessed on 10.04.2009.</p><p>Monica Gavrila and Natalia Elita. 2006. Roger -un corpus paralel aliniat. In <i>In Resurse Lingvistice si Instrumente pentru Prelucrarea Limbii Romane Workshop Proceedings, </i>pages 63-67, December. Workshop held in November 2006, Publisher: Ed.</p><p>Univ. Alexandru Ioan Cuza, ISBN: 978-973-703­208-9.</p><p>Camelia Ignat. 2009. <i>Improving Statistical Align­ment and Translation Using Highly Multilingual</i> <i>Corpora.</i><i> </i>Ph.D. thesis, INSA - LGeco- LICIA,</p><p>Strasbourg, France, June, 16th. It can be found on: http://sites.google.com/site/cameliaignat/home/phd-thesis - last accessed on 3.08.09.</p><p>Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In <i>NAACL '03: Proceedings of the 2003 Conference ofthe North American Chapter ofthe Association for Computational Linguistics on Human Language Technology, </i>pages 48-54, Morristown, NJ, USA. Association for Computational Linguistics.</p><p>Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In <i>Annual Meeting ofthe Association for Compu­tational Linguistics (ACL), demonstration session, </i>pages 177-180, Prague, Czech Republic, June.</p><p>Franz Josef Och and Hermann Ney. 2003. A sys­tematic comparison of various statistical alignment models. <i>Computational Linguistics, </i>29(1):19-51.</p><p>Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic eval­uation of machine translation. In <i>Proceedings ofthe 40th Annual Meeting on Association for Computa­tional Linguistics, Session: Machine translation and evaluation, </i>pages 311 - 318, Philadelphia, Penn­sylvania. Publisher: Association for Computational Linguistics Morristown, NJ, USA.</p><p>Maja Popovic and Hermann Ney. 2006. Statistical machine translation with a small amount of bilin­gual training data. In <i>LREC-2006: Fifth Interna­tional Conference on Language Resources and Eval­uation. 5th SALTMIL Workshop on Minority Lan­guages: Strategies for developing machine transla­tion for minority languages, </i>pages 25-29, Genoa, Italy, May.</p><p>Juho Rousu. 2008. Workpackage 3 advanced language models. Online, January. SMART Project.</p><p>Matthew Snover, Bonnie Dorr, Richard Schwartz, Lin-nea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In <i>Proceedings ofAssociation for Machine Transla­tion in the Americas, </i>pages 223-231, August.</p><p>Ralf Steinberger, Bruno Pouliquen, Anna Widiger, Camelia Ignat, Tomaz Erjavec, Dan Tufis, and Daniel Varga. 2006. The jrc-acquis: A multilin­gual aligned parallel corpus with 20+ languages. In <i>Proceedings ofthe 5th International Conference on Language Resources and Evaluation (LREC'2006), </i>pages 2142-2147, Genoa, Italy, May, 24-16.</p><p>Andreas Stolcke. 2002. Srilm - an extensible lan­guage modeling toolkit. In <i>Proc. Intl. Conf. Spoken Language Processing, </i>pages 901-904, Denver, Col­orado, September.</p><p>Bipin Suresh. 2010. Inclusion of large input corpora in statistical machine translation. Technical report, Stanford University.</p></references></body></article>