<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="514"/><title>Bilingual Cluster Based Models for Statistical Machine Translation</title><pubinfo>Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning,pp. 514-523, Prague, June 2007. ©2007 Association for Computational Linguistics</pubinfo><author surname="Yamamoto" givenname="Hirofumi"><org  name="National Institute of Information and Communications Technology" country="Japan" city="Kyoto"/></author><author surname="Sumita" givenname="Eiichiro"><org  name="TNO Information and Communication Technology" country="The Netherlands" city="Delft"/></author></firstpageheader><frontmatter><p><b>Bilingual Cluster Based Models f</b></p><p><b>Hirofumi Yamamoto</b></p><p>National Institute of Information and Communications Technology / 2-2-2 Hikaridai Seika-cho</p><p>Soraku-gun Kyoto Japan and ATR Spoken Language Communication Research Labs. hirofumi.yamamoto@nict.go.jp</p><p><b>Statistical Machine Translation</b></p><p><b>Eiichiro Sumita</b></p><p>Soraku-gun Kyoto Japan and ATR Spoken Language Communication Research Labs. eiichiro.sumita@nict.go.jp</p></frontmatter><abstract></abstract></header><body><section title="l Introduction"><p>We propose a domain specific model for statistical machine translation. It is well-known that domain speciic language mod­els perform well in automatic speech recog­nition. We show that domain speciic lan­guage and translation models also beneit statistical machine translation. However, there are two problems with using domain speciic models. The irst is the data sparse-ness problem. We employ an adaptation technique to overcome this problem. The second issue is domain prediction. In or­der to perform adaptation, the domain must be provided, however in many cases, the domain is not known or changes dynami­cally. For these cases, not only the trans­lation target sentence but also the domain must be predicted. This paper focuses on the domain prediction problem for statisti­cal machine translation. In the proposed method, a bilingual training corpus, is au­tomatically clustered into sub-corpora. Each sub-corpus is deemed to be a domain. The domain of a source sentence is predicted by using its similarity to the sub-corpora. The predicted domain (sub-corpus) speciic lan­guage and translation models are then used for the translation decoding. This approach gave an improvement of 2.7 in BLEU (Pa-pineni et al., 2002) score on the IWSLT05 Japanese to English evaluation corpus (im­proving the score from 52.4 to 55.1). This is a substantial gain and indicates the validity of the proposed bilingual cluster based mod­els.</p><p>Statistical models, such as n-gram models, are widely used in natural language processing, for ex­ample in speech recognition and statistical machine translation (SMT). The performance of a statistical model has been shown to improve when domain spe-ciic models are used, since similarity of statistical characteristics between model and target is higher. For utilize of domain speciic models, a training data sparseness and target domain estimation prob­lems must be resolved. In this paper, we try to es­timate target domain sentence by sentence, consid­ering cases where the domain changes dynamically. After sentence by sentence domain estimation, do­main speciic models are used for translation using the adaptation technique(Seymore et al., 1997).</p><p>In order to train a classiier to predict the domain, we used an unsupervised clustering technique on an unlabelled bilingual training corpus. We regarded each cluster (sub-corpus) as a domain. Prior to trans­lation, the domain of the source sentence is first pre­dicted and this prediction is then used for model se­lection. The most similar sub-corpus to the transla­tion source sentence is used to represent its domain. After the prediction is made, domain speciic lan­guage and translation models are used for the trans­lation.</p><p>In Section 2 we present the formal basis for our domain specific translation method. In Section 3 we provide a general overview of the two sub-tasks of domain speciic translation: domain prediction, and domain specific decoding. Section 4 presents the domain prediction task in depth. Section 5 offers a more detailed description of the details of domain specific decoding. Section 6 gives details of the ex­periments and presents the results. Finally, Section 7 offers a summary and some concluding remarks.<page local="2" global="515"/></p></section><section number="2" title="Domain Specific Models in SMT"><p>The purpose of statistical machine translation is to ind the most probable translation in the target lan­guage <i>e </i>of a given source language sentence <i>f</i>. This search process can be expressed formally by:</p><doubt alpha="60.0" length="15" tooSmall="False" monospace="0.0">argmaxP(e|f)(1)</doubt><p>In this formula, the target word sequence (sentence) <i>e</i><i> </i>is determined only by the source language word sequence <i>f</i>. However, <i>e</i><i> </i>is heavily dependent on not only on <i>f</i><i> </i>but also on the domain D. When the domain <i>D</i><i> </i>is given, formula (1) can be rewritten as the following formula with the introduction of a new probabilistic variable <i>D.</i></p><doubt alpha="62.5" length="16" tooSmall="False" monospace="0.0">argmaxP(ef,D)(2)</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">e</doubt><p>This formula can be re-expressed using Bayes' Law.</p><doubt alpha="56.5" length="23" tooSmall="False" monospace="0.0">argmaxP(e|D)P(f|e,D)(3)</doubt><p>Here, <i>P(f</i><i> </i>|e, <i>D) </i>represents the domain <i>D </i>specific translation model and <i>P(e|</i><i> D) </i>represents the domain <i>D</i><i> </i>speciic language model.</p><p>When the domain <i>D</i><i> </i>is known, domain speciic models can be created and used in the translation de­coding process. However, in many cases, domain <i>D</i><i> </i>is unknown or changes dynamically. In these cases, both the translation target language sentence <i>e</i><i> </i>and the domain <i>D</i><i> </i>must be dynamically predicted at the same time. The following equation represents the process of domain speciic translation when the do­main <i>D</i><i> </i>is being dynamically predicted.</p><p><i>argmax</i><i> </i><i>P</i><i> </i><i>(e,</i><i> </i><i>Df</i>)</p><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">e,D</doubt><doubt alpha="59.1" length="22" tooSmall="False" monospace="0.0">=argmaxP(Df)P(ef,D)(4)</doubt><p>The major difference between this equation and for­mula (3) is that the probabilistic variable <i>D</i><i> </i>is the prediction target in equation (4). In this equa­tion, <i>P(D|</i><i> </i><i>f)</i><i> </i>represents the domain prediction and <i>P(e|</i><i> </i><i>f,</i><i> D) </i>represents the domain speciic transla­tion.</p></section><section number="3" title="Outline of the Proposed Method"><p>Our method can be analysed into two processes: an off-line process and an on-line process. The pro­cesses are depicted in igure 1. In the off-line pro­cess, bilingual sub-corpora are created by clustering and these clusters represent domains. Domain spe-ciic models are then created from the data contained in the sub-corpora in a batch process. In the on-line process, the domain of the source sentence is irst predicted and following this the sentence is trans­lated using models built on data from the appropriate domain.</p><subsection number="3.1" title="Off-line process"><p>In this process, the training corpus is clustered to sub-corpora, which are regarded as domains. In SMT, a bilingual corpus is used to create the trans­lation model, and typically, bilingual data together with additional monolingual corpora are used to create the language model. In our method, both the bilingual and monolingual corpora are clustered. After clustering, cluster dependent (domain specific) language and translation models are created from the data in the clusters.</p><p>1. A bilingual corpus which is comprised of the training data for the translation model, or equivalently the bilingual part of the training data for the language model is clustered (see Section 4.2).</p><p>2. Each sentence of the additional monolingual corpora (if any) is assigned to a bilingual clus­ter (see Section 4.3).</p><p>3. For each cluster, the domain specific (cluster dependent) language models are created.</p><p>4. The domain speciic translation model is cre­ated using only the clusters formed from clus­tering bilingual data.</p></subsection><subsection number="3.2" title="On-line process"><p>This process is comprised of domain prediction and the domain speciic translation components. The following steps are taken for each source sentence.</p><p>1. Select the cluster to which the source sentence belongs.</p><page local="3" global="516"/><p>2. Translate the source sentence using the appro­priate domain speciic language and translation models.</p></subsection></section><section number="4" title="Domain Prediction"><p>This section details the domain prediction process. To satisfy equation (4), both the domain <i>D</i><i> </i>and the translation target word sequence <i>e,</i><i> </i>which max­imizes both <i>P(Df) </i>and <i>P(ef,D) </i>must be cal­culated at the same time. However, it is dificult to make the calculations without an approximation. Therefore, in the irst step, we ind the best candi­dates for <i>D</i><i> </i>given the input sentence <i>f.</i><i> </i>In the next step, <i>P(e|</i><i> </i><i>f,</i><i> D) </i>is maximized over the candidates for <i>D</i><i> </i>using the following formula.</p><p><i>argmax</i><i> </i><i>P</i><i>(ef, </i><i>argmax</i><i> </i><i>P</i><i>(Df)) </i>(5)</p><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">eD</doubt><p>Equation (5) is approximation offollowing equa­tion in that can <i>D</i><i> </i>is regarded as a hidden variable.</p><doubt alpha="55.2" length="29" tooSmall="False" monospace="0.0">argmaxVP(Df)P(e\D)P(f^,D))(6)</doubt><p>When the following assumptions are introduced to equation (6), equation (5) is obtained as an approxi­mation. For only one domain <i>Di,</i><i> </i><i>P</i><i> (D</i><b><i>i</i></b><i>f) </i>is nearly equal to one. For other domains, <i>P (Df) </i>are almost zero. <i>P(D|</i><i> </i><i>f)</i><i> </i>can be re-written as following equa­tion.</p><doubt alpha="50.0" length="6" tooSmall="False" monospace="0.0">P(D|f)</doubt><doubt alpha="29.4" length="17" tooSmall="False" monospace="0.0">=   P (D,f )/P(f)</doubt><doubt alpha="40.0" length="25" tooSmall="False" monospace="0.0">=   P(f,D)/P(D)xP(D)/P(f)</doubt><doubt alpha="33.3" length="21" tooSmall="False" monospace="0.0">= P(f| D)P(D)/P(f)(7)</doubt><p>Therefore, we can conirm reasonability of this as­sumption by calculating <i>P(f| D)P(D) </i>all domains <i>(P(f) </i>is constant).</p><subsection number="4.1" title="Domain Definition"><p>When the domain is known in advance, it is usu­ally expressible, for example it could be a topic that matches a human-defined category like "sport". On the other hand, when the domain is delimited in an unsupervised manner, it is used only as a probabilistic variable and does not need to be ex­pressed. Equation (4) illustrates that a good model will provide high probabilities to <i>P(D|</i><i> f)P(e| </i><i>f,</i><i> D)</i> for bilingual sentence pairs <i>(f,</i><i> </i><i>e).</i><i></i><i> </i>For the same reason, a good domain deinition will lead to a higher probability for the term: <i>P (Df </i><i>)P</i><i> (ef, D). </i>Therefore, we deine the domain <i>D </i>as that which maximizes <i>P(Df)P(e^) </i>(an approximation of <i>P(D|</i><i> f)P(e| </i><i>f,</i><i> </i><i>D)).</i><i> </i>This approximation ensures that the domain deinition is optimal for only the language model rather than both the language and translation models. <i>P(D|</i><i> f)P(e| D) </i>can be re­written as the following equation using Bayes' Law.</p><doubt alpha="50.0" length="12" tooSmall="False" monospace="0.0">P(D|f)P(e|D)</doubt><doubt alpha="34.6" length="26" tooSmall="False" monospace="0.0">=  P(eD)P(f ^)P(D)/P(f)(8)</doubt><p>Here, <i>P(f)</i><i> </i>is independent of domain <i>D.</i><i> </i>Further­more, we assume <i>P(D) </i>to be constant. The follow­ing formula embodies the search for the optimal do­main.</p><doubt alpha="56.5" length="23" tooSmall="False" monospace="0.0">argDmaxP(e|D)P(f| D)(9)</doubt><p>This formula ensures that the search for the domain maximizes the domain speciic probabilities of both <i>e </i>and <i>f </i>simultaneously.</p></subsection><subsection number="4.2" title="Clustering of the bilingual corpus"><p>As mentioned above, we maximize the domain spe­cific probabilities of <i>e </i>and <i>f </i>to ascertain the domain. We deine our domains as sub-corpora of the bilin­gual corpus, and these sub-corpora are formed by clustering bilingually by entropy reduction. For this clustering, the following extension of monolingual corpus clustering is employed (Carter 1994).</p><p>1. The total number of clusters (domains) is given by the user.</p><p>2. Each bilingual sentence pair is randomly as­signed to a cluster.</p><p>3. For each cluster, language models for <i>e </i>and <i>f </i>are created using the bilingual sentence pairs that belong to the cluster.</p><p>4. For each cluster, the entropy for <i>e </i>and <i>f </i>is cal­culated by applying the language models from the previous step to the sentences in the clus­ter. The total entropy is deined as the total sum of entropy (for both source and target) for each cluster.</p><page local="4" global="517"/><p><b>On-line process</b></p><p><b>Target language models</b></p><p><b>Source language models</b></p><p><b>Bilingual cluster</b></p><p><b>Target language</b></p><p><b>Source language</b> <b>Off-line process</b> <b>Bilingual corpus</b> <b>Monolingual corpus</b></p><figure caption="Figure 1: Outline of the Proposed Method"></figure><p>5. Each bilingual sentence pair is re-assigned to a cluster such that the assignment minimizes the total entropy.</p><p>6. The process is repeated from step (3) until the entropy reduction is smaller than a given threshold.</p></subsection><subsection number="4.3" title="Clustering the monolingual corpus"><p>Any additional monolingual corpora used to train the language model are also clustered. For this clus­tering, the following process is used.</p><p>1. First, bilingual clusters are created using the above process.</p><p>2. For each monolingual sentence its entropy is calculated using all the bilingual cluster depen­dent language models and also the general lan­guage model (see Figure 1 for a description of the general language model).</p><p>3. If the entropy of the general language model is the lowest, this sentence is not used in the cluster dependent language models.</p></subsection></section><section number="4." title="Otherwise, the monolingual sentence is added"><p>to the bilingual cluster that results in the lowest entropy.</p><subsection number="4.4" title="Domain prediction"><p>In the process described in the previous section we describe how clusters are created, and we deine our domains in terms of these clusters. In this step, do­main <i>D </i>is predicted using the given source sentence <i>f </i>. This prediction is equivalent to inding the <i>D </i>that maximizes <i>P(D|</i><i> f). </i><i>P(D|</i><i> </i><i>f)</i><i> </i>can be re-written as <i>P(f</i><i> </i><i>D)P</i><i>(D)/P</i><i>(f)</i><i> </i>using Bayes' law. Here, <i>P(f)</i><i> </i>is a constant, and if <i>P(D) </i>is assumed to be constant (this approximation is also used in the clustering of the bilingual corpus), maximizing the target is re­duced to the maximization of<i>P(f| </i><i>D).</i><i> </i>To maximize <i>P(f| D) </i>we simply select the cluster <i>D,</i><i> </i>that gives the highest likelihood ofa given source sentence <i>f.</i></p></subsection></section><section number="5" title="Domain specific decoding"><p>After domain prediction, domain speciic decoding to maximize <i>P(ef,D), </i>is conducted. <i>P(ef,D) </i>can be re-written as the following equation using Bayes' law.</p><doubt alpha="41.9" length="31" tooSmall="False" monospace="0.0">P (ef,D)=P(f^,D)P (e,D)/P (f,D)</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Decoding</b></p></td><td class="cell"><p></p></td><td class="cell"><p><b>Translation result</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>r</b></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="5" global="518"/><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">eD</doubt><doubt alpha="37.9" length="29" tooSmall="False" monospace="0.0">=P(f^,D)P(e\D)P(D)/P(f,D)(10)</doubt><p>Here, <i>f </i>is a given constant and <i>D </i>has already been selected by the domain prediction process. Therefore, maximizing <i>P(f| </i><i>e,</i><i> </i><i>D)P(e|</i><i> D) </i>is equiv­alent to maximizing the above equation. In <i>P(f</i><i> ^, </i><i>D)P(e\D),</i><i> </i><i>P</i><i>(f</i><i> ^, D) </i>is the domain specific translation model and <i>P(e|</i><i> D) </i>is the domain speciic language model. Equation (10) represents the whole process of translation of <i>f </i>into <i>e </i>using domain <i>D </i>speciic models <i>P(f </i><i>e,</i><i> D) </i>and <i>P(e</i><i> D).</i></p><subsection number="5.1" title="Differences from previous methods"><subsubsection number="5.1.1" title="Cluster language model"><p>Hasan et al. (2005) proposed a cluster language model for inding the domain <i>D.</i><i> </i>This method has three steps. In the irst step, the translation target language corpus is clustered using human-deined regular expressions. In the second step, a regular expression is created from the source sentence <i>f.</i><i> </i>In the last step, the cluster that corresponds to the ex­tracted regular expression is selected, and the cluster speciic language model built from the data in this cluster is used for the translation. The points of dif­ference are:</p><p>• In the cluster language model, clusters are de-ined by human-deined regular expressions. On the other hand, with the proposed method, clusters are automatically (without human knowledge) deined and created by the entropy reduction based method.</p><p>• In the cluster language model, only the trans­lation target language corpus is clustered. In the proposed method, both the translation source and target language corpora are clus­tered (bilingual clusters).</p><p>• In the cluster language model, only a domain (cluster) speciic language model is used. In the proposed method, both a domain speciic language model and a domain speciic transla­tion model are used.</p></subsubsection><subsubsection number="5.1.2" title="Sentence mixture language model"><p>In equation (6), <i>D </i>is regarded as a hidden vari­able. Furthermore, when <i>P(D</i><i> </i><i>f)</i><i> </i>is approximated as <i>P(D) = </i><i>D\,</i><i> </i>and the general translation model <i>P</i><i> </i><i>(f</i><i> ^) </i>is used instead of the domain specific trans­lation model <i>P(f</i><i> D), </i>this equation represents the process of translation using sentence mixture lan­guage models (Iyer et al., 1993) as follows:</p><doubt alpha="60.9" length="23" tooSmall="False" monospace="0.0">argmaxVDXP(eD)P(f^)(11)</doubt><p>The points that differ from the proposed method are as follows:</p><p>• In the sentence mixture model, the mixture weight parameters <i>D\</i><i> </i>are constant. On the other hand, in the proposed method, weight pa­rameters <i>P(D|</i><i> </i><i>f)</i><i> </i>are estimated separately for each sentence.</p><p>• In the sentence mixture model, the probabili­ties of all cluster dependent language models are summed. In the proposed model, only the cluster that gives the highest probability is con­sidered as approximation.</p><p>• In the proposed method, a domain speciic translation model is also used.</p></subsubsection></subsection></section><section number="6" title="Experiments"><subsection number="6.1" title="Japanese to English translation"><subsubsection number="6.1.1" title="Experimental corpus"><p>To evaluate the proposed model, we conducted experiments based on a travel conversation task cor­pus. The experimental corpus was the travel ar­rangements task of the BTEC corpus (Takezawa et al., 2002),(Kikui et al., 2003) and the language pair was Japanese and English. The training, develop­ment, and evaluation corpora are shown in Table 1. The development and evaluation corpora each had sixteen reference translations for each sentence. This training corpus was also used for the IWSLT06 Evaluation Campaign on Spoken Language Transla­tion (Paul 2006) J-E open track, and the evaluation corpus was used as the IWSLT05 evaluation set.</p></subsubsection><subsubsection number="6.1.2" title="Experimental conditions"><p>For bilingual corpus clustering, the sentence en­tropy must be calculated. Unigram language models were used for this calculation. The translation mod­els were pharse-based (Zen et al., 2002) created us­ing the GIZA++ toolkit (Och et al., 2003). The lan­guage models for the domain prediction and transla­tion decoding were word trigram with Good-Turing backoff (Katz 1987).<page local="6" global="519"/> Ten cluster specific source lan­guage models and a general language model were used for the domain prediction. If the general lan­guage model provided the lowest perplexity for an input sentence, the domain speciic models were not used for this sentence. The SRI language model­ing toolkit (Stolcke) was used for the creation of all language models. The PHARAOH phrase-based de­coder (Koehn 2004) was used for the translation de­coding.</p><table caption="Table 1: Japanese to English experimental corpus"></table><p>For tuning of the decoder's parameters, including the language model weight, minimum error train­ing (Och 2003) with respect to the BLEU score us­ing was conducted using the development corpus. These parameters were used for the baseline condi­tions. During translation decoding, the domain spe-ciic language model was used as an additional fea­ture in the log-linear combination according to the PHARAOH decoder's option. That is, the general and domain speciic language models are combined by log-linear rather than linear interpolation. The weight parameters for the general and domain spe-ciic language models were manually tuned using the development corpus. The sum of these language model weights was equal to the language model weight in the baseline. For the translation model, the general translation model (phrase table) and do­main speciic translation model were linearly com­bined. The interpolation parameter was again man­ually tuned using the development corpus.</p></subsubsection><subsubsection number="6.1.3" title="Experimental results"><p>In our bilingual clustering, the number of clus­ters must be ixed in advance. Based on the results of preliminary experiments to estimate model order, ten clusters were used. Ifless than ten clusters were used, domain speciic characteristics cannot be rep­resented. If more than ten clusters were used, data sparseness problems are severe, especially in trans­lation models. The amount of sentences in each cluster is not so different, therefore the approxima­tion that <i>P(D) </i>is reasonable. Two samples ofbilin-gual clusters are recorded in the appendix "Sample of Cluster". The cluster A.1 includes many interrog­ative sentences. The reason is that special words " at the end of Japanese sentence with no correspond­ing word used in English. The cluster A.2 includes numeric expressions in both English and Japanese.</p><doubt alpha="51.2" length="41" tooSmall="False" monospace="0.0">(desu ka)" or "       (masu ka)" are used</doubt><p>Next, we confirm the reasonability of the assump­tion used in equation(5). For this conirmation, we calculate <i>P(Df) </i>for all <i>D </i>for each <i>f </i>(P(D) is ap­proximated as constant). For almost <i>f, </i>only one domain <i>D</i><b><i>i </i></b>has a vary large value compared with other domains. Therefore, this approximation is conirmed to be reasonable.</p><p>In this experiments, we compare three ways ofde-ploying our domain speciic models to a baseline. In the irst method, only the domain speciic language model is used. The ratio of the weight parameter for the general model to the domain speciic model was 6:4 for all the domain specific language models. In the second method, only the domain speciic transla­tion model was used. The ratio of the interpolation parameter of the general model to the domain spe­cific model was 3:7 for all the domain specific mod­els. In the last method, both the domain speciic lan­guage and translation models (LM+TM) were used. The weights and interpolation parameters were the same as in the irst and second methods. The experi­mental results are shown in Table 2. Under all of the conditions and for all of the evaluation measures, the proposed domain speciic models gave better perfor­mance than the baseline. The highest performance came from the system that used both the domain spe-ciic language and translation models, resulting in a 2.<page local="7" global="520"/>7 point BLEU score gain over the baseline. It is a very respectable improvement. Appendix "Sample of Different Translation Results" recodes samples of different translation results with and without the do­main speciic language and translation models. In many cases, better word order is obtained in with the domain speciic models.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p># of sentence</p></td><td class="cell"><p>Total words</p></td><td class="cell"><p># of word entry</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Japanese Training</p></td><td class="cell"><p>40K</p></td><td class="cell"><p>355K</p></td><td class="cell"><p>12.5K</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>English Training</p></td><td class="cell"><p>40K</p></td><td class="cell"><p>315K</p></td><td class="cell"><p>9.2K</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Japanese Development</p></td><td class="cell"><p>510</p></td><td class="cell"><p>3,525</p></td><td class="cell"><p>918</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>English Development</p></td><td class="cell"><p>510x16</p></td><td class="cell"><p>57,388</p></td><td class="cell"><p>2,118</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Japanese Evaluation</p></td><td class="cell"><p>506</p></td><td class="cell"><p>3,647</p></td><td class="cell"><p>951</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></subsubsection></subsection><subsection number="6.2" title="Translation of ASR output"><p>In this experiment, the source sentence used as in­put to the machine translation system was the direct textual output from an automatic speech recognition (ASR) decoder that was a component of a speech-to-speech translation system. The input to our sys­tem therefore contained the kinds of recognition er­rors and disfluencies typically found in ASR output. This experiment serves to determine the robustness of the domain prediction to real-world speech input. The speech recognition process in this experiment had a word accuracy of 88.4% and a sentence ac­curacy of 67.2% . The results shown in Table 3 clearly demonstrate that the proposed method is able to improve the translation performance, even when speech recognition errors are present in the input sentence.</p></subsection><subsection number="6.3" title="Comparison with previous methods"><p>In this section we compare the proposed method to other comtemporary methods: the cluster lan­guage model (CLM) and the sentence mixture model (SMix). The experimental results for these meth­ods were reported by RWTH Aachen University in IWSLT06 (Mauser et al., 2006). We evaluated our method using the same training and evaluation cor­pora. These corpora were used as the training and development corpora in the IWSLT06 Chinese to English open track, the details are given in Table 4. The English side of the training corpus was the same as that used in the earlier Japanese to English experiments reported in this paper. Each sentence in the evaluation corpus had seven reference trans­lations. Our baseline performance was slightly dif­ferent from that reported in the RWTH experiments (21.9 BLEU socre for RWTH's system and 21.7 for our system). Therefore, their improved baseline is shown for comparison. The results are shown in Table 5. The improvements over the baseline of our method in both BLEU and NIST (Doddington 2002) score were greater than those for both CLM and SMix. In particular, our method showed im-provent in both the BLEU and NIST scores, this is in contrast to the CLM and SMix methods which both degraded the translation performance in terms ofthe NIST score.</p><table caption="Table 5: Comparison results with previous methods"></table></subsection><subsection number="6.4" title="Clustering of the monolingual corpus"><p>Finally, we evaluated the proposed method when an additional monolingual corpus was incorporated. For this experiment, we used the Chinese and En­glish bilingual corpora that were used in the NIST MT06 evaluation (NIST 2006). The size of the bilin­gual training corpus was 2.9M sentence pairs. For the language model training, an additional monolin­gual corpus of 1.5M English sentences was used. NIST 2006 development (evaluation set for NIST 2005) is used for evaluation. In this experiment, the test set language model perplexity of a model built on only the monolingual corpus was consider­ably lower than that of a model built from only the target language sentences from the bilingual corpus. Therefore, we would expect the use ofthis monolin­gual corpus to be an important factor affecting the quality of the translation system. These perplexi­ties were 299.9 for the model built on only the bilin­gual corpus, 200.1 for the model built on only the monolingual corpus, and 192.5 for the model built on a combination of the bilingual and monolingual corpora. For the domain speciic models, 50 clus­ters were created from the bilingual and monolin­gual corpora. In this experiment, only the domain speciic language model was used. The experimen­tal results are shown in Table 6. The results in the table show that the incorporation of the additional monolingual data has a pronounced beneicial effect on performance, the performance improved accord­ing to all of the evaluation measures.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>BLEU</p></td><td class="cell"><p>NIST</p></td><td class="cell"><p>WER</p></td><td class="cell"><p>PER</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>RWTH</p></td><td class="cell"><p>21.9</p></td><td class="cell"><p>6.31</p></td><td class="cell"><p>66.4</p></td><td class="cell"><p>50.8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Our</p></td><td class="cell"><p>21.7</p></td><td class="cell"><p>6.79</p></td><td class="cell"><p>70.9</p></td><td class="cell"><p>51.2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CLM</p></td><td class="cell"><p>+0.6</p></td><td class="cell"><p>-0.22</p></td><td class="cell"><p>-2.7</p></td><td class="cell"><p>-1.1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SMix</p></td><td class="cell"><p>+0.2</p></td><td class="cell"><p>-0.06</p></td><td class="cell"><p>-1.1</p></td><td class="cell"><p>-0.9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Proposed</p></td><td class="cell"><p>+1.1</p></td><td class="cell"><p>+0.17</p></td><td class="cell"><p>-1.1</p></td><td class="cell"><p>-0.5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="8" global="521"/><table caption="Table 2: Japanese to English translation evaluation scores"></table></subsection></section><section number="7" title="Conclusion"><p>We have proposed a technique that utilizes domain speciic models based on bilingual clustering for sta­tistical machine translation. It is well-known that domain speciic modeling can result in better perfor­mance. However, in many cases, the target domain is not known or can change dynamically. In such cases, domain determination and domain speciic translation must be performed simultaneously dur­ing the translation process. In the proposed method, a bilingual corpus was clustered using an entropy re­duction based method. The resulting bilingual clus­ters are regarded as domains. Domain speciic lan­guage and translation models are created from the data within each bilingual cluster. When a source sentence is to be translated, its domain is irst pre­dicted. The domain prediction method selects the cluster that assigns the lowest language model per­plexity to the given source sentence. Translation then proceeds using a language model and transla­tion model that are speciic to the domain predicted for the source sentence.</p><p>In our experiments we used a corpus from the travel domain (the subset of the BTEC corpus that was used in IWSLT06). Our experimental results clearly demonstrate the effectiveness of our method. In the Japanese to English translation experiments, the use of our proposed method improved the BLEU score by 2.7 points (from 52.4 to 55.1). We com­pared our approach to two previous methods, the cluster language model and sentence mixture model. In our experiments the proposed method yielded higher scores than either of the competitive meth­ods in terms of both BLEU and NIST. Moreover, our method may also be augmented when an additional monolingual corpus is avaliable for building the lan­guage model. Using this approach we were able to further improve translation performance on the data from the NIST MT06 evaluation task.</p><p><b>A  Sample of Cluster A.1 Cluster 1</b>• E: do you do alterations</p><p>J: (naoshi wa shi tei masu ka)• E: what's the newest color in this season</p><p>J: (kotoshi no shinshoku wa dore desu ka) yakyu no shiai wa ari masu ka)• E: are there any baseball games today</p><doubt alpha="62.5" length="8" tooSmall="False" monospace="0.0">J: (kyou</doubt><p>• E: where's the nearest perfumery</p><p>J: (moyori no kousui ten wa doko desu ka)• E: how much is the breakfast</p><p>J: (choshoku wa ikura desu ka) o o begai shi masu)<page local="9" global="522"/>• E: mr. aoki yes a single room for two nights</p><table caption="Table 3: Evaluation using ASR output" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>BLEU</p></td><td class="cell"><p>NIST</p></td><td class="cell"><p>WER</p></td><td class="cell"><p>PER</p></td><td class="cell"><p>Meteor</p></td><td class="cell"><p>TER</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Baseline</p></td><td class="cell"><p>52.38</p></td><td class="cell"><p>9.316</p></td><td class="cell"><p>42.87</p></td><td class="cell"><p>33.21</p></td><td class="cell"><p>70.63</p></td><td class="cell"><p>35.46</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Domain Specific LM</p></td><td class="cell"><p>53.66</p></td><td class="cell"><p>9.349</p></td><td class="cell"><p>41.73</p></td><td class="cell"><p>32,27</p></td><td class="cell"><p>71.39</p></td><td class="cell"><p>34.17</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Domain Specific TM</p></td><td class="cell"><p>54.30</p></td><td class="cell"><p>9.333</p></td><td class="cell"><p>41.64</p></td><td class="cell"><p>32.50</p></td><td class="cell"><p>71.77</p></td><td class="cell"><p>33.80</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Domain Specific LM+TM</p></td><td class="cell"><p>55.09</p></td><td class="cell"><p>9.451</p></td><td class="cell"><p>41.05</p></td><td class="cell"><p>31.63</p></td><td class="cell"><p>72.09</p></td><td class="cell"><p>33.20</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>BLEU</p></td><td class="cell"><p>NIST</p></td><td class="cell"><p>WER</p></td><td class="cell"><p>PER</p></td><td class="cell"><p>Meteor</p></td><td class="cell"><p>TER</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Baseline</p></td><td class="cell"><p>48.17</p></td><td class="cell"><p>8.892</p></td><td class="cell"><p>47.05</p></td><td class="cell"><p>36.86</p></td><td class="cell"><p>67.40</p></td><td class="cell"><p>39.36</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Domain Specific LM</p></td><td class="cell"><p>48.94</p></td><td class="cell"><p>8.900</p></td><td class="cell"><p>46.26</p></td><td class="cell"><p>36.37</p></td><td class="cell"><p>67.98</p></td><td class="cell"><p>38.42</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Domain Specific TM</p></td><td class="cell"><p>49.11</p></td><td class="cell"><p>8.842</p></td><td class="cell"><p>45.78</p></td><td class="cell"><p>36.55</p></td><td class="cell"><p>68.01</p></td><td class="cell"><p>37.88</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Domain Specific LM+TM</p></td><td class="cell"><p>50.12</p></td><td class="cell"><p>9.001</p></td><td class="cell"><p>45.26</p></td><td class="cell"><p>35.80</p></td><td class="cell"><p>68.05</p></td><td class="cell"><p>37.22</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 4: Training and evaluation corpora used for comparison with previous methods"></table><doubt alpha="53.3" length="15" tooSmall="False" monospace="0.0">A.2   Cluster 2</doubt><doubt alpha="61.6" length="86" tooSmall="False" monospace="0.0">J:T^^rT'-fX.-i/y^JUl-AH       "f~(aoki san desu ne ee shingu-rurumu de 2 haku desu ne)</doubt><p>• E: may i have the key to room two fifteen</p><doubt alpha="51.8" length="56" tooSmall="False" monospace="0.0">J: --51-5fm 0 m£T£V&gt;(2 1 5 gou shitsu no kagi o kudasai)</doubt><p>• E: i'd like extension twenty four please</p><doubt alpha="53.8" length="13" tooSmall="False" monospace="0.0">J: (naisen 24</doubt><p>• E: the flight number is se one o three to tokyo on the second of april</p><doubt alpha="50.0" length="2" tooSmall="False" monospace="0.0">J:</doubt><p>(furaitonanba</p><doubt alpha="65.5" length="55" tooSmall="False" monospace="0.0">wa tokyo iki s e 1 0 3 bin 4 gatsu futsuka no bin desu)</doubt><p>• E: delta airlines flight one one two boarding is delayed</p><p>S "f~ (derutakouku 112 bin wa tojo ga okure tei masu) <b>B   Sample of Different Translation Results</b></p><p>1. Ref: your room is number two ten Base: your room this is room two o one LM: your room is this is room two one zero TM: your room is room two o one LM+TM: your room is this is room two one zero</p><p>2. Ref: where is a spot where there are a lot offish Base: i'm spot where is the lot offish</p><p>LM: where is the spot are a lot offish TM: i'm spot where is the lot offish LM+TM: where is the spot are a lot offish</p><p>3. Ref: i don't like the design Base: design i don't like it LM: i don't like it design TM: i don't like the design LM+TM: i don't like the design</p><p>4. Ref: where can i contact you Base: where contact if i may LM: where contact if i can TM: where can i contact LM+TM: where can i contact</p></section><section number="5." title="Ref: where is a police station where japanese is understood"><p>Base: japanese where's the police station LM: japanese where's the police station TM: where's the police station where someone understands japanese</p><p>LM+TM: where's the police station where someone understands japanese<page local="10" global="523"/></p><table caption="Table 6: Experimental results with monolingual corpus" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p># of sentence</p></td><td class="cell"><p>Total words</p></td><td class="cell"><p>Vocabulary size</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>English Training</p></td><td class="cell"><p>40K</p></td><td class="cell"><p>315K</p></td><td class="cell"><p>9.2K</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Chinese Training</p></td><td class="cell"><p>40K</p></td><td class="cell"><p>304K</p></td><td class="cell"><p>18.7K</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Chinese Evaluation</p></td><td class="cell"><p>489</p></td><td class="cell"><p>5,110</p></td><td class="cell"><p>1.3K</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>BLEU</p></td><td class="cell"><p>NIST</p></td><td class="cell"><p>WER</p></td><td class="cell"><p>PER</p></td><td class="cell"><p>Meteor</p></td><td class="cell"><p>TER</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Baseline</p></td><td class="cell"><p>24.39</p></td><td class="cell"><p>7.918</p></td><td class="cell"><p>86.51</p></td><td class="cell"><p>61.65</p></td><td class="cell"><p>53.36</p></td><td class="cell"><p>68.21</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Proposed</p></td><td class="cell"><p>24.95</p></td><td class="cell"><p>8.030</p></td><td class="cell"><p>85.89</p></td><td class="cell"><p>61.27</p></td><td class="cell"><p>53.86</p></td><td class="cell"><p>67.48</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><references><p>K. Seymore, R. Rosenfeld, 'Using Story Topics for Lan­guage Model Adaptation," Proc. EUROSPEECH, pp. 1987-1990, 1997.</p><p>David Carter, "Improving Language Models by Cluster­ing Training Sentences,"Proc. ACL, pp. 59-64, 1994.</p><p>S. Hasan, H. Ney, "Clustered Language Models Based on Regular Expressions for SMT," Proc. EAMT, Bu­dapest, Hungary, May 2005.</p><p>R. M. Iyer and M. Ostendorf, "Modeling Long Distance Dependence in Language: Topic mixture versus dy­namic cache models," IEEE Transactions on Speech and Audio Processing, 1994.</p><p>T. Takezawa, E. Sumita, F. Sugaya, H. Yamamoto, S. Ya-mamoto, "Toward a broad-coverage bilingual corpus for speech translation of travel conversations in the real world," Proc. Conference on Language Resource and Evaluation, May 2002.</p><p>Genichiro Kikui, Eiichiro Sumita, Toshiyuki Takezawa, Seiichi Yamamoto, "Creating Corpora for Speech-to-Speech Translation," Proc. EUROSPEECH, pp. 381­384, 2003.</p><p>M. Paul, "Overview of the IWSLT 2006 Evaluation Cam­paign," IWSLT 2006, Nov. 2006.</p><p>R. Zens, F. J. Och, H. Ney, "Phrase-based statistical ma­chine translation," 25th German Conference on Artifi­cial Inteligence, Sep 2002.</p><p>F. J. Och, H. Ney, "A Systematic Comparison of Various Statistical Alignment Models," Computational Lin­guistics, No. 1, Vol. 29, pp. 19-51, 2003.</p><p>S. M. Katz, "Estimation of Probabilities from Sparse Data for Language Model Component of a Speech Recognizer," IEEE Trans. on Acoustics, Speech, and Signal Processing, pp. 400-401, 1987.</p><p>A. Stolcke, "SRILM - An Extensible Language Model Toolkit," http://www.speech.sri.com/projects/srilm/</p><p>P. Koehn, "PHARAOH: A beam search decoder for phrase-based statistical machine translation models," http://www.isi.edu/publications/licensed-sw/pharaoh/</p><p>F. J. Och, "Minimum error rate training for statistical ma­chine translation,"Proc. ACL, 2003.</p><doubt alpha="59.6" length="52" tooSmall="False" monospace="0.0">K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, "Bleu: a</doubt><p>method for automatic evaluation of machine transla-tion,"Proc. ACL, 2002.</p><p>A. Mauser, R. Zens, E. Matusov, S. Hasan, H. Ney, "The RWTH Statistical Machine Translation System for IWSLT 2006 Evaluation," IWSLT 2006, Nov. 2006.</p><p>G. Doddington, "Automatic evaluation of machine trans­lation quality using n-gram co-occurrence statistics," Proc. ARPA Workshop on Human Language Technol­ogy, 2002.</p><p>NIST    2006    Machine    Translation Evaluation, http://www.nist.gov/speech/tests/mt/ mt06eval_offi ciaL results.html</p></references></body></article>