<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="1207"/><title>Joint Alignment and Artificial Data Generation: An Empirical Study of Pivot-based Machine Transliteration</title><pubinfo>Proceedings of the 5th International Joint Conference on Natural Language Processing,pages 1207-1215, Chiang Mai, Thailand, November 8 - 13, 2011. ©2011 AFNLP</pubinfo><author surname="Zhang" givenname="Min"><org  name="Institute for Infocomm Research" country="Singapore"/></author><author surname="Duan" givenname="Xiangyu"><org  name="Tsinghua University" country="China" city="Beijing"/></author><author surname="Liu" givenname="Ming"><org  name="Tsinghua University" country="China" city="Beijing"/></author><author surname="Xia" givenname="Yunqing"><org  name="Tsinghua University" country="China" city="Beijing"/></author><author surname="Li" givenname="Haizhou"><org  name="Institute for Infocomm Research" country="Singapore"/></author></firstpageheader><frontmatter><p><b>Joint Alignment and Artificial Data Generation: An Empirical Study of Pivot-based Machine Transliteration</b></p><p><b>Min Zhang<footnote anchor="1"/>    Xiangyu Duan<footnote anchor="1"/>    Ming Liu<footnote anchor="1"/>    Yunqing Xia<footnote anchor="2"/>    Haizhou Li<footnote anchor="1"/></b></p><p>institute for Infocomm Research, A-STAR, Singapore</p><p>{mzhang,xduan,mliu,hli}@i2r.a-star.edu.sg</p><p><footnote anchor="2"/>Dept. of Comp. Sci. &amp; Tech., Tsinghua University, Beijing</p><p>yqxia@tsinghua.edu.cn</p></frontmatter><abstract>In this paper, we first carry out an investi­gation on two existing pivot strategies for statistical machine transliteration, namely <i>system-based </i>and <i>model-based </i>strategies, to figure out the reason why the previous model-based strategy performs much worse than the system-based one. We then propose a joint alignment algorithm to op­timize transliteration alignments jointly across source, pivot and target languages to improve the performance of the model­based strategy. In addition, we further propose a novel <i>synthetic data-based </i>strategy, which artificially generates source-target data using pivot language. Experimental results on benchmarking da­ta show that the proposed joint alignment optimization algorithm significantly im­proves the accuracy of model-based strat­egy and the proposed synthetic data-based strategy is very effective for pivot-based machine transliteration. </abstract></header><body><section number="1" title="Introduction"><p>Machine transliteration refers to the phonetic translation of names across languages by comput­er. With the rapid growth of the Internet data and the dramatic changes in the user demographics especially among the non-English speaking parts of the world, machine transliteration is important in many cross-lingual NLP, MT and CLIR appli­cations as their performances have been shown to positively correlate with the correct conversion of names between the languages in several studies (Demner-Fushman and Oard, 2002; Mandl and Womser-Hacker, 2005; Hermjakob <i>et al., </i>2008; Udupa <i>et al., </i>2009). However, the traditional source for name equivalence, the bilingual dictio­naries — whether handcrafted or statistical built — offer only limited support because new names always emerge.</p><p>All of the above points to the critical need for high-performance machine transliteration tech­nology. Much research effort has been made to address this issue in the research community (Knight and Graehl, 1998; Meng <i>et al., </i>2001; Al-Onaizan and Knight, 2002; Oh and Choi, 2002; Klementiev and Roth, 2006; Sproat, 2006; Zelen-ko and Aone, 2006; Li <i>et al., </i>2004, 2009a, 2009b; Sherif and Kondrak, 2007; Bertoldi <i>et al., </i>2008; Goldwasser and Roth, 2008). These previous work falls into three categories, i.e., grapheme-based, phoneme-based and hybrid methods (Li <i>et al. </i>, 2009a, 2009b). The report of the first machine transliteration shared task NEWS 2009 (Li <i>et al.</i>, 2009a, 2009b) provides common benchmarking data in diverse language pairs and systematically evaluate the state-of-the-art technologies using their provided data.</p><p>Although promising results have been reported, one of major issues is that the current translitera­tion methods rely heavily on significant amount of source-target parallel data to learn translitera­tion model. However, such corpora are not always available and the amounts of the currently availa­ble corpora, even for language pairs with English involved, are far from enough for training, letting alone many low-density language pairs. Indeed, transliteration corpora for most language pairs without English involved are unavailable and are usually rather expensive to manually construct (Khapra <i>et al., </i>2010; Zhang <i>et al., </i>2010). To date, only two previous works (Khapra <i>et al., </i>2010; Zhang <i>et al. </i>, 2010) touch this issue of transliterat­ing names across low-density language pairs. Both of them resort to pivot language-based ap­proaches to address this issue.</p><page local="2" global="1208"/><p>Khapra <i>et al. </i>(2010) proposes the system-based pivot strategy for machine transliteration, which learns a source-pivot model from source-pivot data and a pivot-target model from pivot-target data, respectively. In decoding, it first translite­rates a source name to 7V-best pivot names and then transliterates each pivot name to target names which are finally re-ranked using the com­bined two individual transliteration scores. Zhang <i>et al. </i>(2010) verifies the system-based strategy together with joint source-channel model (Li <i>et al., </i>2004) on Chinese, English, Korean and Japanese data (Li <i>et al., </i>2009a, 2009b) and they further proposes a model-based strategy, which learns a direct source-target transliteration model from two independent <footnote anchor="1"/> source-pivot and pivot-target name pair corpora, and does direct source-target decoding without relying on pivot languages. However, it was reported that the model-based strategy performed much worse than the system-based one (Zhang <i>et al., </i>2010).</p><p>This paper investigates the reason why pre­vious model-based strategy performs worse than system-based one and then proposes a joint alignment algorithm to solve the alignment unit inconsistent issue, which is the main reason of leading to the worse performance of model-based strategy. The key point of the proposed joint alignment algorithm is to jointly optimize transli­teration unit alignments among source, pivot and target languages. In addition, the paper further proposes a novel synthetic data-based strategy for pivot-based machine transliteration. It automati­cally constructs source-target data using source-pivot and pivot-target data, and then trains a direct source-target transliteration model using the syn­thetic data. We verify the proposed methods using the benchmarking data released at NEWS2009 (Li <i>et al., </i>2009a, 2009b). Experiential results show that our proposed joint alignment optimization algorithm is able to effectively solve the translite­ration unit mismatching issue and the proposed synthetic data-based strategy is very effective, achieving the best-reported performance.</p><p>The rest of the paper is organized as follows. Section 2 introduces the direct transliteration model. Section 3 discusses our proposed joint alignment algorithm and synthetic data-based strategy. Experimental results are reported at sec­tion 4. Finally, we conclude the paper in section 5.</p></section><section number="2" title="The Transliteration Model: JSCM"><p>To make our study language-independent, we select joint source-channel model (JSCM, also named as <i>n</i>-gram transliteration model) (Li <i>et al., </i>2004) under grapheme-based framework as our transliteration model due to its state-of-the-art performance and using orthographical information only (Li <i>et al., </i>2009a). In addition, unlike other feature-based methods, such as CRFs (Lafferty <i>et </i>al., 2001), MaxEnt (Berger <i>et al., </i>1996) or SVM (Vapnik, 1995), the JSCM model directly computes model probabilities using maximum likelihood estimation (Dempster <i>et al., </i>1977). This property facilitates the implementation of the model-based strategy.</p><p>JSCM directly models how both source and target names can be generated simultaneously. Given a source name <i>S </i>and a target name T, it estimates the joint probability of <i>S </i>and <i>T </i>as fol­lows:</p><doubt alpha="31.0" length="29" tooSmall="False" monospace="0.0">« n£=iP(&lt;s,t&gt;k\&lt;s,t&gt;k:\+1)(1)</doubt><p>where and is an aligned transliteration unit<footnote anchor="2"/>pair, and <i>n </i>is the <i>n</i>-gram order.</p><p>In our implementation, we compare different unsupervised transliteration alignment methods, including Giza++ (Och and Ney, 2003), JSCM-based EM algorithm (Li <i>et al., </i>2004), edit dis­tance-based EM algorithm (Pervouchine <i>et al., </i>2009) and Oh et al.'s alignment tool (Oh <i>et al., </i>2009). Based on the aligned transliteration corpus, we learn the transliteration model using maximum likelihood estimation (Dempster <i>et al., </i>1977) and decode    the    transliteration result using stack decoder (Schwartz and Chow, 1990).</p><footnote label="1">Here "independent" means the source-pivot and pivot-target data are not from the same English name source.</footnote><footnote label="2">Transliteration unit is language dependent. It can be a Chi­nese character, a sub-string of English words, a Korean Han-gual or a Japanese Kanji or several Japanese Katakanas.</footnote><page local="3" global="1209"/></section><section number="3" title="Joint Alignment and Synthetic Data-based Strategy"><p>In this section, we elaborate our proposed joint where, alignment algorithm and synthetic data-based strategy for pivot-based machine transliteration.</p><subsection number="3.1" title="System-based Strategy _"><p>Given a source name S, a target name <i>T </i>and let <i>Z </i>be the n-best transliterations of <i>S </i>in a pivot lan­guage Z <footnote anchor="3"/>, the system-based transliteration strate- _ gy under JSCM can be formulized as follows:</p><p>where and can be computed using</p><p>JSCM as formalized at Eq. (1). Eq. (2) assumes that is independent of when given because the parallel name corpus between <i>S </i>and <i>T </i>is not available under the pivot transliteration frame­work. The n-best transliterations in pivot language are expected to be able to carry enough informa­tion of the source name <i>S. </i>Following the nature of JSCM, Eq. (2) directly models how the source name <i>S </i>and pivot name and how the pivot name and the target name are generated simulta­neously. Since is considered twice in and , the duplicated impact of    is removed by being divided by .</p></subsection><subsection number="3.2" title="Joint Alignment Algorithm for Model­based Strategy"><p>Rather than combining the transitive translitera­tion results at system level, the model-based strat­egy aims to learn a direct model by combining the two individual models of and . Here we use bigram as an example to illustrate how to learn the JSCM trans­literation model <b><i>s, t&gt; </i></b><b><i>k—1</i></b><b><i> </i></b>using the model-based strategy.</p><footnote label="3">There can be multiple pivot languages used. However, same as Khapra et al. (2010) and Zhang et al. (2010), without loss of generality, we only use one pivot language to facilitate our discussion. It is straightforward to extend one pivot language to multiple ones by considering all the pivot transliterations in all pivot languages.</footnote><p>where , and can be directly estimated at training corpus.</p><p>In summary, eq. (2) formalizes the system-based strategy while eq. (3) formalizes the model­based strategy, where we can find that eq. (2) in­volves the pivot language in modeling and de­coding while eq. (3) does not (its model parameters are pre-computed using eq. (4) and (5) during training).</p><p>In previous work (Zhang <i>et al., </i>2010), the model-based strategy was reported to perform much worse than the system-based. We find that the main reason is due to the size inconsistence of transliteration unit of pivot language in the source-pivot and pivot-target alignments during training. As shown at eq. (4), the source-target model is calculated using the source-pivot el and the pivot-target model</p><p>directly. This requests that the pivot transliteration unit must be consistent in the two individual modes. Thus, all the source-pivot and pivot-target model parameters are of no use if their in­volved pivot unit can only be found at either source-pivot or pivot-target model. Unfor­tunately in the only previous work (Zhang <i>et al., </i>2010), the source-pivot model and pivot-target model are trained separately, i.e., their object function is to maximize <i>P(S,</i><i> </i>Z)and <i>P(T,Z)</i><i> </i>inde­pendently.<page local="4" global="1210"/> This results in serious pivot translitera­tion unit inconsistent issue for some language pairs. For example, in our experiment (Chi­nese English Japanese) with English as pivot language, we find that the English transliteration unit size in Chinese English model is much larger than that in English Japanese model. This is be­cause from phonetic viewpoint, in Chinese Eng­lish model, the English unit is at syllable level (corresponding one Chinese character) while in English Japanese model, the English unit is at sub-syllable level (consonant or vowel or syllable, corresponding one Japanese Katakana). Following example excerpted from our training corpus illu­strates the pivot transliteration unit mismatching issue, where the English word "Aachen" is seg­mented into "Aa" and "chen" in Chinese-English model while it is segmented into "A", "a", "che" and "n" in English-Japanese model. This trilingual pair is then of no use in model-based strategy.</p><p>To solve the mismatching issue, this paper pro­poses a joint alignment algorithm to jointly optim­ize transliteration unit alignments among source, pivot and target languages for model-based strate­gy. To facilitate discussion, we base on the task of using English as pivot language for Chinese-Japanese transliteration (see Table 1) to present our proposed algorithm. The core idea of this al­gorithm is to use Chinese-English alignments as a constraint to do English-Japanese alignment. The algorithm consists of the following 6 steps:</p><p><b>Algorithm 1. Joint Alignment</b></p><doubt alpha="85.7" length="7" tooSmall="False" monospace="0.0">Inputs:</doubt><p>Chinese-English Name List (CE). English-Japanese Name List (EJ).</p><p><b>Outputs:</b></p><p>More consistent CE and EJ alignments at Chi­nese syllable level and a direct Chinese-Japanese (CJ) JSCM.</p><p>1. Align the CE names at Chinese syllable level using the JSCM-based EM algorithm (Li <i>et al., </i>2004).</p><p>2. Train a transliteration unit-based English bi­gram LM with the transliteration unit-segmented (at step 1) English side names of CE using SRILM toolkits (Stolcke, 2002). Note that here the English transliteration units are corresponding to Chinese syllable level.</p><p>3. Align the Chinese-English-Japanese (CEJ) names that are the intersection of the entire CE and EJ names (with the same English names).</p><p>a. CE part of CEJ has been aligned at Step 1.</p><p>b. Align the CJ part of CEJ at Chinese sylla­ble level using the JSCM-based EM algo­rithm (Li <i>et al., </i>2004).</p><p>c. Construct the CEJ name alignments by <b>Chinese:</b></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">_/\</doubt><p><b>English: </b>A a c h e n merging the CE and CJ alignments.</p><doubt alpha="56.2" length="16" tooSmall="False" monospace="0.0">Japanese:7—   ^y</doubt><p>4. Align EJ names at Chinese syllable level.</p><p>a. The intersection part of EJ with CJ (iEJ) has already been aligned at Chinese sylla­ble level at step 3.</p><p>b. Align the remaining non-intersection part of EJ name pairs <i>(niEJ) </i>using Algorithm 2 with the help of the aligned intersection part done at step 4.a and the transliteration unit-based English bi-gram LM learned at step 2.</p><p>c. Merge the above two parts.</p><p>5. Train two individual JSCMs using the Chinese syllable level-aligned CE and EJ name corpus, respectively.</p><p>6. Train a direct CJ JSCM using the two individ­ual JSCMs learned at step 5 by the model­based pivot strategy as formulated at eqs. (3), (4) and (5).</p><p><b>Algorithm 2. Constrained EM-based Align­ment</b></p><p>1. Non-intersection part of EJ name pairs (niEJ).</p><p>2. Intersection part of EJ name pairs (/EJ) aligned at Chinese syllable level and the in­itial JSCM (named as iJSCM) learned from this corpus (step 3.d of Algorithm 1).</p><p>3. The transliteration unit-based English bi­gram LM (named as eLM, step 2 of Algo­rithm 1).</p><doubt alpha="85.7" length="7" tooSmall="False" monospace="0.0">Output:</doubt><p>English-Japanese name pairs aligned at Chi­nese syllable level.</p><p>1. Bootstrap initial alignment of the <i>niEJ </i>name using the initial model iJSCM.</p><page local="5" global="1211"/><p>Chinese-English Name List (CE). English-Japanese Name List (EJ).</p><p><b>Outputs:</b></p><p>2. <b>Expectation: </b>re-train iJSCM using both the input <i>i</i>EJ name alignments and the updated <i>ni</i>EJ name alignments.</p><p>3. <b>Maximization: </b>Apply the re-trained iJSCM and the input eLM to obtain new alignments of the <i>ni</i>EJ names. Note that different from pre­vious EM-based transliteration alignment algo­rithm (Li <i>et al. </i>, 2004) that only maximizes the JSCM probabilities, here we maximize two kinds of probabilities:</p><p>where refers to English and refers to Japa­nese; A is an alignment which defining the transliteration unit segmentations of and , and their mappings; is the JSCM probability of and under ; and is the eLM transliteration unit bigram probability of    segmented by .</p><p>4. Go to step 2 until the alignment converges.</p><p>5. Output the niEJ name alignments.</p><p>The motivation of the joint alignment algorithm (Algorithm 1 and 2) is to address the English transliteration unit mismatching issue by aligning the EJ at Chinese syllable level with the help of CE alignment. While the mismatching issue in the intersection part of the data is easy to solve by step 3 of Algorithm 1, it is more complicated at the non-intersection part. As illustrated at step 3 of Algorithm 2, the core idea is to use English segmentation learned from CE alignment (step 2 of Algorithm 1) and already-aligned intersection part of EJ <i>(i</i>EJ, step 3 of Algorithm 1) to constrain the EM alignment process. Therefore, the English bigram LM and the aligned intersection part <i>(i</i>EJ) keep unchanged during all the EM iterations. But in E step (step 2 of Algorithm 2), the iJSCM model is updated at each iteration using the entire EJ data while in M step (step 3 of Algorithm 2), the alignments are decoded out using both the iJSCM and the English bi-gram LM. Indeed, in our implementation, we introduce more know­ledge sources, including transliteration unit inser­tion penalty and Japanese LM, into the M step by simply considering these two features at eq. (6).</p><p>Given the jointly optimized CE and EJ aligned name corpus, we can easily learn a direct CJ mod­el using the pivot-based strategy (steps 5 and 6 of Algorithm 1).</p></subsection><subsection number="3.3" title="Synthetic Data-based Strategy"><p>Different from previous two strategies, the syn­thetic data-based strategy automatically constructs source-target data using source-pivot and pivottarget data, and then trains a direct source-target transliteration model using the synthetic and any other available source-target data. The philosophy of this strategy is straightforward while the key is how to generate "good" data. Next, we also use Chinese-English-Japanese as example to elaborate this strategy.</p><p><b>Algorithm 3. Artificial Data Generation Inputs:</b></p><p>Synthetic Chinese-Japanese name pairs.</p><p>1. Directly output those CJ names <i>(i</i>CJ), which are the intersection of the entire CE and EJ names (with the same English names).</p><p>2. Transliterate those Chinese names which are not in <i>i</i>CJ to Japanese using either the system-based or model-based strategy. To maintain the transliteration quality, we consider both forward and backward transliteration probabilities as well as the information whether the original Chinese can be recovered from a transliterated Japanese name. The process is formalized as follows:</p><p>where is the forward transliteration probability, is the backward transliteration probability, and is a penalty function to penalize those cases where    is not equal to C, i.e., C fails to be covered from J. <i>P</i>(J) is a Japanese Katakana language model. 3. Translate those Japanese names which are not in <i>i</i>CJ to Chinese in the similar way as step 2.</p><p>Note that the outputs of step 2 and 3 do not overlap with each other. is a Chinese character-based language model. 4. Merge the results of step 1, 2 and 3. Given the merged data, we can easily train a direct Chi­nese-Japanese transliteration model.</p><p>The core idea of Algorithm 3 lies in eqs. (7) and (8). Among the three strategies (system-based, model-based and synthetic data-based), the first and the third ones are transliteration model independent while the second one is not.</p><page local="6" global="1212"/></subsection><subsection number="3.4" title="Comparison with Previous Work"><p>Almost all previous work on machine translitera­tion focuses on direct transliteration or translitera­tion system combination. Only two recent work (Khapra <i>et al., </i>2010; Zhang <i>et al., </i>2010) touches on the issue of pivot transliteration. Khapra <i>et al. </i>(2010) proposes the system-based strategy and does extensively empirical study together with CRF model on Indic/Slavic/Semetic languages and English. Zhang <i>et al. </i>(2010) proposes the model-based strategy, but reporting very bad per­formance. To address the low performance issue of the model-based strategy, this paper proposes the joint alignment algorithm to optimize the source-pivot-target alignment directly, resulting in significant performance improvement. Moreover, the paper proposes a new synthetic data-based strategy for pivot-based machine transliteration.</p><p>Machine translation carries out similar pivot-based translation studies. Bertoldi <i>et al. </i>(2008) studies two pivot approaches for phrase-based statistical machine translation. One is at system level and one is to re-construct source-target data and alignments through pivot data. Cohn and La-pata (2007) explores how to utilize multilingual parallel data (rather than pivot data) to improve translation performance. Wu and Wang (2007, 2009) study the model-level pivot approach and explore how to leverage on rule-based translation results in pivot language to improve translation performance. Utiyama and Isahara (2007) com­pare different pivot approaches for phrase-based statistical machine translation. All of the previous work on machine translation works on phrase-based statistical machine translation. Therefore, their translation model is to calculate phrase-based conditional probabilities at unigram level ( ) while our transliteration model is to calculate joint transliteration unit-based condi­tional  probabilities  at  bigram  level ( ). This is the fundamental difference.</p><p>We use the NEWS 2009 Chinese-English and</p><p>English-Japanese benchmark data as our experi­mental data (Li <i>et al., </i>2009a). All of the names originate from Western names, i.e., no native Chinese and Japanese names are involved in this experiment. Considering the fact that those lan­guage pairs with English involved have the most training data, it is reasonable to select English as pivot language. Table 1 reports the statistics of all the experimental data. The Chinese-English-Japanese data is the intersection of the Chinese-English and English-Japanese data.</p><p>We compare different alignment algorithms on the DEV set (Li <i>et al., </i>2009a). Finally we use Pervouchine <i>et al. </i>(2009)'s alignment algorithm for Chinese-Japanese and Li <i>et al. </i>(2004)'s for Chinese-English and English-Japanese. Given the aligned corpora, we directly learn each individual JSCM model (i.e., <i>n</i>-gram transliteration model) using SRILM toolkits (Stolcke, 2002). We also use SRILM toolkits to do decoding. For the sys­tem-based strategy, we output top-10 pivot trans­literation results. For the evaluation matrix, to save space, we only use top-1 accuracy (ACC) (Li <i>et al., </i>2009a) to measure transliteration perfor­mance since other five evaluation matrix used at Li <i>et al. </i>(2009a) are reported to have great corre­lation with ACC.</p></subsection><subsection number="4.2" title="Experimental Results"><subsubsection number="4.2.1" title="Results of Direct Transliteration"><p>Table 2 reports the performance of direct transli­teration. The first two experiments (line 1-2) are part of the NEWS 2009 share tasks and the others are our additional experiments for our pivot stu­dies. Comparison of the first two experimental results and the results reported at NEWS 2009 shows that we achieve comparable performance with their best-reported systems under the same conditions of using single system and orthograph­ic features only. This indicates that our baseline represents the state-of-the-art performance. In <b>Language Pair</b> <b>Methods Chinese-English-Japanese Japanese-English-Chinese</b><page local="7" global="1213"/></p><table caption="Table 2. Performance of direct transliterations" class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Language Pair</b></p></td><td class="cell"><p><b>ACC</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>English-Chinese</p></td><td class="cell"><p>0.681049</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>English-Japanese</p></td><td class="cell"><p>0.456755</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Chinese-English</p></td><td class="cell"><p>0.394490</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Japanese-English</p></td><td class="cell"><p>0.314970</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Chinese-Japanese</p></td><td class="cell"><p>0.288022</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Japanese-Chinese</p></td><td class="cell"><p>0.366559</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 1. Statistics on the data set" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>4    Experimental Results</b></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>4.1   Experimental Settings</b></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Language Pair</b></p></td><td class="cell"><p><b>Training Test</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Chinese-English (CE)</p></td><td class="cell"><p>31,961 2,896</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>English-Japanese (EJ)</p></td><td class="cell"><p>23,225 1,489</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Chinese-English-Japanese</p><p>(CEJ, the intersection part of CE and EJ)</p></td><td class="cell"><p>10,071 1,030</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">ACC</doubt><doubt alpha="0.0" length="8" tooSmall="False" monospace="0.0">0.314970</doubt><doubt alpha="0.0" length="8" tooSmall="False" monospace="0.0">0.288022</doubt><p>Table 4. Performance of model-based strategy (in ACC/# of unigram/# of bigram of the different transliteration models learned y the model-based strategy) addition, we find that the backward transliteration (line 3-4) consistently performs worse than its corresponding forward transliteration (line 1-2). This observation is consistent with what reported at previous work (Li <i>et al., </i>2004; Zhang <i>et al., </i>2004). The main reason is because English has much more transliteration units than foreign C/J languages. This makes the transliteration from English to C/J a many-to-few mapping issue and backward transliteration a few-to-many mapping issue. Therefore backward transliteration has more ambiguities and thus is more difficult. Moreover, due to the less available training data for the language pairs without English involved (Chinese/Japanese), the lowest two experiments (line 5-6) performs worse than the case with Eng­lish involved . This observation motivates us the study using pivot language for machine translite­ration.</p></subsubsection><subsubsection number="4.2.2" title="Results of System-based Strategy"><p>Table 3 reports the experimental results of sys­tem-based strategy. It confirms the previous ob­servations (Khapra <i>et al., </i>2010; Zhang <i>et al.,</i> 2010).</p><p>• The system-based pivot strategy is very ef­fective, achieving significant performance improvement over direct transliteration.</p><p>• Different from other pipeline methodologies, system-based pivot strategy does not suffer from the error propagation issue. Its ACC is significantly better than the product of the ACCs of the two individual systems.</p><p>The main reasons of the good performance re­ported at the above observations are due to the following reasons:</p><p>• The pivot approach is able to use large amount of source-pivot and pivot-target data.</p><p>• The nature of transliteration is a phonetic translation process. Therefore a little bit var­iation in orthography may not hurt or even help to improve transliteration performance in some cases as long as the orthographical variations keep the phonetic equivalent in­formation.</p><p>The N-best accuracy of machine translitera­tion is very high (Li <i>et al., </i>2004; Zhang <i>et al. </i>, 2004). It means that in most cases the correct transliteration in pivot language can be found in the top-10 results and the other 9 results hold the similar pronunciations with the correct one, which can serve as al­ternative   "quasi-correct"   inputs  to the</p><p>Chinese-English-Japanese <b>(System) </b>Chinese-English <b>(Direct) </b>English-Japanese <b>(Direct) </b>Chinese-Japanese <b>(Direct)</b></p><p>Japanese-English-Chinese (System) Japanese-English <b>(Direct) </b>English-Chinese <b>(Direct) </b>Japanese-Chinese <b>(Direct)</b> second stage transliterations and thus large­ly improve the overall accuracy.<page local="8" global="1214"/></p><doubt alpha="0.0" length="26" tooSmall="False" monospace="0.0">0.324361 0.394490 0.456755</doubt><doubt alpha="0.0" length="10" tooSmall="False" monospace="0.0">0".445748"</doubt><doubt alpha="0.0" length="17" tooSmall="False" monospace="0.0">0.681049 0.366559</doubt><table caption="Table 3. Performance of system-based strategy"></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Baseline 1: </b>Independent alignment of Chinese-English and English-Japanese</p><p>(Zhang <i>et al., </i>2010)</p></td><td class="cell"><p>0.065949 (5816/16989)</p></td><td class="cell"><p>0.043011(5816/16989)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Baseline 2: </b>Linguistically heuristic-based re-construction of Chinese-English and English-Japanese align­ment (Zhang <i>et al., </i>2010)</p></td><td class="cell"><p>0.282638 (26351/34812)</p></td><td class="cell"><p>0.378299 (26351/34812)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Method 1: </b>Joint alignment on intersec­tion part of Chinese-English and Eng­lish-Japanese data (Ours)</p></td><td class="cell"><p>0.287360 (26432/34920)</p></td><td class="cell"><p>0.378796 (26432/34920)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Method 2: </b>Joint alignment on entire data set (Ours)</p></td><td class="cell"><p><b>0.325367 (37437/48590)</b></p></td><td class="cell"><p><b>0.440782 (37437/48590)</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></subsubsection><subsubsection number="4.2.3" title="Results of Model-based Strategy"><p>Table 4 reports the performance of model-based strategy with different alignment refinements, where we can find:</p><p>• Baseline 1 clearly shows that the model­based strategy performs extremely worse than the other three settings if we align the Chinese-English and English-Japanese data independently. The major reason attributes to the size mismatching of the English trans­literation units between the two data sets (syllable level <i>vs. </i>phoneme or syllable level).</p><p>• The other three experiments demonstrate the effectiveness of the alignment refinements. However Baseline 2 is more heuristic while ours are more mathematically principled.</p><p>• Method 1 performs comparably with Base­line 2 even utilizing fewer training data.</p><p>• Method 2 achieves the best performance. It convincingly shows the effectiveness of the proposed joint optimization algorithm.</p><p>• Among all the models, Method 2 has the largest amounts of model parameters (# of unigram and bigram). From modeling view­point, it indicates that this model is more powerful than others. This is due to the con­tribution of the more consistent English transliteration units.</p><p>• Comparing Tables 4 and Table 3, we can see that the model-based strategy performs as well as the system-based strategy. This clearly demonstrates the effectiveness of our proposed joint alignment algorithm.</p></subsubsection><subsubsection number="4.2.4" title="Results of Synthetic Data-based Strategy"><table caption="Table 5 shows the advantage of synthetic data-based strategy over the other methods."></table><p>• The synthetic data-based strategy signifi­cantly outperforms the direct one. This clearly shows the effectiveness of the addi­tional synthetic data.</p><p>• Using the same amount of data, the synthet­ic data-based strategy significantly outper­forms the model-based one. This is not surprising since model-based strategy suf­fers from the transliteration unit mismatch­ing issue and its performance is also compromised by the independent assump­tion of eq. (4) while the synthetic data-based directly learns the model from bilingual data without suffering from the above two issues.</p><p>• Using the same amount of data, the synthet­ic data-based strategy significantly outper­forms the system-based one. This is because the synthetic data-based strategy directly learns a source-target transliteration model while system-based method utilizes two in­direct models and has to bear with the inde­pendent assumption of eq. (2).</p><p>• It is worth noting the transliteration perfor­mance of the synthetic data-based strategy highly depends on the quality of the artifi­cially generated data. Table 5 reports the performance using the default setting of eq. (7) and (8) at Algorithm 3. We expect that the synthetic data-based strategy has the po­tential to further improve its performance by simply introducing more features into eq. (7)</p><p>and (8).</p></subsubsection></subsection></section><section number="5" title="Conclusions"><p>A big challenge to statistical-based machine trans­literation is the lack of the training data, esp. to those low-density language pairs without English involved. To address the above issue, this paper propose a simple, but very effective solution, namely synthetic data-based strategy, to artificial­ly generate direct source-target training data using pivot language. Experimental results on NEWS 2009 data shows that the proposed strategy is very useful, achieving the best-reported performance. The paper also proposes a joint alignment algo­rithm to jointly optimize the alignments between source, pivot and target data. Experimental results show that the joint alignment algorithm is able to largely boost the performance of model-based strategy.</p><table caption="Table 5. Performance comparison of the three pivot strategies" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Language Pair</b></p></td><td class="cell"><p><b>ACC</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Japanese-Chinese <b>(Synthetic Data) </b>Japanese-Chinese <b>(System) </b>Japanese-Chinese <b>(Model) </b>Japanese-Chinese <b>(Direct)</b></p></td><td class="cell"><p>0.465648 0.445748 0.440782 0.366559</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Chinese-Japanese <b>(Synthetic Data) </b>Chinese-Japanese <b>(System) </b>Chinese-Japanese <b>(Model) </b>Chinese-Japanese <b>(Direct)</b></p></td><td class="cell"><p>0.338930 0.324361 0.325367 0.288022</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="9" global="1215"/><p>The system-based and the proposed synthetic-based strategy are transliteration model-independent while model-based strategy is not. However, the three strategies and the proposed joint alignment algorithm are not limited to the machine transliteration task. They can be applied to those tasks which possess the similar "transi­tive" property as machine transliteration, such as paraphrasing, domain adaptation and some multi­lingual tasks.</p></section><references><p>Yaser Al-Onaizan and Kevin Knight. 2002. <i>Translat­ing named entities using monolingual and bilingual resources. </i>ACL-02</p><p>Adam L. Berger, stephen A. Della Pietra and Vincent J. Della Pietra. 1996. <i>A Maximum Entropy Approach to Natural Language Processing. </i>Computational Linguistics. 22(1):39-71</p><p>N. Bertoldi, M. Barbaiant, M. Federico and R. Cattoni. 2008. <i>Phrase-based Statistical Machine Translation with Pivot Languages. </i>IWSLT-08</p><p>Trevor Cohn and Mirella Lapata. 2007. <i>Machine Translation by Triangulation: Making Effective Use of Multi-Parallel Corpora. </i>ACL-07</p><p>Andrew Finch and Eiichiro Sumita. 2008. <i>Phrase-based machine transliteration. </i>IJCNLP-08</p><p>Dan Goldwasser and Dan Roth. 2008. <i>Transliteration as constrained optimization. </i>EMNLP-08</p><p>D. Demner-Fushman and D. W. Oard. 2002. <i>The effect of bilingual term list size on dictionary-based cross-language information retrieval. </i>The 36th Hawaii Int'l. Conf. System Sciences</p><p>A.P. Dempster, N.M. Laird and D.B.Rubin, 1977.</p><p><i>Maximum likelihood from incomplete data via the EM algorithm, </i>J. Roy. Stat. Soc., Ser. B. Vol. 39</p><p>Ulf Hermjakob, Kevin Knight and Hal Daum'e. 2008. <i>Name translation in statistical machine translation: Learning when to transliterate. </i>ACL-08</p><p>John Lafferty, F. Pereira, Andrew McCallum. 2001. <i>Conditional random fields: Probabilistic models for segmenting and labeling sequence data. </i>ICML-01</p><p>Mitesh Khapra, Kumaran A and Pushpak Bhattacha-ryya. 2010. <i>Everybody loves a rich cousin: An em­pirical study of transliteration through bridge languages. </i>NAACL-HLT-10</p><p>A. Klementiev and Dan Roth. 2006. <i>Weakly supervised named entity transliteration and discovery from multilingual comparable corpora. </i>COLING-ACL-</p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">06</doubt><p>K. Knight and J. Graehl. 1998. <i>Machine Translitera­tion, </i>Computational Linguistics, Vol 24, No. 4</p><p>P. Koehn, F. J. Och and D. Marcu. 2003. <i>Statistical phrase-based translation. </i>HLT-NAACL-03</p><p>J. Lafferty, A. McCallum and F. Pereira. 2001. <i>Condi­tional random fields: Probabilistic models for seg­menting and labeling sequence data. </i>ICML-01</p><p>Haizhou Li, A Kumaran, Vladimir Pervouchine and Min Zhang. 2009a. <i>Report of NEWS 2009 Machine</i> <i>Transliteration   Shared   Task.</i><i> </i>IJCNLP-ACL-09</p><p>Workshop: NEWS-09</p><p>Haizhou Li, A Kumaran, Min Zhang and Vladimir <i>Machine Transliteration Shared Task.</i><i> </i>IJCNLP-ACL-09 Workshop: NEWS-09 <i>Channel Model for Machine Transliteration.</i><i> </i>ACL-04</p><doubt alpha="66.7" length="42" tooSmall="False" monospace="0.0">Pervouchine. 2009b.Whitepaper of NEWS 2009</doubt><doubt alpha="57.4" length="47" tooSmall="False" monospace="0.0">H. Li, M. Zhang and J. Su. 2004.A Joint Source-</doubt><p>Thomas Mandl and Christa Womser-Hacker. 2004. <i>How do Named Entities Contribute to Retrieval Ef­fectiveness? </i>CLEF-04</p><p>H. Meng, W. Lo, B. Chen and K. Tang. 2001. <i>Gener­ate Phonetic Cognates to Handle Name Entities in English-Chinese cross-language spoken document</i> <i>retrieval. </i>ASRU-01</p><p>Jong-Hoon Oh and Key-Sun Choi. 2002. <i>An English-Korean Transliteration Model Using Pronunciation and Contextual Rules. </i>COLING-02</p><p>Jong-Hoon Oh, Kiyotaka Uchimoto, and Kentaro Tori-sawa. 2009. <i>Machine Transliteration with Target-Language Grapheme and Phoneme: Multi-Engine Transliteration Approach. </i>NEWS 2009</p><p>Franz Josef Och and Hermann Ney. 2003. <i>A Systematic Comparison of Various Statistical Alignment Mod­els. </i>Computational Linguistics 29(1)</p><p>V. Pervouchine, H. Li and B. Lin. 2009. <i>Translitera­tion Alignment. </i>ACL-IJCNLP-09</p><p>R. Schwartz and Y. L. Chow. 1990. <i>The N-best algo­rithm: An efficient and exact procedure for finding the N most likely sentence hypothesis, </i>ICASSP-90</p><p>Tarek Sherif and Grzegorz Kondrak. 2007. <i>Substring-based transliteration. </i>ACL-07</p><p>Richard Sproat, Tao Tao and ChengXiang Zhai. 2006. <i>Named entity transliteration with comparable cor­pora. </i>COLING-ACL-06</p><p>Andreas Stolcke. 2002. <i>SRILM - an extensible lan­guage modeling toolkit. </i>ICSLP-02</p><p>R. Udupa, K. Saravanan, A. Bakalov and A. Bhole. 2009. <i>They are out there, if you know where to look: Mining transliterations of OOV query terms for cross-language information retrieval. </i>In LNCS: Advances in Information Retrieval, volume 5478, pages 437-448. Springer</p><p>Masao Utiyama and Hitoshi Isahara. 2007. <i>A Compari­son of Pivot Methods for Phrase-based Statistical Machine Translation. </i>NAACL-HLT-07</p><p>Vladimir N. Vapnik. 1995. <i>The Nature of Statistical Learning Theory. </i>Springer</p><p>H. Wu and H. Wang. 2007. <i>Pivot Language Approach</i> <i>for Phrase-based SMT.</i><i> </i>ACL-07</p><p>H. Wu and H. Wang. 2009. <i>Revisiting Pivot Language Approach for Machine Translation. </i>ACL-09</p><p>Dmitry Zelenko and Chinatsu Aone. 2006. <i>Discrimina­tive methods for transliteration. </i>EMNLP-06</p><p>Min Zhang, Haizhou Li and Jian Su. 2004. <i>Direct Or­thographical Mapping for machine transliteration.</i></p><doubt alpha="66.7" length="9" tooSmall="False" monospace="0.0">COLING-04</doubt><p>Min Zhang, Xiangyu Duan, Vladimir Pervouchine and Haizhou Li. 2010. <i>Machine Transliteration: Leve­raging on Third Languages. </i>COLING-10 (poster)</p></references></body></article>