n-gram Language Models (LMs)have been widely used in current Statistical Machine Translation (SMT)systems. Because they treat a sentence as a flat string of tokens, a drawback of traditional n-gram LMs is that they cannot model long range word rela­tions, such as predicate-argument attachments, that are critical to translation quality.

We propose a hierarchical string-to-dependency translation model that exploits a dependency LM while decoding (as opposed to during reranking n-best output)to score alternative translations based on their structural soundness. In order to generate the structured output (dependency trees)required for dependency LM scoring, transla­tion rules in our system represent the target side as dependency structures. We restrict the target side of the rules to well-formed dependency structures to weed out bad translation rules and enable efficient decoding through dynamic programming. Due to the flexibility of well-formed dependency structures, such structures can cover a large set of non-constituent transfer rules (Marcu et al. 2006)that have been shown useful for MT.

For comparison purposes, as our baseline, we replicated the Hiero decoder (Chiang 2005), a state-of-the-art hierarchical string-to-string model. Our experiments show that the string-to-dependency decoder significantly improves MT performance. Overall, the

* 10 Moulton Street, Cambridge, MA 02138. E-mail: libinshen@gmail.com. ** 10 Moulton Street, Cambridge, MA 02138. E-mail: jxu@bbn.com. t 10 Moulton Street, Cambridge, MA 02138. E-mail: ueisched@bbn.com.

Submission received: 6 March 2009; revised submission received: 1 December 2009; accepted for publication: 18 March 2010.

Phrase-based systems (Koehn, Och, and Marcu 2003; Och 2003)had dominated SMT until recently. Such systems typically treat the input as a sequence of phrases (word n-grams), reorder them, and produce a translation for the reordered sentence based on translation options of each source phrase. A prominent feature of such systems is the use of an n-gram LM to measure the quality of translation hypotheses. A drawback of such systems is that the lack of structural information in the output makes it impossible to score translation hypotheses based on their structural soundness.

The Hiero system (Chiang 2007)was a major breakthrough in SMT. Translation rules in Hiero contain non-terminals (NTs), as well as words, which allow the input to be translated in a hierarchical manner. Because both the source and target sides of its translation rules are strings with NTs, Hiero can be viewed as a hierarchical string-to-string model. Despite the hierarchical nature of its decoder, Hiero lacks the ability to measure translation quality based on structural relations such as predicate-argument agreement.

Yamada and Knight (2001)proposed a syntax-based translation model that transfers a source parse tree into a target string. This method depends on the quality of source side parsing, and ignores target information during source side analysis. Mi, Huang, and Liu (2008)later proposed a translation model that takes the source parse forest as MT input to reduce translation errors due to imperfect source side analysis.

Galley et al. (2004)proposed an MT model which produces target parse trees for string inputs in order to exploit the syntactic structure of the target language. Galley et al. (2006)formalized this approach with tree transducers (Graehl and Knight 2004) by using context-free parse trees to represent the target side. However, it was later shown by Marcu et al. (2006)and Wang, Knight, and Marcu (2007)that coverage could be a big issue for the constituent based rules, even though the translation rule set was already very large.

Carreras and Collins (2009)introduced a string-to-tree MT model based on spinal Tree Adjoining Grammar (TAG)(Joshi and Schabes 1997; Shen, Champollion, and Joshi 2008). In this model, a translation rule is composed of a source string and a target elementary tree. Target hypothesis trees are combined with the adjoining operation, and there are no NT slots for substitution as in LTAG-spinal parsing (Shen and Joshi 2005, 2008; Carreras, Collins, and Koo 2008). Without the constraint of NT slots, the adjoining operation allows very flexible composition, so that the search space becomes much larger. One has to carefully prune the search space.

DeNeefe and Knight (2009)proposed another TAG-based MT model. In their implementation, a TAG grammar was transformed to an equivalent Tree Insertion

Grammar (TIG). In this way, they do not have an explicit adjoining operation in their system, and as such reduce the search space in decoding. Sub-trees are combined with NT substitution.

Many researchers followed the tree-to-tree approach (Shieber and Schabes 1990)to take advantage of structural knowledge on both sides—for example, as in the papers by Hajic et al. (2002), Eisner (2003), Ding and Palmer (2005), and Quirk, Menezes, and Cherry (2005). Although tree-to-tree models can represent rich structural information of the input and the output, they have not significantly improved MT performance, possibly due to a much larger grammar and search space. On the other hand, Smith and Eisner (2006)showed the necessity of allowing loose transformations between the trees, which made tree-to-tree models even more complicated.

Our system is designed to address problems with existing SMT approaches. It is novel in two respects. First, it uses a dependency LM to model long-distance relations. Sec­ond, it uses well-formed dependency structures to represent translation hypotheses to achieve an effective trade-off between model coverage and decoding complexity.

We first formally define the well-formed dependency structures, which are used to represent target hypotheses. Then, we define the operations to build well-formed de­pendency structures from the bottom up in decoding.

hong-> the red

Translation rules are central to an MT system. In our system, each rule translates a source sub-string into a target dependency structure. The target side of the translation rules constitutes a tree grammar.

One way to define a tree grammar is in the way we described earlier. Two well-formed structures can be combined into a larger one with adjoining or concatenation, and there is no non-terminal slot for substitution. This is similar to tree grammars without substitution, such as the original TAG (Joshi, Levy, and Takahashi i975)and LTAG-spinal (Shen, Champollion, and Joshi 2008). A corresponding MT model was proposed in Carreras and Collins (2009). Search space is a major problem for such an approach, as we described earlier.

In our system, we introduced NT substitution to combat the search problem. The NT slots for substitution come from what we have observed in training data. Combina­tion of well-formed dependency structures can only happen on NT slots. By replacing NT slots with well-formed structures, we implicitly adjoin or concatenate sub-structures based on the dependency information stored in rules. We extract the translation rules from the training data containing word-to-word alignment and target parse trees, which we will explain in the next section. A similar strategy was employed by DeNeefe and Knight (2009). They turned a TAG into an equivalent TIG.

In addition to these extracted rules, we also have special rules to adjoin or concate­nate two neighboring hypotheses. Each of the special rules has two NT slots, but they vary on target dependency structures. They are comparable to the glue rules in Chiang (2005).

To formalize translation rules and grammars, a string-to-dependency grammar G is a4-tuple G = (R, X, Tf, Te) where R is a set of transfer rules. X is the only non-terminal type. Tf is a set of terminals (words)in the source language, and Te is a set of terminals in the target language.

A string-to-dependency transfer rule R £R is a 4-tuple R = (Sf, Se, D, A) where Sf £ (Tf U{X})+ is a source string, Se £ (Te U{X})+ is a target string, D represents the dependency structure for Se,andA is the alignment between Sf and Se. Non-terminal alignments in A must be one-to-one. We ignore the left hand side for both source and target, since there is only one NT type.

By applying the inference rule recursively, we can generate rules with arbitrary aligned NT slots if there are enough words and alignments. In order to make the size of the grammar manageable, we keep only rules with at most two NT slots and at most seven source elements.

Following previous work (Och and Ney 2003; Chiang 2007), we have three features 2.

2 Pf represents the i to the j words on the source side, and P™'" represents the m to the n words on the target side. By aligned to P^^wemeanall wordsin Pf are either aligned to words in P™,n or unaligned, and vice versa. Furthermore, at least one word in Pf is aligned to a word in P^".

3If D2 is a floating structure, we need to merge several dependency links into one.

The values of the first four features are accumulated on the rules used in a transla­tion. The fifth feature counts the number of times the adjoining and concatenation rules are used in a translation. The string LM score and dependency LM score are the next two features.

In practice, we also allow hypotheses that do not have well-formed structures in derivation, but they are penalized. For this purpose, we introduce the null dependency structure e. For any operation OP and dependency structure X, Y, we have

OP(X, Y) = e if this operation is not defined in Definition 5

Because part of a hypothesis may have a null dependency structure, we cannot calculate dependency LM scores on some of the related words. Therefore, we give a discount for each of these words. This is the eighth feature.

There are two sources for null dependency structures. One is the use of an unde­fined operation, for example, left-adjoining a right floating structure to a fixed structure. The other source is a lack of target structure information in translation rules. The parser that we used may fail to generate parse trees for short segments—for example, dictio­nary items. In these cases, we extracted the so-called phrasal rules with null dependency structures. We limited phrasal rules to at most three lexical items for each side.

The last feature counts the number of substitutions with unmatched labels.

In decoding, partial hypotheses are mapped into states. The states maintain suf­ficient statistics for feature calculation. For example, each state should memorize the leftmost two words and rightmost two words for LM score calculation. Similar exten­sions are required for dependency LM score and NT labels. Therefore, we use beam search with cube pruning as in Chiang (2005)for speedup. Like chart parsing, the computational complexity of decoding time is 0(n x B x\G\), where n is the length of the source sentence, B is beam width, and \G\ is the maximal number of transfer rules applicable to a span with translation grammar G. This number agrees with the empirical results.

We tune the weights with several rounds of decoding and optimization. Following Och (2003), the fc-best results are accumulated as the input to the optimizer. Powell's method is used for optimization with 20 random starting points around the weight vector of the last iteration. For improved results, we rescore 1,000-best translations, generated using the technique described by Huang and Chiang (2005), by replacing tri-gram string LM scores in the output with 5-gram string LM scores. The algorithm to tune the rescoring weights is similar to the one to tune the decoder weights.

We experimented with four models:

• baseline: hierarchical string to string translation, using our own replication of the Hiero system (Chiang 2007)

• filtered: like the baseline, it uses string to string rules, except that rules whose target side does not correspond to a well-formed structure in rule extraction are excluded. No dependency LM is used in decoding

• str-dep: string-to-dependency system. It uses rules with target dependency structures and a dependency LM in decoding

• labeled: an enhanced str-dep model with POS tags as labels

We use the Hiero model as our baseline because it is the closest to our string-to-dependency model. They use similar rule extraction and decoding algorithms. The major difference is in the representation of target structures. We use dependency structures instead of strings; thus, the comparison will show the contribution of using dependency information in decoding.

All models were tuned on BLEU (Papineni, Roukos, and Ward 2001), and evaluated on BLEU, TER (Snover et al. 2006), and METEOR (Banerjee and Lavie 2005). It is well known that all automatic scores are crude approximations of translation quality. It is not uncommon for a technique to improve the metric that is used for tuning but hurt other metrics. The use of multiple metrics helps us avoid drawing false conclusions based on metric-specific improvements. For both Arabic-to-English and Chinese-to-English MT, we tuned on NIST MT02-05 and tested on MT06 and MT08 newswire sets.

The training data for Arabic-to-English MT contains around 1.9 million pairs of

LDC2002E18, LDC2005T06, LDC2005T10, LDC2006E26, LDC2006G05, LDC2002L27, LDC2005T34, and LDC2003E07.

The dependency LMs were trained on the same parallel training data. For that pur­pose, we parsed the English side of the parallel data. Two separate models were trained: one for Arabic from the Arabic training data and the other for Chinese from the Chinese training data. Traditional tri-gram and 5-gram string LMs were trained on

Number of transfer rules.

Model Arabic-to-English Chinese-to-English

the English side of the parallel data as well as the English Gigaword corpus V3.0 in a way described by Bulyko et al. (2007).

Table 1 shows the number of transfer rules extracted from the training data for the tuning and test sets. The constraint of well-formed dependency structures greatly reduced the size of the rule set. Although the rule size increased a little bit after incorpo­rating dependency structures and labels in rules, the size of string-to-dependency rule set is about 10% to 20% of the baseline.

Tables 2 and 3 show the BLEU, TER, and METEOR scores on MT06 and MT08 for

Arabic-to-English MT. Tables 4 and 5 show the scores for Chinese-to-English MT.

For system comparison, we primarily rely on the lower-cased BLEU score of the decoding output because it is the metric on which all systems were tuned. We measured the significance of BLEU, TER, and METEOR with paired bootstrap resampling as proposed by Koehn (2004). In Tables 2 through 5, (+/-)represent being better/worse than the baseline at 95% confidence level, respectively, and (*)represents insignificant difference from the baseline.

For Arabic-to-English MT, the str-dep model decoder improved BLEU by 1.3 on MT06 and 1.2 on MT08 before 5-gram rescoring. For Chinese-to-English MT, the im­provements in BLEU were 1.0 on MT06 and 1.4 on MT08. After rescoring, the improve­ments became smaller, ranging from 0.8 to 1.3. All the BLEU improvements on 5-gram scores are statistically significant.

The use of POS labels in transfer rules further improves the BLEU score by about 0.7 points on average. The overall BLEU improvement on lower-cased decoding output

BLEU, TER, and METEOR percentage scores on MT06 Arabic-to-English newswire set.

Model BLEU TER METEOR

Decoding (3-gram LM)

Rescoring (5-gram LM) Model BLEU TER METEOR

Decoding (3-gram LM)

Rescoring (5-gram LM)

BLEU, TER, and METEOR percentage scores on MT08 Arabic-to-English newswire set.

is 1.8 points on MT06 and 2.1 points on MT08 for Arabic-to-English translation, and 2.0 points on MT06 and 1.6 points on MT08 for Chinese-to-English translation.

METEOR scores became significantly better for all conditions. TER improved sig­nificantly for Arabic-to-English but marginally on Chinese-to-English tasks. The results on METEOR and TER suggested that the new model did improve translation accuracy.

The filtered string-to-string rules can be viewed as the string projection of string-to-dependency rules. It shows the performance of using dependency structure for rule filtering only. The results are very interesting. On Arabic-to-English, the filtered model was significantly worse, which means that many useful rules were lost due to the structural constraints. On Chinese-to-English, the tri-gram scores of the filtered model were a little bit worse. However, after 5-gram rescoring, the BLEU scores became higher than the baseline, and METEOR scores were even significantly better. We suspect that the different performance that we observed is due to the difference in source languages and their tokenization methods.

BLEU, TER, and METEOR percentage scores on MT06 Chinese-to-English newswire set. Model BLEU TER METEOR Model BLEU TER METEOR

BLEU, TER, and METEOR percentage scores on MT08 Chinese-to-English newswire set.

Decoding (3-gram L

Rescoring (5-gram L

In any case, the purpose of the filtered model is not to propose the use of structural constraints for rule filtering, although it greatly reduced the rule size and allowed the use of more useful training data potentially. The use of structural constraints is compulsory for the introduction of dependency LMs and non-terminal labels, which compensated for the loss of rule filtering, and led to significant overall improvement.

Fox (2002), Ding and Palmer (2005), and Quirk, Menezes, and Cherry (2005) showed that, for the purpose of representing word relations, dependency structures are ad­vantageous over CFG structures because they do not require complete constituents. A number of techniques have been proposed to improve rule coverage. Marcu et al. (2006) and Galley et al. (2006)introduced artificial constituent nodes dominating the phrase of interest. The binarization method used by Wang, Knight, and Marcu (2007)can cover many non-constituent rules also, but not all of them. DeNeefe et al. (2007)showed that the best results were obtained by combining these methods.

Charniak, Knight, and Yamada (2003)described a two-step string-to-CFG-tree translation model which employed a syntax-based language model to select the best translation from a target parse forest built in the first step. A crucial difference from our work is that they only used the tree-based LM in rescoring, possibly due to the com­plexity of the syntax-based LM. In contrast, our system uses a dependency LM directly in decoding and as such can prune out unpromising hypotheses as soon as possible.

The use of a dependency LM in MT is similar to the use of a structured LM in ASR (Chelba and Jelinek 2000; Xu, Chelba, and Jelinek 2002), with the same motivation of exploiting long-distance relations. A difference is that the dependency LM is used bottom-up in our MT system, whereas the structured LM is used left-to-right in ASR. Another difference is that long-distance relations are more important in MT due to word re-orderings.

The well-formed dependency structures defined here are similar to the data struc­tures in previous work on monolingual parsing (Eisner and Satta 1999; McDonald, Crammer, and Pereira 2005), which allowed floating structures as well defined states in derivation, too. However, as for monolingual parsing, one usually wants exactly one derivation for each parse tree, so as to avoid spurious ambiguity of derivations for the same parse. The derivation model proposed by Eisner and Satta (1999)satisfied this prerequisite, and had 0(n)complexity with a bi-lexical probability model, which was 0(n)in many other derivation models. In our MT model, the motivation is to exploit various translation fragments learned from the training data, and the opera­tions in monolingual parsing were designed to avoid artificial ambiguity of derivation. Another difference is that we have fixed structures growing on both sides, whereas fixed structures in (Eisner and Satta 1999)can only grow in one direction.

The formalism for well-formed structures and the operations over them were inspired by the well-known approach of Combinatory Categorial Grammar (CCG) (Steedman 2000). In fact, the names of left raising and right raising stem from the raising operation in CCG.

The string-to-dependency formalism can be viewed as a special case of Synchro­nous Tree Adjoining Grammar (STAG)(Shieber and Schabes 1990). Trees on the source side are weakened to strings, and multi-rooted structures are employed on the tar­get side. The adjoining operation in our model is similar to attachment in LTAG-spinal (Shen, Champollion, and Joshi 2008)and sister adjunction in variants (Rambow, Shanker, and Weir 1995; Chiang 2000; Carreras, Collins, and Koo 2008)of TAG (Joshi and Schabes 1997). Translation rules can be viewed as constraints on the tree operations.

In this article, we propose a novel string-to-dependency algorithm for statistical ma­chine translation. It employs a target dependency language model to exploit long dis­tance word relations in decoding, which cannot be captured with a traditional n-gram language model.

Compared with a state-of-the-art hierarchical string-to-string system, our string-to-dependency system generates about 80% fewer rules. The overall gain in BLEU score on lower-cased decoding output is about two points.

Dependency structures provide a desirable platform for employing linguistic knowledge in MT. We will extend our approach with deeper linguistic features such as propositional structures (Palmer, Gildea, and Kingsbury 2005). The fixed and floating structures proposed in this article can be extended to model predicates and arguments.

This work was supported by DARPA/ IPTO Contract No. HR0011-06-C-0022 under the GALE program. The views, opinions, and/or findings contained in this article/presentation are those of the author/presenter and should not be interpreted as representing the official views or policies, either expressed or implied, of the Defense Advanced Research Projects Agency or the Department of Defense. We are grateful to our colleagues Roger Bock, Ivan Bulyko, Mike Kayser, Jeff Ma, John

4 Distribution Statement "A" (Approved for Public Release, Distribution Unlimited).

Makhoul, Spyros Matsoukas, Antti-Veikko Rosti, Rich Schwartz, Bing Zhang, and Rabih Zbib for their help in running the experiments and constructive comments to improve this article. Mike Kayser helped to proofread the manuscript. We also thank Zhifei Li and the anonymous reviewers for their suggestions to improve this article.