Classical SVMs (Cortes and Vapnik, 1995) are binary classifiers that classify an input instance based on decision rules which minimize the reg­ularized error function in (Eq. 1):

where (x, y) G Rn x {1, —1} are l training in­stances. w is the weight vector, £ is the relaxation variable and C > 0 is the penalty parameter.

The SVM classification algorithm is extended to the ranking case in (Joachims, 2002). For a certain group of instances, the Ranking SVM aims at producing a ranking r that has the maximum Kendall's t coefficient with the the gold standard ranking r*.

Kendall's t measures the relevance of two rank­ings: t(ra,rb) = Q, where P and Q are the amount of concordant and discordant pairs in ra and rb. In practice, this is done by building constraints to minimize the discordant pairs Q. Following the basic idea, we show how Ranking SVM can be applied to MT-TM integration as fol­lows.

Assume that for each source sentence s, we have a set of outputs from MT, M and a set of outputs from TM, T. If we have a ranking r(s) over translation outputs M T where for each translation output d G MIJ T, (di, dj) G r(s) iff di <r(s) dj, we can rewrite the ranking constraints as optimization constraints in an SVM, as in Eq.

(2).

subject to:

where $(sn, di) is a feature vector of translation output di given source sentence sn. The Ranking SVM minimizes the discordant number of rank­ings with the gold standard according to Kendall's t .

When the instances are not linearly separable, we use a mapping function 0 to map the features xi ($(sn, di) in the case of ranking) to high di­mensional space, and solve the SVM with a kernel function K in where K(xi, xj) = 0(xi)T0(xj).

We perform our experiments with the Radial Basis Function (RBF) kernel, as in Eq. (3).

Ranking lists is a well-researched problem in the information retrieval community, and Ranking SVMs (Joachims, 2002), which optimizes on the ranking correlation t have already been applied successfully in machine translation evaluation (Ye et al., 2007). We apply the same method here to rerank a merged list of MT and TM outputs.

Formally given an MT-produced N-best list M = {mi, m2,mn}, a TM-produced m-best list T = {ti,t2,...,tm} for a input sentence s, we define the gold standard using the TER met­ric (Snover et al., 2006): for each d G M T, (di,dj) G r(s) iff TER(di) < TER(dj). We train and test a Ranking SVM using cross vali­dation on a data set created according to this cri­terion. Ideally the gold standard would be cre­ated by human annotators. We choose to use TER as large-scale annotation is not yet available for this task. Furthermore, TER has a high correla­tion with the HTER score (Snover et al., 2006), which is the TER score using the post-edited MT output as a reference, and is used as an estimation of post-editing effort.

When building features for the Ranking SVM, we are limited to features that are independent of the MT and TM system. We experiment with system-independent fluency and fidelity features below, which capture translation fluency and adequacy, respectively.

Our raw data set is an English-French trans­lation memory with technical translation from a multi-national IT security company, consisting of 51K sentence pairs. We randomly select 43K to train an SMT system and translate the English side of the remaining 8K sentence pairs, which is used to run cross validation. Note that the 8K sentence pairs are from the same TM, so that we are able to create a gold standard by ranking the TER scores of the MT and TM outputs.

Duplicated sentences are removed from the data set, as those will lead to an exact match in the TM system and will not be translated by trans­lators. The average sentence length of the training set is 13.5 words and the size of the training set is comparable to the (larger) translation memories used in the industry.

Ranking SVM

We run training and prediction of the Ranking SVM in 4-fold cross validation. We use the SVMlight toolkit to perform training and testing.

When using the Ranking SVM with the RBF kernel, we have two free parameters to tune on: the cost parameter C in Eq. (1) and the radius parameter 7 in Eq. (3). We optimize C and 7 using a brute-force grid search before running cross-validation and maximize precision at top-5, with an inner 3-fold cross validation on the (outer) Fold-1 training set. We search within the range [2-6, 2], the step size is 2 on the exponent.

Figure 1 shows the composition of translations in the gold standard. Each source sentence is asso­ciated with a list of translations from two sources, i.e. MT output and TM matches. This list of translations is ranked from best to worst accord­ing TER scores. The figure shows that over 80% of the translations are from the MT system if we only consider the top-1 translation. As the num­ber of top translations we consider increases, more TM matches can be seen. On the one hand, this does show a large gap in quality between MT out­put and TM matches; on the other hand, however, it also reveals that we will have to ensure two ob­jectives in ranking: the first is to rank the 80% MT translations higher and the second is to keep the 20% 'good' TM hits in the Top-5. We design our evaluation metrics accordingly.

The aim of this research is to provide post-editors with translations that in many cases are easier to edit than the original TM output. As we formulate this as a ranking problem, it is natural to measure the quality of the ranking output by the number of better translations that are ranked high. Some­times the top TM output is the easiest to edit; in such a case we need to ensure that this translation has a high rank, otherwise the system performance will degrade.

Based on this observation, we introduce the idea of relevant translations, and our evaluation metrics: PREC@k and HIT@k.

Relevant Translations. We borrow the idea of relevence from the IR community to define the idea of translations worth ranking high. For a source sentence s which has a top TM hit t, we define an MT/TM output m as relevant, if TER(m) < TER(t). According to the defini­tion, relevant translations should need no more post-edits than the original top hit from the TM system. Clearly the top TM hit is always relevant.

PREC@k. We calculate the precision (PREC@k) of the ranking for evaluation. As­suming that there are n relevant translations in the top k list for a source sentence s, we have PREC@k= n/k for s. We test PREC@k, for k = 1...10, in order to evaluate the overall quality of the ranking.

HIT@k. We also estimate the probability of having one of the relevant translations in the top k, denoted as HIT@k. For a source sentence s, HIT@k equals to 1 if there is at least one relevant translation in top k, and 0 otherwise. This mea­sures the quality of the best translation in top k, which is the translation the post-editor will find and work on if she reads till the kth place in the list. HIT@k equals to 1.0 at the end of the list.

We report the mean PREC@k and HIT@k for all s with the 0.95 confidence interval.

In Table 1 we report PREC@k and HIT@k for k = 1..10. The ranking receives 0.8747 PREC@1, which means that most of the top ranked translations have at least the same quality as the top TM output. We notice that precision re­mains above 0.8 till k = 5, leading us to conclude that most of the relevant translations are ranked in the top-5 positions in the list.

We report the results on the 1-best output of TM, MT and our ranking system in Table 3.

In the single best results, it is easy to see that the 1-best output from the MT system requires the least post-editing effort. This is not surpris­ing given the distribution of the gold standard in Section 5.3, where most MT outputs are of better quality than the TM hits.

Moreover, since TM translations are generally of much lower quality as is indicated by the num­bers in Table 3 (e.g. 2x as many substitutions and 3x as many deletions compared to MT), un­justly including very few of them in the ranking output will increase loss in the edit statistics. This explains why the ranking model has better rank­ing precision in Tables 1 and 2, but seems to in­cur more edit efforts. However, in practice post­editors can neglect an obvious 'bad' translation very quickly.

We report edit statistics of the Top-3 and Top-5 outputs in Tables 4 and 5, respectively. For each system we report two sets of statistics: the Best-statistics calculated on the best output (according

Insertion Substitution Deletion Shift to TER score) in the list, and the Mean- statistics calculated on the whole Top-k list.

The Mean- numbers allow us to have a general overview of the ranking quality, but it is strongly influenced by the poor TM hits that can easily be neglected in practice. To control the impact of those TM hits, we rely on the Best- numbers to es­timate the edits performed on the translations that are more likely to be used by post-editors.

In Table 4, the ranking output's edit statistics is closer to the MT output than the Top-1 case in Table 3. Table 5 continues this tendency, in which the Best-in-Top5 Ranking output requires marginally less Substitution and Deletion opera­tions and significantly less Insertion and Shift op­erations (starred) than its MT counterpart. This shows that when more of the list is explored, the advantage of the ranking model - utilizing mul­tiple translation sources - begins to compensate for the possible large number of edits required by poor TM hits and finally leads to reduced post­editing effort.

There are several explanations to why the rel­ative performance of the ranking model improves when k increases, as compared to other models. The most obvious explanation is that a single poor translation is less likely to hurt edit statistics on a k-best list with large k, if most of the transla­tions in the k-best list are of good quality. We see from Tables 1 and 2 that the ranking output is of better quality than the MT and TM outputs w.r.t. precision. For a larger k, the small number of in­correctly ranked translations are less likely to be chosen as the Best- translation and hold back the Best- numbers.

A further reason is related to our ranking model which optimizes on Kendall's t score. Accord­ingly the output might not be optimal when we evaluate the Top-1 output, but will behave better when we evaluate on the list. This is also in ac­cordance with our aim, which is to enrich the TM with MT outputs and help the post-editor, instead of choosing the translation for the post-editor.

One of the interesting findings from Tables 3 and 4 is that according to the TER edit statistics, the MT outputs generally need a smaller number of edits than the TM and Ranking outputs. This cer­tainly confirms the necessity to integrate MT into today's TM systems.

However, this fact should not lead to the con­clusion that TMs should be replaced by MT com­pletely. First of all, all of our experiments exclude exact TM matches, as those translations will sim­ply be reused and not translated. While this is a realistic setting in the translation industry, it re­moves all sentences for which the TM works best from our evaluations.

Furthermore, Table 5 shows that the Best-in-Top5 Ranking output performs better than the MT outputs, hence there are TM outputs that lead to smaller number of edits. As k increases, the rank­ing model is able to better utilize these outputs.

Finally, in this task we concentrate on rank­ing useful translations higher, but we are not in­terested in how useless translations are ranked. Ranking SVM optimizes on the ranking of the whole list, which is slightly different from what we actually require. One option is to use other optimization techniques that can make use of this property to get better Top-k edit statistics for a smaller k. Another option is obviously to perform regression directly on the number of edits instead of modeling on the ranking. We plan to explore these ideas in future work.