<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<TEI xmlns="http://www.tei-c.org/ns/1.0" xmlns:ns2="http://www.tei-c.org/ns/Examples">
    <teiHeader>
        <fileDesc>
            <titleStmt>
                <title>Using Machine Translation Evaluation Techniques to Determine Sentence-level Semantic Equivalence</title>
            </titleStmt>
        </fileDesc>
    </teiHeader>
    <text>
        <front>
            <div type="abs">
                <head>Abstract</head>
                <p>The task of machine translation (MT) evaluation is closely related to the task of sentence-level semantic equivalence classification. This paper investigates the utility of applying standard MT evaluation methods (BLEU, NIST, WER and PER) to building classifiers to predict semantic equivalence and entailment. We also introduce a novel classification method based on PER which leverages part of speech information of the words contributing to the word matches and non-matches in the sentence. Our results show that MT evaluation techniques are able to produce useful features for paraphrase classification and toa lesser extent entailment. Our technique gives a substantial improvement in paraphrase classification accuracy over all of the other models used in the experiments.</p>
            </div>
        </front>
        <body>
            <div>
                <p/>
            </div>
            <div1>
                <head xml:id="sec1">Introduction</head>
                <p>Automatic machine translation evaluation is a means of scoring the output from a machine translation system with respect to a small corpus of reference translations. The basic principle being that an output is a good translation if it is ‘close’ in some way to a member of a set of perfect translations for the input sentence. The closeness that these techniques are trying to capture is in essence the notion of semantic equivalence. Two sentences being semantically equivalent if they convey the same meaning.</p>
                <p>MT evaluation techniques have found application in the field of entailment recognition, a close relative of semantic equivalence determination that seeks methods for deciding whether the information provided by one sentence is included in an another. (Perez and Alfonseca, 2005) directly applied the BLEU score to this task and (Kouylekov and Magnini, 2005) applied both a word and tree edit distance algorithm. In this paper we evaluate these techniques or variants of them and other MT evaluation techniques on both entailment and semantic equivalence determination, to allow direct comparison to our results.</p>
                <p>When using a single reference sentence for each candidate the task of deciding whether a pair of sentences are paraphrases and the task of MT evaluation are very similar. Differences arise from the nature of the sentences being compared, that is MT output might not consist of grammatically correct sentences. Moreover, MT evaluation scoring need not necessarily be computed on a sentence-by-sentence basis, but can be based on statistics derived at the corpus level. Finally, the process of MT evaluation is asymmetrical. That is, there is a distinction between the references and the candidate machine translations. Fortunately, the automatic MT evaluation techniques commonly in use do not make any explicit attempt to score grammaticality, and (except BLEU) decompose naturally into their component scores at the sentence level. (Blatz et al., 2004) used a variant of the WER score and the NIST score at the sentence level to assign correctness to translation candidates, by scoring them with respect to a reference set. These correctness labels were used as the ‘ground truth’ for classifiers for the correctness of translation candidates for candidate sentence confidence estimation. We too adopt sentence level versions of these scores and use them to classify paraphrase candidates.</p>
                <p>The motivation for these experiments is twofold: firstly to determine how useful the features used by these MT evaluation techniques to semantic equivalence classifiers. One would expect that systems that perform well in one domain should also perform well in the other. After all, determining sentence level semantic equivalence is “part of the job” of an MT evaluator. Our second motivation is the conjecture that successful techniques and strategies will be transferable between the two tasks.</p>
            </div1>
            <div1>
                <head xml:id="sec2">MT Evaluation Methods</head>
                <p>MT evaluation schemes score a set of MT system output segments (sentences in our case) S = {s 1 ,s 2 ,...,s I } with respect to a set of references R corresponding to correct translations for their respective segments. Since we classify sentence pairs, we only consider the case of using a single reference for evaluation. Thus the set of references is given by: R = {r 1 ,r 2 ,...,r I } .</p>
                <div2>
                    <head xml:id="sec2.1">WER</head>
                    <p>Word error rate (WER) (Su et al., 1992) is a measure of the number of edit operations required to transform one sentence into another, defined as: WER ( s i ,r i )= [BAR] I ( s i ,r i )+ D ( |r s i i ,r | i )+ S ( s i ,r i ) where I ( s i ,r i ) , D ( s i ,r i ) and S ( s i ,r i ) are the number of insertions, deletions and substitutions respectively.</p>
                </div2>
                <div2>
                    <head xml:id="sec2.2">PER</head>
                    <p>Position-independent word error rate (PER) (Tillmann et al., 1997) is similar to WER except that word order is not taken into account, both sentences are treated as bags of words: PER ( s i ,r i )= [BAR] max [ diff ( s i ,r |r i i ) | ,diff ( r i ,s i )]</p>
                    <p>where diff ( s i ,r i ) is the number of words observed only in s i .</p>
                </div2>
                <div2>
                    <head xml:id="sec2.3">BLEU</head>
                    <p>The BLEU score (Papineni et al., 2001) is based on the geometric mean of n -gram precision. The score is given by: BLEU = BP ×exp [ n N∑ =1 [BAR] N 1 × log ( p n ) ] where N is the maximum n -gram size. The n -gram precision p n is given by:</p>
                    <p>∑ ∑ count (ngram)</p>
                    <p>i =1 ..I ngram ∈si p n = [BAR] ∑ ∑ count sys (ngram)</p>
                    <p>i =1 ..I ngram ∈si where count ( ngram ) is the count of ngram found in both s i and r i and count sys ( ngram ) is the count of ngram in s i .</p>
                    <p>The brevity penalty BP penalizes MT output for being shorter than the corresponding references and is given by: [</p>
                    <p>]] where L sys is the number of words in the MT output sentences and L ref is the number of words in the corresponding references.</p>
                    <p>The BLEU brevity penalty is a single value computed over the whole corpus rather than an average of sentence level penalties which would have made its effect too severe. For this reason, in our experiments we omit the brevity penalty from the BLEU score. Its effect is small since the reference sentences and system outputs are drawn from the same sample and have approximately the same average length.</p>
                    <p>We ran experiments for N =1 ... 4 , these are referred to as BLEU1 to BLEU4 respectively.</p>
                </div2>
                <div2>
                    <head xml:id="sec2.4">NIST</head>
                    <p>The NIST score (Doddington, 2002) also uses n -gram precision, differing in that an arithmetic mean is used, weights are used to emphasize informative word sequences and a different brevity penalty is used: ∑ info (ngram) N all ngram NIST = n ∑ =1 BP × [BAR] that co − occur [ BP = exp min 1 − [BAR] Lref Lsys , 1 ∑ 1 ngram ∈s i Sentence pair 1 (semantically equivalent): Sentence pair 2 (not semantically equivalent): Sentence pair 3 (semantically equivalent): Figure 1: Example sentences from the Microsoft Research Paraphrase Corpus (MSRP) info is defined to be: info (ngram) = log 2 [ [BAR] count count ((n (ngram) − 1)gram) ]</p>
                    <p>where count (ngram) is the count of ngram = ww 1 2 ...w n in all the reference translations, and (n − 1)gram is ww 1 2 ...w n− 1 .</p>
                    <p>For NIST the brevity penalty is computed on a segment-by-segment basis and is given by:</p>
                    <p>]] where L sys is the length of the MT system output, [BAR] L ref is the average number of words in a reference translation and β is chosen to make referred to as NIST1 to NIST5 respectively. We include the brevity penalty in the scores used for our experiments. [ [ BP = exp β log 2 min [BAR] L L ref sys , 1</p>
                </div2>
                <div2>
                    <head xml:id="sec2.5">Introducing Part of Speech Information</head>
                    <p>Early experiments based on the PER score revealed that removing certain classes of function words from the edit distance calculation had a positive impact on classification performance. Instead of simply removing these words, we created a mechanism that would allow the classifier to learn for itself the usefulness of various classes of word. For example, one would expect edits involving nouns or verbs to cost more than edits involving interjections or punctuation. We used a POS tagger for the UPENN tag set (Marcus et al., 1994) to label all the data. We then divided the total edit distance, into components, one for each POS tag which hold the amount of edit distance that words bearing this POS tag contributed to the total edit distance. The feature vector therefore having one element for each UPENN POS tag. Let W − be the bag of words from s i that have no matches in r i and let W + be the bag of words from s i that have matches in r i . The value of the ~ feature vector f − corresponding to the contribution to the PER from POS tag t is given by: f t − = [BAR] ∑ w∈W − |s count i | − t ( w )</p>
                    <p>where count − t ( w ) is the number of times word occurs in W − w with tag t .</p>
                    <p>The feature vector defined above characterizes the nature of the words in the sentences that do not match. However it might also be important to include information on the words in the sentence that match. To investigate this, we augment the ~ feature vector f − with an analogous set of fea~ tures f + (again one for each UPENN POS tag) that represent the distribution over the tag set of word unigram precision, given by: f t + = [BAR] ∑ w∈W + |s count i | + t ( w )</p>
                    <p>where count + t ( w ) is the number of times word w occurs in W + with tag t .</p>
                    <p>This technique is analogous to the NIST score in that it allows the classifier to weight the importance of matches, but differs in that this weight is learned rather than defined, and is with respect to the word’s grammatical/semantic role rather than of rarity. When ~ + ~ function both f and f − as a are Table 1: Corpus statistics (columns CD-IR are sub-tasks of PASCAL), “length difference ratio” is explained in Section 3, “edit distance” is the average Levenstein distance between the sentences of the pairs used in combination the method differs again by utilizing information about the nature of both the matching words and the non-matching words.</p>
                    <p>We will refer to the system based only on the ~ feature vector f − as POS- , that based only on ~ f + as POS+ and that based on both as POS.</p>
                </div2>
                <div2>
                    <head xml:id="sec2.6">Dealing with Synonyms</head>
                    <p>Often in paraphrases the semantic information carried by a word in one sentence is conveyed by a synonymous word in its paraphrase. To cover these cases we investigated the effect of allowing words to match with synonyms in the edit distance calculations. Another pilot experiment was run with a modified edit distance that allowed words in the sentences to match if their semantic distance was less than a specific threshold (chosen by visual inspection of the output of the system). The semantic distance measure we used was that of (Jiang and Conrath, 1997) defined using the relationships between words in the WordNet database (Fellbaum, 1998). A performance improvement of approximately 0.6% was achieved on the semantic equivalence task using the strategy.</p>
                </div2>
            </div1>
            <note n="1." place="below">Amrozi accused his brother, whom he called “the witness”, of deliberately distorting his evidence. 2. Referring to him as only “the witness”, Amrozi accused his brother of deliberately distorting his evidence.</note>
            <note n="1." place="below">Yucaipa owned Dominick’s before selling the chain to Safeway in 1998 for $2.5 billion. 2. Yucaipa bought Dominick’s in 1995 for $693 million and sold it to Safeway for $1.8 billion in 1998.</note>
            <note n="1." place="below">The stock rose $2.11, or about 11 percent, to close Friday at $21.51 on the New York Stock Exchange. 2. PG&amp;E Corp. shares jumped $1.63 or 8 percent to $21.03 on the New York Stock Exchange on Friday.</note>
            <div1>
                <head xml:id="sec3">Experimental Data</head>
                <p>Two corpora were used for the experiments in this paper: the Microsoft Research Paraphrase Corpus (MSRP) and the PASCAL Challenge’s entailment recognition corpus (PASCAL). Corpus statistics for these corpora (after pre-processing) are presented in Table 1.</p>
                <p>The MSRP corpus consists of 5801 sentence pairs drawn from a corpus of news articles from the internet. The sentences were annotated by human annotators with labels indicating whether or not the two sentences are close enough in meaning to be close paraphrases. Multiple annotators were used to annotate each sentence: two annotators labeled the data and a third resolved the cases where they disagreed. The average interannotator agreement on this task was 83%, indicating the difficulty in defining the task and the ambiguity of the labeling. Approximately 67% of the sentences were judged to be paraphrases. The data was divided randomly into 4076 training sentences and 1725 test sentences. For full details of how the corpus was collected we refer the reader to the corpus documentation. To give an idea of the nature of the data and the difficulty of the task, three sentences from the corpus are shown in Figure 1. The example sentences show the ambiguity inherent in this task. The first sentence pair is clearly a pair of paraphrases. The second pair of sentences share semantic information, but were judged to be not semantically equivalent. The third pair are not paraphrases, they are clearly describing the movements of totally different stocks, but the sentences share sufficient semantic content to be labeled equivalent.</p>
                <p>For the MSRP corpus we present results using the provided training and test sets to allow comparison with our results. To obtain more accurate figures and to get an estimate of the confidence intervals we also conducted experiments by 10fold jackknifing over all the data. The results from each fold were then averaged and 95% confidence intervals were estimated for the means.</p>
                <p>The PASCAL data consists of 567 development sentences and 800 test sentences drawn from 7 domains: comparable document (CD), information extraction (IE), machine translation (MT), question answering (QA), reading comprehension (RC), paraphrasing (PP) and information retrieval (IR). A full description of this corpus is given in Figure 2: Example feature vector is labeled with these POS tags (in the upper part of the squares) and the the corpus documentation 1 . The data differs from the MSRP corpus in that it is annotated for entailment rather than semantic equivalence. This explains the asymmetry in the sentence lengths, which is apparent even in the PP component of the corpus. We do not present results for 10-fold jackknifing on the PASCAL data since the data were too small in number for this type of analy- sis.</p>
                <p>In Table 1 “Sentence 1” refers to the first sentence of a sentence pair in the corpus, and “Sentence 2” the second. The length distance ratio (LDR) is defined to be the average over the corpus of: LDR ( s i ,r i )= [BAR] max ||s i ( |−|r |s i |,|r i || i | )</p>
                <p>This measures the similarity of the lengths of the sentences in the pairs, it has the property of being 0 when all sentence pairs have sentences of the same length and 1 when all sentence pairs differ maximally in length. For the PASCAL corpus the LDR is around 0.5 for the corpus as a whole, corresponding to a large difference in the sentence lengths. The CD component of the corpus being considerably more consistent in terms of sentence length. The differences among the tasks in terms of edit distance are less clear-cut, with the PP task having the lowest average edit distance despite its higher LDR. The MSRP corpus has an LDR of only 0.14. The sentences pairs are more similar in terms of their length and edit distance than those in the PASCAL corpus. We will argue later that this length similarity has a significant effect on the performance and applicability of these techniques.</p>
            </div1>
            <note n="1" place="below">http://www.pascal-network.org/Challenges/RTE/</note>
            <div1>
                <head xml:id="sec4">Experimental Methodology</head>
                <div2>
                    <head xml:id="sec4.1">Tokenization</head>
                    <p>In order that the sentences could be tagged with UPENN tags (Marcus et al., 1994), they were preprocessed by a tokenizer. After tokenization the average MSRP sentence length was 21 words.</p>
                </div2>
                <div2>
                    <head xml:id="sec4.2">Stemming</head>
                    <p>Stemming conflates morphologically related words to the same root and has been shown to have a beneficial effect on IR tasks (Krovetz, 1993). A pilot experiment showed that the performance of a PER-based system degraded if the stemmed form of the word was used in place of the surface form. However, if the stemmer was applied only to words labeled by a POS tagger as verbs and nouns, a performance improvement of around 0.8% was observed on the semantic equivalence task. Therefore, for the purposes of the experiments, the nouns and verbs in the sentences were all pre-processed by a stemmer.</p>
                </div2>
                <div2>
                    <head xml:id="sec4.3">Classification</head>
                    <p>We used a support vector machine (SVM) classifier (Vapnik, 1995) with radial basis function kernels to classify the data. The training sets for the respective corpora were used for training, except in the jackknifing experiments. Feature vectors (an example is given in Figure 2) were constructed directly from the output of the MT evaluation systems, when used. The vector has 2 parts, one due to matches and one due to non-matches. The sum of the elements corresponding to nonmatches is equal to the PER. We calculated the vectors for each sentence in the pair as both reference and system output and averaged to get the vector for the pair.</p>
                </div2>
            </div1>
            <div1>
                <head xml:id="sec5">Results</head>
                <div2>
                    <head xml:id="sec5.1">MSRP Corpus</head>
                    <p>The results for the jackknifing experiments are shown in Table 2 and the results using the provided training and test sets are shown in Table 3. In the tables the rows labeled “PER POS+”, refer to models built using feature vectors made by combining both the PER and POS+ feature vectors. The rows labeled POS refer to models built from the combination of features from the POS+ and POS- models. The rows labeled ALL refer to models built from combining all of the features used in these experiments.</p>
                    <p>The results show that decomposing the PER edit distance score into components for each POS tag is not able to better the classification performance of PER. The accuracy (jackknifing) for PER alone was 71.25% and the accuracy for the analogous technique which divides this information in contributions for each POS tag (POS-) was 70.99%. However, when the features from PER and POS- are combined there is an improvement in performance (to 72.71%) indicating that the components for each POS tag are useful, but only in addition to the more primitive feature encoding the total edit distance. Moreover, comparing the results from POS-, POS+ and POS it is clear that there lot to be gained by considering the contributions from both the matching words and the non-matching words. Using both together gives a classification performance of 74.2% whereas using either component in isolation can give a performance no better than 71.5%.</p>
                    <p>The one of the worst performing systems was that based on the WER score. However, it is possible that the way the sentences were selected handicapped this system, since only sentences pairs with a word-based Levenshtein distance of 8 or higher were included in the corpus. Choosing sentence pairs with larger edit distances makes large structural differences more likely, and the editing effort needed to correct such structural differences may obscure the lexical comparison that this score relies upon.</p>
                    <p>The results for the BLEU score were unexpected because the performance degrades as the order of n -gram considered increases. This effect is much less apparent in the NIST scores where the performance degrades but to a lesser extent. Paraphrases exhibit variety in their grammatical structure and perhaps changes in word ordering can explain this effect. If so, the geometric mean employed in the BLEU score would make the effect of higher order n -grams considerably more detrimental than with the arithmetic mean used in the NIST score.</p>
                </div2>
                <div2>
                    <head xml:id="sec5.2">PASCAL Challenge Corpus</head>
                    <p>The results for the PASCAL corpus are given in Table 4. As expected our results are consistent with those of (Perez and Alfonseca, 2005). The 5% overall gain in accuracy may be accounted for by the stemming and synonym extensions to our technique and the fact that we used BLEU1. Our approach also differs by being symmetrical over source and reference sentences, however it is not clear whether this would improve performance. The number of test examples for the sub-experiments for each task is low (50 to 150), therefore the results here are likely to be noisy, but it is apparent from our results that the CD task is the most suitable for approaches based on word/ n -gram matching. Our POS technique performed well on overall and particularly well on the CD and MT tasks, but the overall performance improvement relative to the other techniques is not as clear-cut. We believe this is due to difficulties arising from the asymmetrical nature of the data, and we explore this in the next section.</p>
                </div2>
                <div2>
                    <head xml:id="sec5.3">Sentence length similarity</head>
                    <p>In this experiment we investigate whether there is any advantage to be gained by using these techniques on corpora consisting of sentence pairs of similar length. Both the BLEU and NIST scores use some form of count of the total number of n -grams in the denominator of their n -gram precision formulae. When the sentences differ in length, the total number of n -grams is likely to be large in relation to the number of matching n grams since this is bounded by the number of n grams in the shorter sentence. This may result in an increase in the ‘noise’ in the score due to variations in sentence length similarity, degrading its effectiveness. To address the more general issue of whether sentence length similarity has an impact on the effectiveness of these techniques we Table 2: Experimental Results (10-fold Jackknifing) Table 3: Experimental Results (Microsoft’s Provided Train and Test Set) sorted the sentences pairs of the MSRP corpus according to the length difference ratio (LDR) defined in Section 3, and partitioned the sorted corpus into two: lowand highLDR.We thenselected as many sentences as possible from the corpus such that the training and test sets for each data set (high and low LDR) contained the same number positive and negative examples. This gave two sets (high and low LDR) of 1008 training examples and 438 test examples, all training and test data consisiting of 50% positive and 50% negative examples. The results are shown in Table 5. The experimental results validate our concerns. In all of the cases the performance was higher on the data with low LDR. Moreover, the effect was most for the BLEU and NIST scores for which we have an explanation of the cause.</p>
                </div2>
            </div1>
            <div1>
                <head xml:id="sec6">Conclusion</head>
                <p>We have shown that it is possible to derive features that can be used to determine whether similar sentences are paraphrases of each other from methods currently being used to automatically evaluate machine translation systems. The experiments also show that using features that encode the distribution over the POS tag set of both matching words and non-matching words can significantly enhance the performance of a PERbased system on this task. Table 4: Accurracy Results (PASCAL Train and PASCAL Test Set) Table 5: Accuracy Results Length Similarity (MSRP)</p>
                <p>This research begs the important question “Is there any correlation between performance on the semantic equivalence classification task and performance of the underlying evaluation technique on the task of MT evaluation?”. Intuitively at least, there certainly should be. If there is, it may be possible to use the task of classifying sentences for semantic equivalence as a proxy for the complex and time-consuming task of evaluating evaluation schemes by correlating automatic scores with human scores during the development process of MT evaluation techniques. In future work we look forward to addressing this question, as well as incorporating new features into the models to increase their potency.</p>
            </div1>
            <div1>
                <head xml:id="sec7">Acknowledgments</head>
                <p>The research reported here was supported in part by a contract with the National Institute of Information and Communications Technology entitled “A study of speech dialogue translation technology based on a large corpus”. References J. Blatz, E. Fitzgerald, G. Foster, S. Gandrabur,</p>
                <p>C. Goutte, A. Kulesza, A. Sanchis, and N. Ueffing.</p>
                <p>2004. Confidence estimation for machine transla-</p>
                <p>tion. Technical report, Final report JHU / CLSP</p>
                <p>2003 Summer Workshop, Baltimore. G. Doddington. 2002. Automatic Evaluation of</p>
                <p>Machine Translation Quality Using N-gram Co-</p>
                <p>Occurrence Statistics. In Proceedings of the HLT</p>
                <p>Conference , San Diego, California. C. Fellbaum. 1998. WordNet: An Electronic Lexical</p>
                <p>Database. MIT Press. Jay J. Jiang and David W. Conrath. 1997. Seman-</p>
                <p>tic similarity based on corpus statistics and lexical</p>
                <p>taxonomy. CoRR, 9709008. Milen Kouylekov and Bernardo Magnini. 2005.</p>
                <p>Recognizing textual entailment with tree edit dis-</p>
                <p>tance algorithms. In Proceedings PASCAL Chal-</p>
                <p>lenges Worshop on Recognising Textual Entailment ,</p>
                <p>Southampton, UK. Robert Krovetz. 1993. Viewing morphology as an</p>
                <p>inference process. Technical Report UM-CS-1993-</p>
                <p>036, University of Mass-Amherst, April. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann</p>
                <p>Marcinkiewicz. 1994. Building a large annotated</p>
                <p>corpus of english: The penn treebank. Computa-</p>
                <p>tional Linguistics , 19(2):313–330. K. Papineni, S. Roukos, T. Ward, and W.J. Zhu. 2001.</p>
                <p>Bleu: a Method for Automatic Evaluation of Ma-</p>
                <p>chine Translation . IBM Research Report rc22176</p>
                <p>(w0109022), Thomas J. Watson Research Center. Diana Perez and Enrique Alfonseca. 2005. Appli-</p>
                <p>cation of the bleu algroithm for recognising tex-</p>
                <p>tual entailments. In Proceedings PASCAL Chal-</p>
                <p>lenges Worshop on Recognising Textual Entailment ,</p>
                <p>Southampton, UK. K.Y. Su, M.W. Wu, and J.S. Chang. 1992. A new</p>
                <p>quantitative quality measure for machine transla-</p>
                <p>tion systems. In Proceedings of COLING-92 , pages</p>
                <p>433–439, Nantes, France. C. Tillmann, S. Vogel, H. Ney, A. Zubiaga, and</p>
                <p>H. Sawaf. 1997. Accelerated dp based search</p>
                <p>for statistical translation. In Proceedings of</p>
                <p>Eurospeech-97 , pages 2667–2670, Rhodes, Greece. Vladimir N. Vapnik. 1995. The nature of statistical</p>
                <p>learning theory . Springer-Verlag New York, Inc.,</p>
                <p>New York, NY, USA.</p>
            </div1>
        </body>
        <back/>
    </text>
</TEI>
