<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="564"/><title>Further Meta-Evaluation of Broad-Coverage Surface Realization</title><pubinfo>Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing,pages 564—574, MIT, Massachusetts, USA, 9-11 October 2010. ©2010 Association for Computational Linguistics</pubinfo><author surname="Espinosa" givenname="Dominic"><org  name="Ohio State University" country="USA" city="Columbus"/></author><author surname="Rajkumar" givenname="Rajakrishnan"><org  name="Ohio State University" country="USA" city="Columbus"/></author><author surname="White" givenname="Michael"><org  name="Ohio State University" country="USA" city="Columbus"/></author><author surname="Berleant" givenname="Shoshana"><org  name="Bath University" country="United Kingdom"/></author></firstpageheader><frontmatter><p><b>Further Meta-Evaluation of Broad-Coverage Surface Realization</b></p><p><b>Dominic Espinosa </b>and <b>Rajakrishnan Rajkumar </b>and <b>Michael White </b>and <b>Shoshana Berleant</b></p><p>Department of Linguistics The Ohio State University Columbus, Ohio, USA</p><p>{espinosa,raja,mwhite,berleant}@ling.ohio-state.edu</p></frontmatter><abstract>We present the first evaluation of the utility of automatic evaluation metrics on surface real­izations of Penn Treebank data. Using outputs of the OpenCCG and XLE realizers, along with ranked WordNet synonym substitutions, we collected a corpus of generated surface re­alizations. These outputs were then rated and post-edited by human annotators. We eval­uated the realizations using seven automatic metrics, and analyzed correlations obtained between the human judgments and the auto­matic scores. In contrast to previous NLG meta-evaluations, we find that several of the metrics correlate moderately well with human judgments of both adequacy and fluency, with the TER family performing best overall. We also find that all of the metrics correctly pre­dict more than half of the significant system­level differences, though none are correct in all cases. We conclude with a discussion of the implications for the utility of such metrics in evaluating generation in the presence of varia­tion. A further result of our research is a cor­pus of post-edited realizations, which will be made available to the research community. </abstract></header><body><section number="1" title="Introduction and Background"><p>In building surface-realization systems for natural language generation, there is a need for reliable automated metrics to evaluate the output. Unlike in parsing, where there is usually a single gold­standard parse for a sentence, in surface realization there are usually many grammatically-acceptable ways to express the same concept. This parallels the task of evaluating machine-translation (MT) sys­tems: for a given segment in the source language, there are usually several acceptable translations into the target language. As human evaluation of trans­lation quality is time-consuming and expensive, a number of automated metrics have been developed to evaluate the quality of MT outputs. In this study, we investigate whether the metrics developed for MT evaluation tasks can be used to reliably evaluate the outputs of surface realizers, and which of these metrics are best suited to this task.</p><p>A number of surface realizers have been devel­oped using the Penn Treebank (PTB), and BLEU scores are often reported in the evaluations of these systems. But how useful is BLEU in this con­text? The original BLEU study (Papineni et al., 2001) scored MT outputs, which are of generally lower quality than grammar-based surface realiza­tions. Furthermore, even for MT systems, the usefulness of BLEU has been called into question (Callison-Burch et al., 2006). bleu is designed to work with multiple reference sentences, but in tree-bank realization, there is only a single reference sen­tence available for comparison.</p><p>A few other studies have investigated the use of such metrics in evaluating the output of NLG sys­tems, notably (Reiter and Belz, 2009) and (Stent et al., 2005). The former examined the performance of BLEU and ROUGE with computer-generated weather reports, finding a moderate correlation with human fluency judgments. The latter study applied sev­eral MT metrics to paraphrase data from Barzilay and Lee's corpus-based system (Barzilay and Lee, 2003), and found moderate correlations with human adequacy judgments, but little correlation with flu­ency judgments. Cahill (2009) examined the perfor­mance of six MT metrics (including BLEU) in evalu­ating the output of a LFG-based surface realizer for<page local="2" global="565"/></p><p>German, also finding only weak correlations with the human judgments.</p><p>To study the usefulness of evaluation metrics such as bleu on the output of grammar-based surface realizers used with the PTB, we assembled a cor­pus of surface realizations from three different re-alizers operating on Section 00 of the PTB. Two human judges evaluated the adequacy and fluency of each of the realizations with respect to the ref­erence sentence. The realizations were then scored with a number of automated evaluation metrics de­veloped for machine translation. In order to investi­gate the correlation of targeted metrics with human evaluations, and gather other acceptable realizations for future evaluations, the judges manually repaired each unacceptable realization during the rating task. In contrast to previous NLG meta-evaluations, we found that several of the metrics correlate moder­ately well with human judgments of both adequacy and fluency, with the ter family performing best. However, when looking at statistically significant system-level differences in human judgments, we found that some of the metrics get some of the rank­ings correct, but none get them all correct, with dif­ferent metrics making different ranking errors. This suggests that multiple metrics should be routinely consulted when comparing realizer systems.</p><p>Overall, our methodology is similar to that of previous MT meta-evaluations, in that we collected human judgments of system outputs, and com­pared these scores with those assigned by auto­matic metrics. A recent alternative approach to para­phrase evaluation is ParaMetric (Callison-Burch et al., 2008); however, it requires a corpus of annotated (aligned) paraphrases (which does not yet exist for PTB data), and is arguably focused more on para­phrase analysis than paraphrase generation.</p><p>The plan of the paper is as follows: Section 2 dis­cusses the preparation of the corpus of surface real­izations. Section 3 describes the human evaluation task and the automated metrics applied. Sections 4 and 5 present and discuss the results of these evalua­tions. We conclude with some general observations about automatic evaluation of surface realizers, and some directions for further research.</p></section><section number="2" title="Data Preparation"><p>We collected realizations of the sentences in Sec­tion 00 of the WSJ corpus from the following three sources:</p></section><section number="1." title="OpenCCG, a CCG-based chart realizer (White,"><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">2006)</doubt><p>2. The XLE Generator, a LFG-based system de­veloped by Xerox PARC (Crouch et al., 2008)</p><p>3. WordNet synonym substitutions, to investigate how differences in lexical choice compare to grammar-based variation.<footnote anchor="1"/></p><p>Although all three systems used Section 00 of the PTB, they were applied with various parame­ters (e.g., language models, multiple-output versus single-output) and on different input structures. Ac­cordingly, our study does not compare OpenCCG to XLE, or either of these to the WordNet system.</p><subsection number="2.1" title="OpenCCG realizations"><p>OpenCCG is an open source parsing/realization library with multimodal extensions to CCG (Baldridge, 2002). The OpenCCG chart realizer takes logical forms as input and produces strings by combining signs for lexical items. Alternative realizations are scored using integrated n-gram and perceptron models. For robustness, fragments are greedily assembled when necessary. Realiza­tions were generated from 1,895 gold standard logical forms, created by constrained parsing of development-section derivations. The following OpenCCG models (which differ essentially in the way the output is ranked) were used:</p></subsection></section><section number="1." title="Baseline 1: Output ranked by a trigram word model"><p>2. Baseline 2: Output ranked using three language models (3-gram words + 3-gram words with named entity class replacement + factored lan­guage model of words, POS tags and CCG su­pertags)<page local="3" global="566"/></p><footnote label="1">Not strictly surface realizations, since they do not involve an abstract input specification, but for simplicity we refer to them as realizations throughout.</footnote><p>3. Baseline 3: Perceptron with syntax features and the three LMs mentioned above</p><p>4. Perceptron full-model: n-best realizations ranked using perceptron with syntax features and the three n-gram models, as well as dis­criminative n-grams</p><p>The perceptron model was trained on sections 02­21 of the CCGbank, while a grammar extracted from section 00-21 was used for realization. In addition, oracle supertags were inserted into the chart during realization. The purpose of such a non-blind test­ing strategy was to evaluate the quality of the output produced by the statistical ranking models in isola­tion, rather than focusing on grammar coverage, and avoid the problems associated with lexical smooth­ing, i.e. lexical categories in the development sec­tion not being present in the training section.</p><p>To enrich the variation in the generated realiza­tions, dative-alternation was enforced during real­ization by ensuring alternate lexical categories of the verb in question, as in the following example:</p><p>(1) the executives gave [the chefs] [a stand­ing ovation]</p><p>(2) the executives gave [a standing ovation] [to the chefs]</p><subsection number="2.2" title="XLE realizations"><p>The corpus of realizations generated by the XLE system contained 42,527 surface realizations of ap­proximately 1,421 section 00 sentences (an aver­age of 30 per sentence), initially unranked. The LFG f-structures used as input to the XLE genera­tor were derived from automatic parses, as described in (Riezler et al., 2002). The realizations were first tokenized using Penn Treebank conventions, then ranked using perplexities calculated from the same trigram word model used with OpenCCG. For each sentence, the top 4 realizations were selected. The XLE generator provides an interesting point of comparison to OpenCCG as it uses a manually-developed grammar with inputs that are less abstract but potentially noisier, as they are derived from au­tomatic parses rather than gold-standard ones.</p></subsection><subsection number="2.3" title="WordNet synonymizer"><p>To produce an additional source of variation, the nouns and verbs of the sentences in section 00 of the PTB were replaced with all of their WordNet synonyms. Verb forms were generated using verb stems, part-of-speech tags, and the <i>morphg </i>tool.<footnote anchor="2"/>These substituted outputs were then filtered using the n-gram data which Google Inc. has made avail-able.<footnote anchor="3"/> Those without any 5-gram matches centered on the substituted word (or 3-gram matches, in the case of short sentences) were eliminated.</p></subsection></section><section number="3" title="Evaluation"><p>From the data sources described in the previous sec­tion, a corpus of realizations to be evaluated by the human judges was constructed by randomly choos­ing 305 sentences from section 00, then selecting surface realizations of these sentences using the fol­lowing algorithm:</p><p>1. Add OpenCCG's best-scored realization.</p><p>2. Add other OpenCCG realizations until all four models are represented, to a maximum of 4.</p><p>3. Add up to 4 realizations from either the XLE system or the WordNet pool, chosen randomly.</p><p>The intent was to give reasonable coverage of all realizer systems discussed in Section 2 without over­loading the human judges. "System" here means any instantiation that emits surface realizations, in­cluding various configurations of OpenCCG (using different language models or ranking systems), and these can be multiple-output, such as an n-best list, or single-output (best-only, worst-only, etc.). Ac­cordingly, more realizations were selected from the OpenCCG realizer because 5 different systems were being represented. Realizations were chosen ran­domly, rather than according to sentence types or other criteria, in order to produce a representative sample of the corpus. In total, 2,114 realizations were selected for evaluation.</p><footnote label="2">http://www.informatics.sussex.ac.uk/ research/groups/nlp/carroll/morph.html</footnote><footnote label="3">http://www.ldc.upenn.edu/Catalog/docs/ LDC2 0 0 6T13/readme.txt</footnote><page local="4" global="567"/><subsection number="3.1" title="Human judgments"><p>Two human judges evaluated each surface realiza­tion on two criteria: <i>adequacy, </i>which represents the extent to which the output conveys all and only the meaning of the reference sentence; and <i>fluency, </i>the extent to which it is grammatically acceptable. The realizations were presented to the judges in sets con­taining a reference sentence and the 1-8 outputs se­lected for that sentence. To aid in the evaluation of adequacy, one sentence each of leading and trailing context were displayed. Judges used the guidelines given in Figure 1, based on the scales developed by the NIST Machine Translation Evaluation Work­shop.</p><p>In addition to rating each realization on the two five-point scales, each judge also repaired each out­put which he or she did not judge to be fully ade­quate and fluent. An example is shown in Figure 2. These repairs resulted in new reference sentences for a substantial number of sentences. These repaired realizations were later used to calculate <i>targeted </i>ver­sions of the evaluation metrics, i.e., using the re­paired sentence as the reference sentence. Although targeted metrics are not fully automatic, they are of interest because they allow the evaluation algorithm to focus on what is actually wrong with the input, rather than all textual differences. Notably, targeted ter (hter) has been shown to be more consistent with human judgments than human annotators are with one another (Snover et al., 2006).</p></subsection><subsection number="3.2" title="Automatic evaluation"><p>The realizations were also evaluated using seven au­tomatic metrics:</p><p>• IBM's bleu, which scores a hypothesis by counting n-gram matches with the reference sentence (Papineni et al., 2001), with smooth­ing as described in (Lin and Och, 2004)</p><p>• The nist n-gram evaluation metric, similar to bleu, but rewarding rarer n-gram matches, and using a different length penalty</p><p>• meteor, which measures the harmonic mean of unigram precision and recall, with a higher weight for recall (Banerjee and Lavie, 2005)</p><p>• ter (Translation Edit Rate), a measure of the number of edits required to transform a hy­pothesis sentence into the reference sentence (Snoveretal.,2006)</p><p>• terp, an augmented version of ter which performs phrasal substitutions, stemming, and checks for synonyms, among other improve­ments (Snover et al., 2009)</p><p>• terpa, an instantiation of terp with edit weights optimized for correlation with ade­quacy in MT evaluations</p><p>• gtm (General Text Matcher), a generaliza­tion of the F-measure that rewards contiguous matching spans (Turian et al., 2003)</p><p>Additionally, targeted versions of bleu, me­teor, ter, and gtm were computed by using the human-repaired outputs as the reference set. The human repair was different from the reference sen­tence in 193 cases (about 9% of the total), and we expected this to result in better scores and correla­tions with the human judgments overall.</p></subsection></section><section number="4" title="Results"><subsection number="4.1" title="Human judgments"><p>Table 1 summarizes the dataset, as well as the mean adequacy and fluency scores garnered from the hu­man evaluation. Overall adequacy and fluency judg­ments were high (4.16, 3.63) for the realizer sys­tems on average, and the best-rated realizer systems achieved mean fluency scores above 4.</p></subsection><subsection number="4.2" title="Inter-annotator agreement"><p>Inter-annotator agreement was measured using the <i>k </i>-coefficient, which is commonly used to measure the extent to which annotators agree in category judgment tasks. k is defined as <i>P^p(ßE</i>, where <i>P(A) </i>is the observed agreement between annota-tors and P(E) is the probability of agreement due to chance (Carletta, 1996). Chance agreement for this data is calculated by the method discussed in Carletta's squib. However, in previous work in MT meta-evaluation, Callison-Burch et al. (2007), assume the less strict criterion of uniform chance agreement, i.e. ^ for a five-point scale. They also <i>Score   Adequacy Fluency</i> <i>Ref.</i><page local="5" global="568"/><i>       </i>It wasn't clear how NL and Mr. Simmons would respond if Georgia Gulf spurns them again <i>Realiz.   </i>It <u>weren't</u> clear how NL and Mr. Simmons would respond if Georgia Gulf again spurns them <i>Repair   </i>It <u>wasn't</u> clear how NL and Mr. Simmons would respond if Georgia Gulf again spurns them introduce the notion of "relative" k, which measures how often two or more judges agreed that A &gt; B, <i>A = B, </i>or A &lt; B for two outputs A and B, irre­spective of the specific values given on the five-point scale; here, uniform chance agreement is taken to be <footnote anchor="1"/>. We report both absolute and relative k in Table 2, using actual chance agreement rather than uniform chance agreement.</p><figure caption="Figure 2: Example of repair"></figure><p>The k scores of 0.60 for adequacy and 0.63 for flu­ency across the entire dataset represent "substantial" agreement, according to the guidelines discussed in (Landis and Koch, 1977), better than is typically reported for machine translation evaluation tasks; for example, Callison-Burch et al. (2007) reported "fair" agreement, with k = 0.281 for fluency and k = 0.307 for adequacy (relative). Assuming the uniform chance agreement that the previously cited work adopts, our inter-annotator agreements (both absolute and relative) are still higher. This is likely due to the generally high quality of the realizations evaluated, leading to easier judgments.</p></subsection><subsection number="4.3" title="Correlation with automatic evaluation"><p>To determine how well the automatic evaluation methods described in Section 3 correlate with the human judgments, we averaged the human judg­ments for adequacy and fluency, respectively, for each of the rated realizations, and then computed both Pearson's correlation coefficient and Spear­man's rank correlation coefficient between these scores and each of the metrics. Spearman's corre­lation makes fewer assumptions about the distribu­tion of the data, but may not reflect a linear relationship that is actually present. Both are frequently reported in the literature. Due to space constraints, we show only Spearman's correlation, although the ter family scored slightly better on Pearson's coef­ficient, relatively.</p><p>The results for Spearman's correlation are given in Table 3. Additionally, the average scores for ad­equacy and fluency were themselves averaged into a single score, following (Snover et al., 2009), and the Spearman's correlation of each of the automatic metrics with these scores are given in Table 4. All reported correlations are significant at <i>p</i><i> </i><i>&lt;</i><i> </i>0.001.</p></subsection><subsection number="4.4" title="Bootstrap sampling of correlations"><p>For each of the sub-corpora shown in Table 1, we computed confidence intervals for the correlations between adequacy and fluency human scores with selected automatic metrics (bleu, hbleu, ter, terp, and hter) as described in (Koenh, 2004). We sampled each sub-corpus 1000 times with replace­ment, and calculated correlations between the rank­ings induced by the human scores and those induced by the metrics for each reference sentence. We then used these coefficients to estimate the confidence in­terval, after excluding the top 25 and bottom 25 co­efficients, following (Lin and Och, 2004). The re­sults of this for the bleu metric are shown in Table 5. We determined which correlations lay within the 95% confidence interval of the best performing met­ric in each row of Table Table 3; these figures are italicized.</p><table caption="Figure 1: Rating scale and guidelines" class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>5</p></td><td class="cell"><p>All the meaning of the reference</p></td><td class="cell"><p>Perfectly grammatical</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>4</p></td><td class="cell"><p>Most of the meaning</p></td><td class="cell"><p>Awkward or non-native; punctuation errors</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3</p></td><td class="cell"><p>Much of the meaning</p></td><td class="cell"><p>Agreement errors or minor syntactic problems</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2</p></td><td class="cell"><p>Meaning substantially different</p></td><td class="cell"><p>Major syntactic problems, such as missing words</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p>Meaning completely different</p></td><td class="cell"><p>Completely ungrammatical</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="6" global="569"/></subsection></section><section number="5" title="Discussion"><subsection number="5.1" title="Human judgments of systems"><p>The results for the four OpenCCG perceptron mod­els mostly confirm those reported in (White and Ra-jkumar, 2009), with one exception: the B-3 model was below B-2, though the P-B (perceptron-best) model still scored highest. This may have been due to differences in the testing scenario. None of the differences in adequacy scores among the individ­ual systems are significant, with the exception of the WordNet system. In this case, the lack of word­sense disambiguation for the substituted words re­sults in a poor overall adequacy score (e.g., <i>wage floor — wage story). </i>Conversely, it scores highest for fluency, as substituting a noun or verb with a syn­onym does not usually introduce ungrammaticality.</p></subsection><subsection number="5.2" title="Correlations of human judgments with MT metrics"><p>Of the non-human-targeted metrics evaluated, bleu and ter/terp demonstrate the highest correla­tions with the human judgments of fluency <i>(r = </i>0.62, 0.64). The ter family of evaluation metrics have been observed to perform very well in MT-evaluation tasks, and although the data evaluated here differs from typical MT data in some impor­tant ways, the correlation of terp with the human judgments is substantial. In contrast with previous MT evaluations where terp performs considerably better than ter, these scored close to equal on our data, possibly because terp's stem, synonym, and paraphrase matching are less useful when most of the variation is syntactic.</p><p>The correlations with bleu and meteor are lower than those reported in (Callison-Burch et al., 2007); in that study, bleu achieved adequacy and fluency correlations of 0.690 and 0.722, respec­tively, and meteor achieved 0.701 and 0.719. The correlations for these metrics might be expected to be lower for our data, since overall quality is higher, making the metrics' task more difficult as the out­puts involve subtler differences between acceptable and unacceptable variation.</p><p>The human-targeted metrics (represented by the prefixed <i>H </i>in the data tables) correlated even more strongly with the human judgments, compared to the non-targeted versions. hter demonstrated the best correlation with realizer fluency (r = 0.75).</p><p>For several kinds of acceptable variation involv­ing the rearrangement of constituents (such as da­tive shift), terp gives a more reasonable score than bleu, due to its ability to directly evaluate phrasal shifts. The following realization was rated 4.5 for fluency, and was more correctly ranked by terp than bleu:</p><p>(3) <i>Ref: </i>The deal also gave Mitsui access to a high-tech medical product.</p><p>(4) <i>Realiz.: </i>The deal also gave access to a high-tech medical product to Mitsui.</p><p>For each reference sentence, we compared the ranking of its realizations induced from the human scores to the ranking induced from the terp score, and counted the rank errors by the latter, infor­mally categorizing them by error type (see Table 7). In the 50 sentences with the highest numbers of rank errors, 17 were affected by punctuation differ­ences, typically involving variation in comma place­ment. Human fluency judgments of outputs with only punctuation problems were generally high, and many realizations with commas inserted or removed were rated fully fluent by the annotators. However, terp penalizes such insertions or deletions. Agree­ment errors are another frequent source of rank­ing errors for terp. The human judges tended to harshly penalize sentences with number-agreement or tense errors, whereas terp applies only a single substitution penalty for each such error. We expect that with suitable optimization of edit weights to avoid over-penalizing punctuation shifts and under-penalizing agreement errors, terp would exhibit an even stronger correlation with human fluency judg­ments.</p><p>None of the evaluation metrics can distinguish an acceptable movement of a word or constituent from an unacceptable movement, with only one reference sentence. A substantial source of error for both terp and bleu is variation in adverbial placement, as shown in (7).</p><p>Similar errors are seen with prepositional phrases and some commonly-occurring temporal adverbs, which typically admit a number of variations in placement. Another important example of accept­able variation which these metrics do not generally rank correctly is dative alternation:</p><page local="7" global="570"/><p><i>Ref. </i>We need to clarify what exactly is wrong with it.</p><p>(5) <i>Ref. </i>When test booklets were passed out 48 hours ahead of time, she says she copied questions in the social studies sec­tion and gave the answers to students.</p><p>(6) <i>Realiz. </i>When test booklets were passed out 48 hours ahead of time , she says she copied questions in the social studies sec­tion and gave students the answers.</p><p>The correlations of each of the metrics with the human judgments of fluency for the realizer systems indicate at least a moderate relationship, in contrast with the results reported in (Stent et al., 2005) for paraphrase data, which found an inverse correlation for fluency, and (Cahill, 2009) for the output of a sur­face realizer for German, which found only a weak correlation. However, the former study employed a corpus-based paraphrase generation system rather than grammar-driven surface realizers, and the re­sulting paraphrases exhibited much broader varia­tion. In Cahill's study, the outputs of the realizer were almost always grammatically correct, and the automated evaluation metrics were ranking marked-ness instead of grammatical acceptability.</p></subsection><subsection number="5.3" title="System-level comparisons"><p>In order to investigate the efficacy of the metrics in ranking different realizer systems, or competing realizations from the same system generated using different ranking models, we considered seven dif­ferent "systems" from the whole dataset of realiza­tions. These consisted of five OpenCCG-based re­alizations (the best realization from three baseline models, and the best and the worst realization from the full perceptron model), and two XLE-based sys­tems (the best and the worst realization, after rank­ing the outputs of the XLE realizer with an n-gram model). The mean of the combined adequacy and fluency scores of each of these seven systems was compared with that of every other system, result­ing in 21 pairwise comparisons. Then Tukey's HSD test was performed to determine the systems which differed significantly in terms of the average ade­quacy and fluency rating they received.<footnote anchor="4"/> The test revealed five pairwise comparisons where the scores were significantly different.</p><p>Subsequently, for each of these systems, an over­all system-level score for each of the MT metrics was calculated. For the five pairwise comparisons where the adequacy-fluency group means differed significantly, we checked whether the metric ranked the systems correctly. Table 8 shows the results of a pairwise comparison between the ranking induced by each evaluation metric, and the ranking induced by the human judgments. Five of the seven non-targeted metrics correctly rank more than half of the systems. nist, meteor, and gtm get the most comparisons right, but neither nist nor gtm cor­rectly rank the OpenCCG-baseline model 1 with re­spect to the XLE-best model. ter and terp get two of the five comparisons correct, and they incorrectly rank two of the five OpenCCG model comparisons, as well as the comparison between the XLE-worst and OpenCCG-best systems.</p><p>For the targeted metrics, hnist is correct for all five comparisons, while neither hb leu nor hme-teor correctly rank all the OpenCCG models. On the other hand, hter and hgtm incorrectly rank the XLE-best system versus OpenCCG-based models.</p><p>In summary, some of the metrics get some of the rankings correct, but none of the non-targeted met­rics get all of them correct. Moreover, different met­rics make different ranking errors. This argues for the use of multiple metrics in comparing realizer systems.<page local="8" global="571"/></p><footnote label="4">This particular test was chosen since it corrects for multiple post-hoc analyses conducted on the same data-set.</footnote><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Realiz.</i></p></td><td class="cell"><p><i>Flu.</i></p></td><td class="cell"><p>terp</p></td><td class="cell"><p>bleu</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>We need to clarify exactly what is wrong with it.</p></td><td class="cell"><p>5</p></td><td class="cell"><p>0.1</p></td><td class="cell"><p>0.5555</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>We need to clarify exactly what's wrong with it.</p></td><td class="cell"><p>5</p></td><td class="cell"><p>0.2</p></td><td class="cell"><p>0.4046</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(7) We need to clarify what , exactly , is wrong with it.</p></td><td class="cell"><p>5</p></td><td class="cell"><p>0.2</p></td><td class="cell"><p>0.5452</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>We need to clarify what is wrong with it exactly.</p></td><td class="cell"><p>4.5</p></td><td class="cell"><p>0.1</p></td><td class="cell"><p>0.6756</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>We need to clarify what exactly , is wrong with it.</p></td><td class="cell"><p>4</p></td><td class="cell"><p>0.1</p></td><td class="cell"><p>0.7017</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>We need to clarify what , exactly is wrong with it.</p></td><td class="cell"><p>4</p></td><td class="cell"><p>0.1</p></td><td class="cell"><p>0.7017</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>We needs to clarify exactly what is wrong with it.</p></td><td class="cell"><p>3</p></td><td class="cell"><p>0.103</p></td><td class="cell"><p>0.346</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></subsection></section><section number="6" title="Conclusion"><p>Our study suggests that although the task of evalu­ating the output from realizer systems differs from the task of evaluating machine translations, the au­tomatic metrics used to evaluate MT outputs deliver moderate correlations with combined human fluency and adequacy scores when used on surface realiza­tions. We also found that the MT-evaluation met­rics are useful in evaluating different versions of the same realizer system (e.g., the various OpenCCG re­alization ranking models), and finding cases where a system is performing poorly. As in MT-evaluation tasks, human-targeted metrics have the highest cor­relations with human judgments overall. These re­sults suggest that the MT-evaluation metrics are use­ful for developing surface realizers. However, the correlations are lower than those reported for MT data, suggesting that they should be used with cau­tion, especially for cross-system evaluation, where consulting multiple metrics may yield more reliable comparisons. In our study, the targeted version of TERP correlated most strongly with human judg­ments of fluency.</p><p>In future work, the performance of the TER family of metrics on this data might be improved by opti­mizing the edit weights used in computing its scores, so as to avoid over-penalizing punctuation move­ments or under-penalizing agreement errors, both of which were significant sources of ranking errors. Multiple reference sentences may also help mitigate these problems, and the corpus of human-repaired realizations that has resulted from our study is a step in this direction, as it provides multiple references for some cases. We expect the corpus to also prove useful for feature engineering and error analysis in developing better realization models.<footnote anchor="5"/></p></section><section title="Acknowledgements"><p>We thank Aoife Cahill and Tracy King for providing us with the output of the XLE generator. We also thank Chris Callison-Burch and the anonymous re­viewers for their helpful comments and suggestions.</p><footnote label="5">The corpus can be downloaded from http://www . ling.ohio- state.edu/ ~mwhite/data/emnlp10/.</footnote><p>This material is based upon work supported by the National Science Foundation under Grant No. 0812297.</p><table caption="Table 1: Descriptive statistics" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Type</p></td><td class="cell"><p>System</p></td><td class="cell"><p>#Refs</p></td><td class="cell"><p>#Paraphrases</p></td><td class="cell"><p>Average Paraphrases/Ref</p></td><td class="cell"><p>#Exact Matches</p></td><td class="cell"><p>Adq</p></td><td class="cell"><p>Flu</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Single output</p></td><td class="cell"><p>OpenCCG Baseline 1</p></td><td class="cell"><p>296</p></td><td class="cell"><p>296</p></td><td class="cell"><p>1.0</p></td><td class="cell"><p>72</p></td><td class="cell"><p>4.17</p></td><td class="cell"><p>3.65</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>OpenCCG Baseline 2</p></td><td class="cell"><p>296</p></td><td class="cell"><p>296</p></td><td class="cell"><p>1.0</p></td><td class="cell"><p>82</p></td><td class="cell"><p>4.34</p></td><td class="cell"><p>3.94</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>OpenCCG Baseline 3</p></td><td class="cell"><p>296</p></td><td class="cell"><p>296</p></td><td class="cell"><p>1.0</p></td><td class="cell"><p>76</p></td><td class="cell"><p>4.31</p></td><td class="cell"><p>3.86</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>OpenCCG Perceptron Best</p></td><td class="cell"><p>296</p></td><td class="cell"><p>1.0</p></td><td class="cell"><p>1.0</p></td><td class="cell"><p>112</p></td><td class="cell"><p>4.37</p></td><td class="cell"><p>4.09</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>OpenCCG Perceptron Worst</p></td><td class="cell"><p>117</p></td><td class="cell"><p>117</p></td><td class="cell"><p>1.0</p></td><td class="cell"><p>5</p></td><td class="cell"><p>4.34</p></td><td class="cell"><p>3.36</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>XLE Best</p></td><td class="cell"><p>154</p></td><td class="cell"><p>154</p></td><td class="cell"><p>1.0</p></td><td class="cell"><p>24</p></td><td class="cell"><p>4.41</p></td><td class="cell"><p>4.07</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>XLE Worst</p></td><td class="cell"><p>157</p></td><td class="cell"><p>157</p></td><td class="cell"><p>1.0</p></td><td class="cell"><p>13</p></td><td class="cell"><p>4.08</p></td><td class="cell"><p>3.73</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Multiple output</p></td><td class="cell"><p>OpenCCG-Perceptron All</p></td><td class="cell"><p>296</p></td><td class="cell"><p>767</p></td><td class="cell"><p>2.6</p></td><td class="cell"><p>158</p></td><td class="cell"><p>4.45</p></td><td class="cell"><p>3.91</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>OpenCCG All</p></td><td class="cell"><p>296</p></td><td class="cell"><p>1131</p></td><td class="cell"><p>3.8</p></td><td class="cell"><p>162</p></td><td class="cell"><p>4.20</p></td><td class="cell"><p>3.61</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>XLE All</p></td><td class="cell"><p>174</p></td><td class="cell"><p>557</p></td><td class="cell"><p>3.2</p></td><td class="cell"><p>54</p></td><td class="cell"><p>4.17</p></td><td class="cell"><p>3.81</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Wordnet Subsitutions</p></td><td class="cell"><p>162</p></td><td class="cell"><p>486</p></td><td class="cell"><p>3.0</p></td><td class="cell"><p>0</p></td><td class="cell"><p>3.66</p></td><td class="cell"><p>4.71</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Realizer All</p></td><td class="cell"><p>296</p></td><td class="cell"><p>1628</p></td><td class="cell"><p>5.0</p></td><td class="cell"><p>169</p></td><td class="cell"><p>4.16</p></td><td class="cell"><p>3.63</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>All</p></td><td class="cell"><p>296</p></td><td class="cell"><p>2114</p></td><td class="cell"><p>7.1</p></td><td class="cell"><p>169</p></td><td class="cell"><p>4.05</p></td><td class="cell"><p>3.88</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 2: Corpora-wise inter-annotator agreement (absolute and relativekvalues shown)" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>System</p></td><td class="cell"><p>Adq</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>Flu</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>p(A)</p></td><td class="cell"><p>p(E)</p></td><td class="cell"><p>K</p></td><td class="cell"><p>p(A)</p></td><td class="cell"><p>p(E)</p></td><td class="cell"><p>K</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>OpenCCG-Abs</p></td><td class="cell"><p>0.73</p></td><td class="cell"><p>0.47</p></td><td class="cell"><p>0.48</p></td><td class="cell"><p>0.70</p></td><td class="cell"><p>0.24</p></td><td class="cell"><p>0.61</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>OpenCCG-Rel</p></td><td class="cell"><p>0.76</p></td><td class="cell"><p>0.47</p></td><td class="cell"><p>0.54</p></td><td class="cell"><p>0.76</p></td><td class="cell"><p>0.34</p></td><td class="cell"><p>0.64</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>XLE-Abs</p></td><td class="cell"><p>0.68</p></td><td class="cell"><p>0.42</p></td><td class="cell"><p>0.44</p></td><td class="cell"><p>0.69</p></td><td class="cell"><p>0.27</p></td><td class="cell"><p>0.58</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>XLE-Rel</p></td><td class="cell"><p>0.73</p></td><td class="cell"><p>0.45</p></td><td class="cell"><p>0.50</p></td><td class="cell"><p>0.69</p></td><td class="cell"><p>0.37</p></td><td class="cell"><p>0.50</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Wordnet-Abs</p></td><td class="cell"><p>0.57</p></td><td class="cell"><p>0.25</p></td><td class="cell"><p>0.43</p></td><td class="cell"><p>0.77</p></td><td class="cell"><p>0.66</p></td><td class="cell"><p>0.33</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Wordnet-Rel</p></td><td class="cell"><p>0.74</p></td><td class="cell"><p>0.34</p></td><td class="cell"><p>0.61</p></td><td class="cell"><p>0.73</p></td><td class="cell"><p>0.60</p></td><td class="cell"><p>0.33</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Realizer-Abs</p></td><td class="cell"><p>0.70</p></td><td class="cell"><p>0.44</p></td><td class="cell"><p>0.47</p></td><td class="cell"><p>0.69</p></td><td class="cell"><p>0.24</p></td><td class="cell"><p>0.59</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Realizer-Rel</p></td><td class="cell"><p>0.74</p></td><td class="cell"><p>0.41</p></td><td class="cell"><p>0.56</p></td><td class="cell"><p>0.73</p></td><td class="cell"><p>0.33</p></td><td class="cell"><p>0.60</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>All-Abs</p></td><td class="cell"><p>0.67</p></td><td class="cell"><p>0.38</p></td><td class="cell"><p>0.47</p></td><td class="cell"><p>0.71</p></td><td class="cell"><p>0.29</p></td><td class="cell"><p>0.59</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>All-Rel</p></td><td class="cell"><p>0.74</p></td><td class="cell"><p>0.36</p></td><td class="cell"><p><b>0.60</b></p></td><td class="cell"><p>0.75</p></td><td class="cell"><p>0.34</p></td><td class="cell"><p><b>0.63</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Sys</p></td><td class="cell"><p>N</p></td><td class="cell"><p>B</p></td><td class="cell"><p>M</p></td><td class="cell"><p>G</p></td><td class="cell"><p>TP</p></td><td class="cell"><p>TA</p></td><td class="cell"><p>T</p></td><td class="cell"><p>HT</p></td><td class="cell"><p>HN</p></td><td class="cell"><p>HB</p></td><td class="cell"><p>HM</p></td><td class="cell"><p>HG</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>OpenCCG-Adq</p></td><td class="cell"><p>0.27</p></td><td class="cell"><p><i>0.39</i></p></td><td class="cell"><p>0.35</p></td><td class="cell"><p>0.18</p></td><td class="cell"><p><i>0.39</i></p></td><td class="cell"><p>0.34</p></td><td class="cell"><p><b>0.4</b></p></td><td class="cell"><p><b>0.43</b></p></td><td class="cell"><p>0.3</p></td><td class="cell"><p><b>0.43</b></p></td><td class="cell"><p><b>0.43</b></p></td><td class="cell"><p>0.23</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>OpenCCG-Flu</p></td><td class="cell"><p>0.49</p></td><td class="cell"><p>0.55</p></td><td class="cell"><p>0.4</p></td><td class="cell"><p>0.42</p></td><td class="cell"><p><b>0.6</b></p></td><td class="cell"><p>0.46</p></td><td class="cell"><p><b>0.6</b></p></td><td class="cell"><p><b>0.72</b></p></td><td class="cell"><p>0.58</p></td><td class="cell"><p>0.69</p></td><td class="cell"><p>0.57</p></td><td class="cell"><p>0.53</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>XLE-Adq</p></td><td class="cell"><p><i>0.52</i></p></td><td class="cell"><p><i>0.51</i></p></td><td class="cell"><p><b>0.55</b></p></td><td class="cell"><p>0.31</p></td><td class="cell"><p><i>0.5</i></p></td><td class="cell"><p><i>0.5</i></p></td><td class="cell"><p><i>0.5</i></p></td><td class="cell"><p>0.52</p></td><td class="cell"><p>0.47</p></td><td class="cell"><p>0.51</p></td><td class="cell"><p><b>0.61</b></p></td><td class="cell"><p>0.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>XLE-Flu</p></td><td class="cell"><p><b>0.56</b></p></td><td class="cell"><p><b>0.56</b></p></td><td class="cell"><p>0.48</p></td><td class="cell"><p>0.37</p></td><td class="cell"><p><i>0.55</i></p></td><td class="cell"><p>0.5</p></td><td class="cell"><p><i>0.55</i></p></td><td class="cell"><p><b>0.61</b></p></td><td class="cell"><p>0.54</p></td><td class="cell"><p><b>0.61</b></p></td><td class="cell"><p>0.51</p></td><td class="cell"><p>0.51</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Wordnet-Adq</p></td><td class="cell"><p>0.17</p></td><td class="cell"><p>0.14</p></td><td class="cell"><p>0.24</p></td><td class="cell"><p>0.15</p></td><td class="cell"><p><b>0.37</b></p></td><td class="cell"><p>0.26</p></td><td class="cell"><p>0.22</p></td><td class="cell"><p><b>0.64</b></p></td><td class="cell"><p>0.52</p></td><td class="cell"><p>0.56</p></td><td class="cell"><p>0.32</p></td><td class="cell"><p>0.6</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Wordnet-Flu</p></td><td class="cell"><p>0.26</p></td><td class="cell"><p>0.21</p></td><td class="cell"><p>0.24</p></td><td class="cell"><p>0.24</p></td><td class="cell"><p>0.22</p></td><td class="cell"><p><b>0.27</b></p></td><td class="cell"><p>0.26</p></td><td class="cell"><p><b>0.34</b></p></td><td class="cell"><p>0.32</p></td><td class="cell"><p><b>0.34</b></p></td><td class="cell"><p>0.3</p></td><td class="cell"><p><b>0.34</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Realizer-Adq</p></td><td class="cell"><p>0.47</p></td><td class="cell"><p><b>0.6</b></p></td><td class="cell"><p><i>0.57</i></p></td><td class="cell"><p>0.42</p></td><td class="cell"><p><i>0.59</i></p></td><td class="cell"><p><i>0.57</i></p></td><td class="cell"><p><b>0.6</b></p></td><td class="cell"><p><i>0.62</i></p></td><td class="cell"><p>0.49</p></td><td class="cell"><p><i>0.62</i></p></td><td class="cell"><p><b>0.65</b></p></td><td class="cell"><p>0.48</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Realizer-Flu</p></td><td class="cell"><p>0.51</p></td><td class="cell"><p><i>0.62</i></p></td><td class="cell"><p>0.52</p></td><td class="cell"><p>0.5</p></td><td class="cell"><p><i>0.63</i></p></td><td class="cell"><p>0.53</p></td><td class="cell"><p><b>0.64</b></p></td><td class="cell"><p><b>0.75</b></p></td><td class="cell"><p>0.59</p></td><td class="cell"><p><i>0.73</i></p></td><td class="cell"><p>0.65</p></td><td class="cell"><p>0.63</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>All-Adq</p></td><td class="cell"><p>0.37</p></td><td class="cell"><p>0.37</p></td><td class="cell"><p>0.33</p></td><td class="cell"><p>0.32</p></td><td class="cell"><p><i>0.42</i></p></td><td class="cell"><p>0.31</p></td><td class="cell"><p><b>0.43</b></p></td><td class="cell"><p><b>0.53</b></p></td><td class="cell"><p>0.44</p></td><td class="cell"><p>0.48</p></td><td class="cell"><p>0.44</p></td><td class="cell"><p>0.45</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>All-Flu</p></td><td class="cell"><p>0.21</p></td><td class="cell"><p><b>0.62</b></p></td><td class="cell"><p>0.51</p></td><td class="cell"><p>0.32</p></td><td class="cell"><p><i>0.61</i></p></td><td class="cell"><p>0.55</p></td><td class="cell"><p><i>0.6</i></p></td><td class="cell"><p><i>0.7</i></p></td><td class="cell"><p>0.33</p></td><td class="cell"><p><b>0.71</b></p></td><td class="cell"><p>0.62</p></td><td class="cell"><p>0.48</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Sys</p></td><td class="cell"><p>N</p></td><td class="cell"><p>B</p></td><td class="cell"><p>M</p></td><td class="cell"><p>G</p></td><td class="cell"><p>TP</p></td><td class="cell"><p>TA</p></td><td class="cell"><p>T</p></td><td class="cell"><p>HT</p></td><td class="cell"><p>HN</p></td><td class="cell"><p>HB</p></td><td class="cell"><p>HM</p></td><td class="cell"><p>HG</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>OpenCCG</p></td><td class="cell"><p>0.49</p></td><td class="cell"><p>0.57</p></td><td class="cell"><p>0.42</p></td><td class="cell"><p>0.4</p></td><td class="cell"><p>0.61</p></td><td class="cell"><p>0.46</p></td><td class="cell"><p><b>0.62</b></p></td><td class="cell"><p><b>0.73</b></p></td><td class="cell"><p>0.58</p></td><td class="cell"><p>0.7</p></td><td class="cell"><p>0.59</p></td><td class="cell"><p>0.51</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>XLE</p></td><td class="cell"><p>0.63</p></td><td class="cell"><p><b>0.64</b></p></td><td class="cell"><p>0.59</p></td><td class="cell"><p>0.39</p></td><td class="cell"><p>0.62</p></td><td class="cell"><p>0.58</p></td><td class="cell"><p>0.63</p></td><td class="cell"><p><b>0.69</b></p></td><td class="cell"><p>0.6</p></td><td class="cell"><p>0.68</p></td><td class="cell"><p>0.63</p></td><td class="cell"><p>0.54</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Wordnet</p></td><td class="cell"><p>0.21</p></td><td class="cell"><p>0.14</p></td><td class="cell"><p>0.21</p></td><td class="cell"><p>0.19</p></td><td class="cell"><p><b>0.38</b></p></td><td class="cell"><p>0.25</p></td><td class="cell"><p>0.23</p></td><td class="cell"><p><b>0.65</b></p></td><td class="cell"><p>0.56</p></td><td class="cell"><p>0.57</p></td><td class="cell"><p>0.31</p></td><td class="cell"><p>0.63</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Realizer</p></td><td class="cell"><p>0.55</p></td><td class="cell"><p>0.68</p></td><td class="cell"><p>0.57</p></td><td class="cell"><p>0.5</p></td><td class="cell"><p>0.68</p></td><td class="cell"><p>0.58</p></td><td class="cell"><p><b>0.69</b></p></td><td class="cell"><p><b>0.78</b></p></td><td class="cell"><p>0.61</p></td><td class="cell"><p>0.77</p></td><td class="cell"><p>0.7</p></td><td class="cell"><p>0.63</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>All</p></td><td class="cell"><p>0.34</p></td><td class="cell"><p>0.58</p></td><td class="cell"><p>0.47</p></td><td class="cell"><p>0.38</p></td><td class="cell"><p><b>0.61</b></p></td><td class="cell"><p>0.48</p></td><td class="cell"><p><b>0.61</b></p></td><td class="cell"><p><b>0.75</b></p></td><td class="cell"><p>0.48</p></td><td class="cell"><p>0.73</p></td><td class="cell"><p>0.61</p></td><td class="cell"><p>0.58</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>System</p></td><td class="cell"><p>Adq</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>Flu</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Sp</p></td><td class="cell"><p>95%L</p></td><td class="cell"><p>95%U</p></td><td class="cell"><p>Sp</p></td><td class="cell"><p>95%L</p></td><td class="cell"><p>95%U</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Realizer</p></td><td class="cell"><p>0.60</p></td><td class="cell"><p>0.58</p></td><td class="cell"><p>0.63</p></td><td class="cell"><p>0.62</p></td><td class="cell"><p>0.59</p></td><td class="cell"><p>0.65</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>XLE</p></td><td class="cell"><p>0.51</p></td><td class="cell"><p>0.47</p></td><td class="cell"><p>0.56</p></td><td class="cell"><p>0.56</p></td><td class="cell"><p>0.51</p></td><td class="cell"><p>0.61</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>OpenCCG</p></td><td class="cell"><p>0.39</p></td><td class="cell"><p>0.35</p></td><td class="cell"><p>0.42</p></td><td class="cell"><p>0.55</p></td><td class="cell"><p>0.52</p></td><td class="cell"><p>0.59</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>All</p></td><td class="cell"><p>0.37</p></td><td class="cell"><p>0.34</p></td><td class="cell"><p>0.4</p></td><td class="cell"><p>0.62</p></td><td class="cell"><p>0.6</p></td><td class="cell"><p>0.64</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Wordnet</p></td><td class="cell"><p>0.14</p></td><td class="cell"><p>0.06</p></td><td class="cell"><p>0.21</p></td><td class="cell"><p>0.21</p></td><td class="cell"><p>0.13</p></td><td class="cell"><p>0.28</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Sys</p></td><td class="cell"><p>HJ</p></td><td class="cell"><p>N</p></td><td class="cell"><p>B</p></td><td class="cell"><p>M</p></td><td class="cell"><p>G</p></td><td class="cell"><p>TP</p></td><td class="cell"><p>TA</p></td><td class="cell"><p>T</p></td><td class="cell"><p>HT</p></td><td class="cell"><p>HN</p></td><td class="cell"><p>HB</p></td><td class="cell"><p>HM</p></td><td class="cell"><p>HG</p></td><td class="cell"><p>HJ1-HJ2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>OpenCCG</p></td><td class="cell"><p>HJ-1</p></td><td class="cell"><p>0.44</p></td><td class="cell"><p>0.52</p></td><td class="cell"><p>0.39</p></td><td class="cell"><p>0.36</p></td><td class="cell"><p>0.56</p></td><td class="cell"><p>0.43</p></td><td class="cell"><p><b>0.58</b></p></td><td class="cell"><p><b>0.75</b></p></td><td class="cell"><p>0.58</p></td><td class="cell"><p>0.72</p></td><td class="cell"><p>0.62</p></td><td class="cell"><p>0.52</p></td><td class="cell"><p>0.76</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>HJ-2</p></td><td class="cell"><p>0.5</p></td><td class="cell"><p>0.58</p></td><td class="cell"><p>0.43</p></td><td class="cell"><p>0.4</p></td><td class="cell"><p>0.62</p></td><td class="cell"><p>0.46</p></td><td class="cell"><p><b>0.63</b></p></td><td class="cell"><p><b>0.7</b></p></td><td class="cell"><p>0.55</p></td><td class="cell"><p>0.68</p></td><td class="cell"><p>0.56</p></td><td class="cell"><p>0.49</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>XLE</p></td><td class="cell"><p>HJ-1</p></td><td class="cell"><p><b>0.6</b></p></td><td class="cell"><p><b>0.6</b></p></td><td class="cell"><p>0.55</p></td><td class="cell"><p>0.37</p></td><td class="cell"><p>0.57</p></td><td class="cell"><p>0.55</p></td><td class="cell"><p>0.58</p></td><td class="cell"><p><b>0.69</b></p></td><td class="cell"><p>0.63</p></td><td class="cell"><p>0.68</p></td><td class="cell"><p>0.64</p></td><td class="cell"><p>0.54</p></td><td class="cell"><p>0.75</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>HJ-2</p></td><td class="cell"><p>0.6</p></td><td class="cell"><p>0.6</p></td><td class="cell"><p>0.56</p></td><td class="cell"><p>0.39</p></td><td class="cell"><p>0.6</p></td><td class="cell"><p>0.55</p></td><td class="cell"><p><b>0.61</b></p></td><td class="cell"><p><b>0.64</b></p></td><td class="cell"><p>0.54</p></td><td class="cell"><p>0.61</p></td><td class="cell"><p>0.57</p></td><td class="cell"><p>0.51</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Wordnet</p></td><td class="cell"><p>HJ-1</p></td><td class="cell"><p>0.2</p></td><td class="cell"><p>0.18</p></td><td class="cell"><p>0.26</p></td><td class="cell"><p>0.16</p></td><td class="cell"><p><b>0.37</b></p></td><td class="cell"><p>0.28</p></td><td class="cell"><p>0.24</p></td><td class="cell"><p><b>0.7</b></p></td><td class="cell"><p>0.59</p></td><td class="cell"><p>0.64</p></td><td class="cell"><p>0.35</p></td><td class="cell"><p>0.65</p></td><td class="cell"><p>0.72</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>HJ-2</p></td><td class="cell"><p>0.25</p></td><td class="cell"><p>0.16</p></td><td class="cell"><p>0.23</p></td><td class="cell"><p>0.19</p></td><td class="cell"><p><b>0.37</b></p></td><td class="cell"><p>0.25</p></td><td class="cell"><p>0.25</p></td><td class="cell"><p><b>0.59</b></p></td><td class="cell"><p>0.52</p></td><td class="cell"><p>0.51</p></td><td class="cell"><p>0.32</p></td><td class="cell"><p>0.56</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Realizer</p></td><td class="cell"><p>HJ-1</p></td><td class="cell"><p>0.51</p></td><td class="cell"><p>0.65</p></td><td class="cell"><p>0.56</p></td><td class="cell"><p>0.49</p></td><td class="cell"><p>0.64</p></td><td class="cell"><p>0.56</p></td><td class="cell"><p><b>0.66</b></p></td><td class="cell"><p><b>0.8</b></p></td><td class="cell"><p>0.62</p></td><td class="cell"><p>0.78</p></td><td class="cell"><p>0.72</p></td><td class="cell"><p>0.64</p></td><td class="cell"><p>0.82</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>HJ-2</p></td><td class="cell"><p>0.55</p></td><td class="cell"><p><b>0.68</b></p></td><td class="cell"><p>0.56</p></td><td class="cell"><p>0.5</p></td><td class="cell"><p>0.67</p></td><td class="cell"><p>0.57</p></td><td class="cell"><p><b>0.68</b></p></td><td class="cell"><p><b>0.74</b></p></td><td class="cell"><p>0.58</p></td><td class="cell"><p>0.73</p></td><td class="cell"><p>0.66</p></td><td class="cell"><p>0.6</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>All</p></td><td class="cell"><p>HJ-1</p></td><td class="cell"><p>0.32</p></td><td class="cell"><p>0.53</p></td><td class="cell"><p>0.45</p></td><td class="cell"><p>0.37</p></td><td class="cell"><p><b>0.57</b></p></td><td class="cell"><p>0.44</p></td><td class="cell"><p><b>0.57</b></p></td><td class="cell"><p><b>0.77</b></p></td><td class="cell"><p>0.5</p></td><td class="cell"><p>0.74</p></td><td class="cell"><p>0.62</p></td><td class="cell"><p>0.59</p></td><td class="cell"><p>0.79</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>HJ-2</p></td><td class="cell"><p>0.35</p></td><td class="cell"><p>0.58</p></td><td class="cell"><p>0.46</p></td><td class="cell"><p>0.37</p></td><td class="cell"><p><b>0.61</b></p></td><td class="cell"><p>0.47</p></td><td class="cell"><p>0.6</p></td><td class="cell"><p><b>0.71</b></p></td><td class="cell"><p>0.44</p></td><td class="cell"><p>0.69</p></td><td class="cell"><p>0.57</p></td><td class="cell"><p>0.54</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 7: Factors influencingTERPranking errors for 50 worst-ranked realization groups" class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Factor</i></p></td><td class="cell"><p><i>Count</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Punctuation</p></td><td class="cell"><p>17</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Adverbial shift</p></td><td class="cell"><p>16</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Agreement</p></td><td class="cell"><p>14</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Other shifts</p></td><td class="cell"><p>8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Conjunct rearrangement</p></td><td class="cell"><p>8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Complementizer ins/del</p></td><td class="cell"><p>5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PP shift</p></td><td class="cell"><p>4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Metric</p></td><td class="cell"><p>Score</p></td><td class="cell"><p>Errors</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>nist</p></td><td class="cell"><p>4</p></td><td class="cell"><p>C1-XB</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>bleu</p></td><td class="cell"><p>3</p></td><td class="cell"><p>XB-PW C1-XB</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>meteor</p></td><td class="cell"><p>4</p></td><td class="cell"><p>XW-PB</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ter</p></td><td class="cell"><p>2</p></td><td class="cell"><p>PW-PB XW-PB C1-PB</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>terp</p></td><td class="cell"><p>2</p></td><td class="cell"><p>PW-PB XW-PB C1-PB</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>terpa</p></td><td class="cell"><p>3</p></td><td class="cell"><p>XW-PB C1-PB</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>gtm</p></td><td class="cell"><p>4</p></td><td class="cell"><p>C1-XB</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>hnist</p></td><td class="cell"><p>5</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>hbleu</p></td><td class="cell"><p>3</p></td><td class="cell"><p>PW-PB XW-PB</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>hmeteor</p></td><td class="cell"><p>2</p></td><td class="cell"><p>PW-PB XW-PB C1-PB</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>hter</p></td><td class="cell"><p>3</p></td><td class="cell"><p>XB-PW C1-XB</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>hgtm</p></td><td class="cell"><p>3</p></td><td class="cell"><p>XB-PW C1-XB</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><references><p>Jason Baldridge. 2002. <i>Lexically Specified Derivational Control in Combinatory Categorial Grammar. </i>Ph.D. thesis, University of Edinburgh.</p><p>S. Banerjee and A. Lavie. 2005. METEOR: An auto­matic metric for MT evaluation with improved corre­lation with human judgments. In <i>Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summariza­tion, </i>pages 65-72.</p><p>R. Barzilay and L. Lee. 2003. Learning to paraphrase: An unsupervised approach using multiple-sequence alignment.  In <i>proceedings ofHLT-NAACL, </i>volume 2003, pages 16-23.</p><p>Aoife Cahill. 2009. Correlating human and automatic evaluation of a german surface realiser. In <i>Proceed­ings of the ACL-IJCNLP 2009 Conference Short Pa­pers, </i>pages 97-100, Suntec, Singapore, August. Asso­ciation for Computational Linguistics.</p><p>C. Callison-Burch, M. Osborne, and P. Koehn. 2006. Re-evaluating the role of BLEU in machine translation re­search. In <i>Proceedings ofEACL, </i>volume 2006, pages 249-256.</p><p>Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (meta-) evaluation of machine translation. In <i>StatMT '07: Pro­ceedings ofthe Second Workshop on Statistical Ma­chine Translation, </i>pages 136-158, Morristown, NJ, USA. Association for Computational Linguistics.</p><p>C. Callison-Burch, T. Cohn, and M. Lapata. 2008. Para­metric: An automatic evaluation metric for paraphras­ing. In <i>Proceedings ofthe 22nd International Con­ference on Computational Linguistics-Volume 1, </i>pages 97-104. Association for Computational Linguistics.</p><p>J. Carletta. 1996. Assessing agreement on classification tasks: the kappa statistic. <i>Computational linguistics, </i>22(2):249-254.</p><p>Dick Crouch, Mary Dalrymple, Ron Kaplan, Tracy King, John Maxwell, and Paula Newman. 2008. Xle docu­mentation. Technical report, Palo Alto Research Cen­ter.</p><p>Philip Koenh. 2004. Statistical significance tests for ma­chine translation evaluation. In <i>Proceedings </i><i>ofthe</i><i> 2009 Conference on Empirical Methods in Natural Language Processing.</i></p><p>J.R. Landis and G.G. Koch. 1977. The measurement of observer agreement for categorical data. <i>Biometrics, </i>33(1):159-174.</p><page local="9" global="572"/><p>Chin-Yew Lin and Franz Josef Och. 2004. Orange: a method for evaluating automatic evaluation metrics for machine translation. In <i>COLING '04: Proceedings ofthe 20th international conference on Computational Linguistics, </i>page 501, Morristown, NJ, USA. Associ­ation for Computational Linguistics.</p><p>K. Papineni, S. Roukos, T. Ward, and W. Zhu. 2001. Bleu: a method for automatic evaluation of machine translation. Technical report, IBM Research.</p><p>E. Reiter and A. Belz. 2009. An investigation into the validity of some metrics for automatically evaluating natural language generation systems. <i>Computational Linguistics, </i>35(4):529-558.</p><p>Stefan Riezler, Tracy H. King, Ronald M. Kaplan, Richard Crouch, John T. III Maxwell, and Mark John­son. 2002. Parsing the wall street journal using a lexical-functional grammar and discriminative esti­mation techniques. In <i>Proceedings of40th Annual Meeting ofthe Association for Computational Lin­guistics, </i>pages 271-278, Philadelphia, Pennsylvania, USA, July. Association for Computational Linguistics.</p><p>Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of trans­lation edit rate with targeted human annotation. In <i>In Proceedings ofAssociation for Machine Translation in the Americas, </i>pages 223-231.</p><p>M. Snover, N. Madnani, B.J. Dorr, and R. Schwartz. 2009. Fluency, adequacy, or HTER?: exploring dif­ferent human judgments with a tunable MT metric. In <i>Proceedings ofthe Fourth Workshop on Statistical Machine Translation, </i>pages 259-268. Association for Computational Linguistics.</p><p>Amanda Stent, Matthew Marge, and Mohit Singhai. 2005. Evaluating evaluation methods for generation in the presence of variation. In <i>Proceedings ofCICLing.</i></p><p>J.P. Turian, L. Shen, and I.D. Melamed. 2003. Evalua­tion of machine translation and its evaluation. <i>recall (C—R), </i>100:2.</p><p>Michael White and Rajakrishnan Rajkumar. 2009. Per-ceptron reranking for CCG realization. In <i>Proceedings ofthe 2009 Conference on Empirical Methods in Nat­ural Language Processing, </i>pages 410-419, Singapore, August. Association for Computational Linguistics.</p><p>Michael White. 2006. Efficient Realization of Coordi­nate Structures in Combinatory Categorial Grammar. <i>Research on Language and Computation, </i>4(1):39-75.</p><page local="10" global="573"/><p>Table 3: Spearman's correlations among NIST (N), BLEU (B), METEOR (M), GTM (G), TERp (TP), TERpa (TA), TER (T), human variants (HN, HB, HM, HT, HG) and human judgments (-Adq: adequacy and -Flu: Fluency); Scores which fall within the 95 %CI of the best are italicized.</p><p>Table 4: Spearman's correlations among NIST (N), BLEU (B), METEOR (M), GTM (G), TERp (TP), TERpa (TA), TER (T), human variants (HN, HB, HM, HT, HG) and human judgments (combined adequacy and fluency scores)<page local="11" global="574"/></p><p>Table 5: Spearman's correlation analysis (bootstrap sampling) of the BLEU scores of various systems with human adequacy and fluency scores</p><p>Table 6: Spearman's correlations of NIST (N), BLEU (B), METEOR (M), GTM (G), TERp (TP), TERpa (TA), human variants (HT, HN, HB, HM, HG), and individual human judgments (combined adq. and flu. scores)</p><p>Table 8: Metric-wise ranking performance in terms of agreement with a ranking induced by combined adequacy and fluency scores; each metric gets a score out of 5 (i.e. number of system-level comparisons that emerged significant as per the Tukey's HSD test)</p><doubt alpha="65.4" length="127" tooSmall="False" monospace="0.0">Legend: Perceptron Best (PB); Perceptron Worst (PW); XLE Best (XB); XLE Worst (XW); OpenCCG baseline mod­els 1 to 3 (C1 ... C3)</doubt></references></body></article>