<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="215"/><title>Multi-document Summarisation and the PASCAL Textual Entailment Challenge</title><pubinfo>Proceedings of the Australasian Language Technology Workshop 2005,pages 215-223, Sydney, Australia, December 2005.</pubinfo><author surname="Stokes" givenname="Nicola"><org  name="University of Melbourne" country="Australia" city="Melbourne"/></author><author surname="Newman" givenname="Eamonn"><org  name="University College Dublin" country="Ireland" city="Dublin"/></author></firstpageheader><frontmatter><p><b>Multi-document Summarisation and the PASCAL Textual</b></p><p><b>Entailment Challenge</b></p><p><b>Nicola Stokes</b></p><p>NICTA Victoria Laboratory, Department of Computer Science and Software Engineering, University of Melbourne. nicola.stokes@nicta.com.au</p><p><b>Eamonn Newman</b></p><p>School of Computer Science</p><p>and Informatics, University College Dublin, Ireland. eamonn.newman@ucd.ie</p></frontmatter><abstract>A fundamental problem for systems that re­quire natural language understanding capabili­ties is the identification of instances of semantic equivalence and paraphrase in text. The PAS­CAL Recognising Textual Entailment (RTE) challenge is a recently proposed research ini­tiative that addressed this problem by provid­ing an evaluation framework for the develop­ment of generic "semantic engines" that can be used to identify language variability in a variety of applications such as Information Retrieval, Machine Translation and Question Answering. This paper discusses the suitability of the RTE evaluation datasets as a framework for evaluat­ing the problem of redundancy recognition in multi-document summarisation, i.e. the iden­tification of repetitive information across docu­ments. This paper also reports on the develop­ment of an additional dataset containing exam­ples of informationally equivalent sentence pairs that are typically found in machine generated summaries. The performance of a competitive entailment recognition system on this dataset is also reported. </abstract></header><body><section number="1" title="Introduction"><p>The aim of multi-document summarisation (MDS) is to generate a concise and coherent summary given a cluster of related documents. Although this process is a natural extension of single-document summarisation, MDS poses a number of unique challenges such as, how to manage contradictory and repetitive informa­tion in the cluster, and how to order extracted information in the resultant summary. A pop­ular approach to the MDS problem is to first identify and cluster repetitive information units across documents, then select representative sentences from the "dominant" clusters, and finally generate an extractive summary from these sentences. This approach assumes, like many others in text summarisation, that the repetition of information is an indication of in­formation importance and consequently sum­mary relevancy. The simplest method for de­termining commonality across documents is to group text units (e.g. sentences, paragraphs) that exhibit a high concentration of word over­lap. However, this approximate method for recognising similar semantic content is often in­sufficient due to instances of language variabil­ity such as paraphrase and synonymy. Figure 1 shows two sentences (A and B) that are seman-tically equivalent but syntactically different.</p><p>Text A: Agassi's dream run is ended by world's number one player.</p><p>Text B: Federer beats Agassi.</p><figure caption="Figure 1: Paraphrases with minimal word over­lap"></figure><p>In this paper we discuss the suitability of the recently proposed PASCAL Recognising Tex­tual Entailment (RTE) challenge (Dagan et al., 2005a) as an evaluation methodology for deter­mining the performance of redundant informa­tion identification techniques in the context of MDS. The aim of the RTE challenge is to aid the development of generic "semantic engines" that can be used in a number of applications such as Information Retrieval, Information Extraction, Text Summarisation and Machine Translation. Two types of language variability were investi­gated in this year's challenge: exact paraphrases and textual entailment or subsumption. The evaluation was defined as a binary classification problem where participating systems were re­quired to identify entailment relationships be­tween sentence pairs, i.e. a sentence <i>A </i>entails another sentence <i>B </i>if the meaning of <i>B </i>can be inferred from the meaning of <i>A </i>(Dagan et al., 2005b). During the data collection effort for the challenge, annotators were asked to limit the number of "difficult" cases of entailment that they included in the dataset.<page local="2" global="216"/> The entail­ment pairs shown in Figure 2 are representa­tive of the level of difficulty of the subsumption relationships found in the data, where entail­ment, in the majority of cases, requires syntac­tic matching, and synonym/paraphrase recog­nition rather than complex logical inference. In this way techniques that recognise redundant information in MDS and entailment in the RTE challenge have a lot in common. This point will be discussed in more detail in Section 2 of the paper.</p><p>The RTE development and test sets are com­posed of entailment examples taken from seven distinct application settings. The "Compara­ble Documents" portion of the collection is in­tended to be representative of the types of entailment and semantic equivalence found in multi-document summarisation. In general, participating systems at the RTE workshop per­formed significantly better (achieving as high as 87% accuracy) on this portion of the corpus. This result suggests that the types of entailment and semantic equivalence found in MDS are sig­nificantly less challenging than entailment found in other application settings. In this paper we will show that this result is misleading, and that the difficulty of identifying language variability in MDS is comparable with the level of diffi­culty observed in the other application domains explored by the RTE initiative.</p><p>The following section motivates the need for evaluating sub-tasks in MDS such as redundant information identification, and provides a brief overview of the techniques that have been used to identify language variability in MDS and at the RTE challenge. Section 3 discusses the RTE framework in the context of MDS and argues for the inclusion of additional examples of language variablity that frequently require identification in MDS but are not represented in the current RTE evaluation dataset. Section 5 describes the University College Dublin (UCD) RTE system, which detects entailment between sentence pairs using linguistic and statistical language analy­sis techniques<footnote anchor="1"/>. Section 6 discusses the perfor­mance of this system at the RTE workshop. In addition, the results of some initial experiments are provided that support the assertion that the performance of a competitive RTE system in an MDS application is comparable with its performance in other RTE applications settings. Fi­nally Section 7, discusses some conclusions and future directions for this work.</p><footnote label="1">The author was involved in the development of this system before moving to the NICTA Victoria Research Laboratory.</footnote><figure caption="Figure 2: Examples of syntactic variation, para­phrase and information subsumption in the"></figure><p>RTE dataset</p></section><section number="2" title="MDS and RTE"><p>So why is the RTE challenge an attractive sub-task evaluation methodology for MDS? Firstly, identifying semantic relationships and correctly clustering informationally equivalent sentences is a critical analysis component in many MDS systems for the following reasons: if sentences are incorrectly clustered then the commonal­ity between the documents is harder to deter­mine, and redundant (i.e. repetitive) infor­mation will be included in the summary - an outcome that summarisation systems want to avoid at all costs. Secondly, there are inher­ent limitations with the current summarisation evaluation standard provided by the Document Understanding Conference (DUC)<footnote anchor="2"/>, where both automatic and manual evaluation strategies are used to measure summary quality in terms of coverage, information redundancy, readability, <b>Task=Comparable Documents; Para­phrase Example; Judgement=TRUE</b></p><footnote label="2">DUC is an annual NIST sponsored workshop that provides participants with summarisation tasks and a corresponding evaluation framework, i.e. corpora, gold standard summaries and evaluation metrics. http://duc.nist.gov</footnote><p>Text A: Satomi Mitarai died of blood loss. Text B: Satomi Mitarai bled to death.</p><p><b>Task=Comparable Documents; Textual Entailment Example; Judgement=TRUE</b></p><p>Text A: The Kota (Fort), or Old City, for ex­ample, sometimes called the downtown section, is the central business district and Indonesia's financial capital.</p><p>Text B: The Kota is the country's business center.</p><p><b>Task=Comparable Documents; Syntactic Variation Example; Judgement=TRUE</b></p><p>Text A: Jakarta floods easily during the rainy season.</p><p>Text B: Jakarta is easily flooded during the rainy season.</p><page local="3" global="217"/><p>coherence and grammaticality. Since its cre­ation in 2001, the DUC initiative has helped to ensure that real and transparent progress is being made in summarisation research; how­ever, because the DUC evaluation methodology is determining the performance of many diffi­cult natural language processing (NLP) com­ponents concurrently (i.e. semantic analysis, content selection, sentence ordering and natu­ral language generation), it is often difficult to establish which techniques employed by a par­ticular high performing summarisation system have contributed most to its overall success. As such summarisation researchers are recog­nising the need for distinct evaluation frame­works for each of these sub-components. For example, researchers at Columbia University have separately evaluated their sentence cluster­ing algorithm, SimFinder, which is employed in their NewsBlaster summarisation system (McK-eown et al., 2002). More recently Barzilay and Lapata (Barzilay and Lapata, 2005) describe an evaluation methodology for text coherence techniques, which are commonly used by sum­marisation systems to improve text readability. The following subsection provides a flavour of the Entailment and Semantic Equivalence tech­niques presented at the PASCAL RTE-2005 challenge, followed by a description of two im­portant contributions made by Text Summari­sation researchers in this area.</p><subsection number="2.1" title="Language Variability Recognition Techniques"><p>The 2005 PASCAL RTE challenge is described by the organisers as "an initial attempt to form a generic empirical task that captures major se­mantic inferences across applications" (Dagan et al., 2005b). Sixteen groups submitted their RTE system results to the workshop. The sys­tems used a broad range of linguistic knowl­edge resources, statistical association metrics and logical inference mechanisms. As already stated, the simplest type of semantic equiva­lence measure that can be used to identify en-tailment is a measure of vocabulary overlap. Consequently, nearly all of the systems at the workshop considered uni-gram or n-gram over­lap metrics when classifying entailment. A number of more sophisticated methods were also proposed. These measures either used sta­tistical cooccurrence metrics (e.g. latent seman­tic indexing), lexical resources for detecting se­mantic relationships between verbs, nouns, and adjectives (e.g. WordNet (Millar, 1995)) or a combination of both. Syntactic-based overlap measures, which involves calculating the degree of match between parse tree representations of the sentence pair, were also popular. A few groups also incorporated a logical prover with some additional world knowledge resource such as a geospatial ontology or a semantic taxon­omy. Many of the submitted systems, such as the UCD submission described in the following section, considered more than just one of these measures during the entailment recognition pro­cess. More specifically, these lexical, syntac­tic, semantic or logical-based inference mea­sures were used as partial (rather than conclu­sive) evidence of the presence or absence of an entailment relationship between two sentences.</p><p>Overall the entailment recognition <i>accuracy </i>(see Section 6 for definition) of the participating systems at the workshop ranged from 50-60% where accuracy measures greater than 0.535 and 0.546 are better than chance at the 0.05 and 0.01 level, respectively (Dagan et al., 2005b). The general conclusion of the workshop was that relatively simple metrics used in combination performed better than more complex, "deeper" metrics such as logical inference or the incorpo­ration of world knowledge into the classification computation. An obvious explanation for this outcome is that deep linguistic analysis meth­ods are more prone to errors than simple term overlap metrics due to additional complexities such as word sense disambiguation.</p><p>So how do RTE techniques compare to the repetitive information detection methods used by the text summarisation community? Well as already stated, summarisation researchers have tended to favour simple similarity metrics based on the number of shared words. There are a couple of notable exceptions, however, which have been investigated by researchers at Columbia University.</p><p>Possibly the most well-known and success­ful approach to similarity detection in auto­matic summarisation is the SimFinder (Hatzi-vassiloglou et al., 2001) algorithm. This al­gorithm clusters sentences that share thematic content determined by a set of similarity fea­tures based on word, stem and Wordnet con­cept overlap as well as more complex features that capture match at a syntactic level such as subject-verb and verb-object relations. The subsequent clustering of sentences is then per­formed using a non-hierarchical clustering technique.<page local="4" global="218"/> Representative sentences from these clusters are then used to generate a summary.</p><p>(Barzilay and McKeown, 2005a) describe a revision strategy for improving the readabil­ity of the summary output of the SimFinder algorithm. Their revision system, MultiGen, searches for semantically equivalent textual units in the dependency tree graph represen­tations of the summary sentences. Semanti-cally similar words and phrases are identified using the WordNet taxonomy and a paraphrase dictionary, automatically constructed from par­allel monolingual corpora. So once an over­lapping paraphrase has been detected in the dependency trees this analysis then facilitates "information fusion", i.e. the generation of a single sentence that represents the information in the overlapping sentences. This text gen­eration technique has been integrated into the Columbia NewsBlaster multi-document sum­marisation system (McKeown et al., 2002).</p><p>It is clear from this discussion that the Text Summarisation community has much to gain from, and contribute to, the advancement of En­tailment and Semantic Equivalence recognition research.</p></subsection></section><section number="3" title="RTE and language variability in MDS"><p>In this section of the paper we comment on the coverage of the RTE evaluation corpora with re­spect to the type of real-world examples of se­mantic equivalence that require detection dur­ing multi-document summarisation. For the RTE 2005 challenge two development collec­tions and one test collection where released to participants<footnote anchor="3"/>. In each case, the datasets con­sisted of an even number of positive and neg­ative examples of entailment between sentence pairs. During the development of these datasets annotators were asked to collect relevant exam­ples that corresponded to typical success and failure settings in seven different applications, i.e. Information Retrieval (IR), Information Ex­traction (IE), Machine Translation (MT), Ques­tion Answering (QA), Paraphrase Acquisition (PP), Reading Comprehension (RC) and Com­parable Documents-style tasks (CD) such as multi-document summarisation. A more de­tailed discussion of the annotation process can be found in (Dagan et al., 2005b).</p><footnote label="3">The RTE datasets can be downloaded from: http://www.pascal-network.org/Challenges/RTE/ Datasets</footnote><p>As already stated, the motivation behind this paper is to establish whether or not these exam­ples of language variablity are reflective of the types of information redundancy found in an MDS setting. Particularly in the case of the CD sentence pairs which are reportedly representive of the MDS task. To answer this question we considered Mani's analysis of this problem in his review of MDS methods, where he defines 4 distinct types of redundancy between text ele­ments in MDS (Mani, 2001):</p><p>1. Two text elements are string identical when they are exact repetitions, i.e. the same sentence is repeated in multiple articles.</p><p>2. Two text elements are semantically equiv­alent when they are exact paraphrases of each other.</p><p>3. Two text elements are informationally equivalent if they are judged by humans to contain the same information.</p><p>4. A text element A informationally subsumes text element B if the information in ele­ment B is contained in A.</p><p>A manual examination of the RTE datasets shows that string identity and informational equivalence are not represented in these col­lections. Figure 2 provides examples of para­phrase and informational subsumption, i.e. tex­tual entailment in the RTE data. The exclu­sion of string identical examples isn't consid­ered critical as the detection of exact repetition is trivial. However, the lack of Mani's informa­tional equivalence type examples is more trou­blesome. An example of informational equiva­lence is shown in Figure 3. What differentiates this example of language variablity from those in Figure 2, is that the common information unit is an embedded paraphrase surrounding in both sentences by additional information. More specifically, while Text A and B share the infor­mation unit: "American Airlines laid off flight attendants", they also contain additional non-overlapping information units, i.e. the federal judge turned aside a union bid to block the job losses; unions warned travellers to expect long delays due to protests. From our analysis we can conclude that examples of <i>exact </i>paraphrase and entailment are the exception rather than the rule in MDS and other CD-type applica­tions. More often than not these systems will be required to deal with noisier instances of seman­tic equivalence where sentences repeat embed<b>Task=MDS; Embedded Paraphrase Ex­ample; Judgement=TRUE</b><page local="5" global="219"/></p><p>Text A: <i>American Airlines began laying off hundreds of flight attendants on Tuesday, </i>after a federal judge turned aside a union bid to block the job losses.</p><p>Text B: Unions have warned travellers that they can expect long delays this weekend as protests begin after <i>American Airlines let a large number of flight attendants go last week.</i></p><figure caption="Figure 3: An example of informational equiva­lence and embedded paraphrase"></figure><p>ded information units rather than exhibit com­plete semantic overlap (i.e. exact paraphrase) or subsumption.</p><p>In MDS, if the system can successfully de­tect these fuzzier examples of information re­dundancy it can make an informed decision on whether to: (a) substituted one sentence for an­other in the summary without any critical loss of information or (b) fuse these sentences to­gether as proposed by (Barzilay and McKeown, 2005a). Sentence fusion would probably be the most appropriate option in the case of the em­bedded paraphrase example shown in Figure 3. With this type of natural language generation application in mind, it would be beneficial if the RTE classification task also required systems to explicitly identify and return the common infor­mation unit(s) between each sentence pair, i.e. the system must justify its classification deci­sion.</p></section><section number="4" title="An MDS-based Informational Equivalence Dataset"><p>This section describes the development of a complementary RTE-style corpus of sentence-pairs that are more reflective of the types of information redundancy observed during multi-document summarisation.<footnote anchor="4"/>. Annotators were asked to use Columbia's online News-Blaster summarisation system<footnote anchor="5"/> (a consistent top-performer at the annual DUC summarisa­tion evaluation workshop) to aquire relevant sentence pairs. This curation strategy was em­ployed to ensure that the MDS dataset was representive of the types of informational equiv­alence that are problematic in MDS. A subse­quent analysis of the official DUC summary sub­missions to the multi-document summarisation task defined for the 2004 challenge (i.e. DUC task 2) indicates that these NewsBlaster ex­amples are consistent with the types of repet­itive information that were missed by sentence clustering strategies employed by other top per­forming summarisation systems at the work­shop.</p><footnote label="4">The MDS corpus can be downloaded from: http://www.cs.mu.oz.au/~nstokes/TE/MDS_corpus_ i.O.xml</footnote><footnote label="5">The NewsBlaster summarisation system: http://newsblaster.cs.columbia.edu</footnote><p>In line with the task-specific subsets in the RTE collection, the MDS dataset consists of 100 sentence pairs: 50 positive and 50 negative in­stances of informational equivalence. Figure 4 shows an example of each classification type. In the previous section it was explained that in or­der for a sentence pair to be tagged as a positive instance of informational equivalence it had to share an information unit; however, no formal definition of what constitutes such as unit was provided. The formulation of such a definition is a challenge in itself, and is currently receiving significant attention from the Text Summari­sation community in the context of summari­sation evaluation (Nenkova and Passonneau, 2004; Amigo, 2004). In the context of this task, an information unit is defined as a unit of text that contains at least one subject-verb relation­ship, (i.e. a noun phrase like "Air France Flight 358" is not a large enough information unit but "Air France Flight 358 crashed" is). In addition, when choosing these examples annotators were asked to be mindful of the underlying classifi­cation task in the context of a summarisation application, i.e. would the inclusion of both sentences result in unnecessary repetition in a summary. Any disagreement between annota-tor regarding the classification of certain pairs was discussed and resolved before experimenta­tion on the corpus began.</p><p>From the MDS examples in Figure 4 it can also be seen that these sentences often make reference to vague temporal expressions such as "deadline...set for Monday" and "Monday dead­line". In order to ground these temporal refer­ences to points in time the full text of the orig­inal source document would need to be anal­ysed. However, temporal resolution is not nec­essary in this classification task since examples were carefully chosen to ensure that if an event (such as a "suicide bomb attack") is mentioned in both sentences, then the system can assume that this information unit is referring to the same instance of the event in time.<page local="6" global="220"/></p><p>Figure 4: Pair 4 and Pair 62 are examples of positive and negative informational equivalence in the MDS dataset.</p><p>With regard to the negative examples of in­formation overlap in the MDS corpus, sentence pairs were picked from summaries that con­tained some word overlap, but which would still be considered unique information contributors to a summary. This helped to ensure that these negative sentence pairs were non-trivial.</p><p>During the creation of this corpus a num­ber of examples of "contradiction" (i.e. con­flicting news reports on the details of a specific event) between potential informationally equiv­alent sentence pairs were found. Although these examples represent another important problem in MDS, they were not included in the final ver­sion of the corpus because they frequently oc­cur in the RTE challenge datasets in the form of negative entailment examples as shown in Fig­ure 5.</p><p>In the following sections we describe the UCD RTE system, and compare its performance on the MDS dataset to its performance on the RTE test set. As already stated, this experiment is used to investigate our claim that the CD task data in the RTE challenge is unrepresentative of language variability in MDS.</p><p><b>Task=Comparable Documents; Judge-ment=False;</b></p><p>Text A: Jennifer Hawkins is the 21-year-old beauty queen from Australia. Text B: Jennifer Hawkins is Australia's 20-year-old beauty queen.</p><figure caption="Figure 5: An example of contradiction in the RTE data collection."></figure></section><section number="5" title="The UCD Textual Entailment Recognition System"><p>In this section, we present an overview of the UCD Textual Entailment Recognition system, which was originally presented at the PASCAL RTE workshop (Newman et al., 2005). This system uses a decision tree classifier to detect an entailment relationship between pairs of sen­tences that are represented using a number of difference features such as lexical, semantic and grammatical attributes of nouns, verbs and ad­jectives. This entailment classifier was gener­ated from the RTE training data using the C5.0 machine learning algorithm (Quinlan, 1993). The features used to train and test the classi­fier were calculated using the following similar­ity measures:</p><p>• The ROUGE (Recall-Oriented Understudy for Gisting Evaluation) (Lin and Hovy, 2004) n-gram overlap metrics, which have been used as a means of evaluating sum­mary quality at the DUC summarisation workshop. The Rouge package provides measurement options such as uni-gram, bi­gram, tri-gram and 4-gram term overlap, and a weighted and unweighted longest common subsequence overlap measure.</p><p>• The Cosine Similarity metric calculates the cosine of the angle between the respective term vectors of the sentence pair.</p><p>• The Hirst-St-Onge WordNet-based mea­sure (Millar, 1995), is an edge counting metric that estimates the semantic dis­tance between words by counting the num­ber of relational links between them in the WordNet taxonomy (Budanitsky and Hirst, 2001). This metric also defines con­straints on the length of the path and the types of transitive relationships that are allowed between concepts (nodes) in the taxonomy. These constraints are impor­tant because unlike other WordNet-based</p><p><b>Task=MDS; Pair Id=4; Judge-ment=TRUE;</b></p><p>Text A: The United States ratcheted up its pressure Saturday on Iraqi negotiators who are trying to meet a deadline for writing a draft constitution set for Monday. Text B: With Iraq's parliament facing a Monday deadline to approve a new constitution, Presi­dent Bush said Saturday that the document "is a critical step on the path to Iraqi self-reliance".</p><doubt alpha="50.0" length="36" tooSmall="False" monospace="0.0">Task=MDS;     Pair     Id=62; Judge-</doubt><p><b>ment=FALSE;</b></p><p>Text A: Discovery was loaded with nearly 7,000 pounds of garbage that had accumulated in the space station since it was last visited by a shuttle in December 2002.</p><p>Text B: The Discovery crew spent nine of their first 13 days in orbit transferring supplies to the space station.</p><page local="7" global="221"/><p>semantic relatedness measures (which only consider IS-A relationships) the Hirst-St Onge metric searches for paths that tra­verse the IS-A and HAS-A hierarchies in the noun taxonomy. Hence, this metric provides better coverage at an increased risk of detecting spurious relationships if unrestricted paths were allowed between concepts. This feature was implemented using the Perl Wordnet Similarity modules developed by (Patwardhan et al., 2003).</p><p>• A verb-specific semantic overlap met­ric, that uses the VerbOcean semantic network (Chklovski and Pantel, 2004b; Chklovski and Pantel, 2004a) to identify instances of antonymy and near-synonym between verbs. The relationships between verb-pairs in VerbOcean were gleaned from the web using lexico-syntactic patterns. Although WordNet provides a verb taxon­omy, the VerbOcean data was used because it appears to provide better coverage of the types of relationships needed for detecting entailment.</p><p>• A Latent Semantic Indexing (Deerwester et al., 1990) measure, like the WordNet measure, attempts to calculated similarity beyond vocabulary overlap by identifying latent relationships between words though the analysis of cooccurrence statistics in an auxiliary news corpus.</p><p>• The final similarity measure is based on a more thorough examination of verb se­mantics. This measure finds the longest common subsequence in the sentence-pair, and then detects evidence of contradiction or entailment in the subsequence (such as verb negation, synonymy, near-synonymy, and antonymy) using the VerbOcean tax­onomy. An example is shown in Figure 6.</p><p>A more detailed description of the UCD system can be found in (Newman et al., 2005).</p></section><section number="6" title="Language Variability Recognition Experiments and Results"><p>This section of the paper reports on the perfor­mance of the UCD RTE system on the RTE and MDS datasets. The RTE challenge defined two evaluation metrics:</p><p>• An <i>accuracy score </i>which is calculated as the number of correctly classified sentence</p><p>pairs (positive and negative) returned by the system divided by the number of sen­tence pairs in the dataset.</p><p>• A <i>confidence-weighted score (CWS) </i>that ranges between 0 (no correct judgements at all) and 1 (perfect score), and rewards the system when it assigns a higher confidence score to correct judgements rather than to incorrect ones.</p><figure caption="Figure 6: The Longest Common Subsequence is highlighted in italics."></figure><p>The UCD RTE and MDS results are shown in Table 1. The entailment classifier in the MDS and RTE experiments was trained using the RTE corpus training sets <b>(devl </b>and <b>dev2). </b>The average accuracy and CWS scores (0.565 and 0.6 respectively), and the task results listed below this row in the table represent the official</p><p>UCD results reported at the RTE 2005 work­shop. A manual analysis of these results showed that many of the misclassified errors made by the UCD system could be attributed to the oc­currence of equivalence phrasal and composi­tional paraphrases e.g. "X invented Y" = "Y was incubated in the mind of X". As explained in Section 5 the system can only identify word­level, atomic paraphrase units (e.g., child = kid; eat = devour) that are defined in the VerbO-cean and WordNet lexical resources. A more detailed discussion of system misclassifications is provided in (Newman et al., 2005).</p><p>Out of 16 groups UCD's average accuracy and CWS scores were ranked 4th and 5th respec­tively, where system accuracy results ranged from 0.586 to 0.495 and CWS scores from 0.686 to 0.507. In general, systems performed signif­icantly better on the CD entailment examples, and for many it was this score that added some respectability to their average accuracy score. The most plausible explanation for these high CD scores (as high as 87% accuracy), accord<b>Task=Paraphrase Acquisition; Judge-ment=FALSE</b></p><p>Text A: <i>France on Saturday </i>flew <i>a planeload of United Nations aid into eastern Chad </i>where French soldiers prepared to deploy from their base in Abeche towards the border with Su­dan's Darfur region.</p><p>Text B: <i>France on Saturday </i>crashed <i>a planeload of United Nations aid into eastern Chad</i> ing to (Dagan et al., 2005b), is that vocabu­lary overlap metrics performed very well on this task because sentence pairs containing common terms were more likely to have the same mean­ings than in the other tasks.<page local="8" global="222"/> This implies that MDS systems need nothing more than vocabu­lary overlap metrics, and that the negative ef­fect of errors from this component of an MDS system is minimal. However, a comparison of the UCD system results on the CD and MDS language variablity examples suggests that re­dundant information detection is as difficult as the other tasks investigated, and that further research effort is also required in this area.</p><p>results for the UCD entailment classifier.</p></section><section number="7" title="Conclusions"><p>This paper evaluates the RTE challenge as a potential evaluation framework for compar­ing the performance of redundant information recognition strategies used in multi-document summarisation (MDS) to detect informational equivalence across documents. Most MDS sys­tems use simple word counts to identify repet­itive information. The problem with this ap­proach is that many sentences that convey the same information show little surface resem­blance due to linguistic phenomenon such as paraphrase and synonymy. The RTE challenge provides an opportunity for summarisation re­searchers to evaluate more sophisticate redun­dancy identification techniques independent of the summarisation task. However, an analysis of the RTE development and test sets show that this data is not representative of the types of in­formational equivalence that require detection during the MDS process. More specifically, al­though subsumption relationships are a natu­ral occurrence in applications such as Question Answering and Information Retrieval (where the answer/relevant document will always entails the question/query) this is not the case for Comparable Documents-style tasks. The results of an experiment on a complementary dataset of MDS informational equivalence ex­amples using a competitive RTE system showed that identifying redundancy in MDS is more challenging than the results on the Comparable Documents portion of the RTE test set would suggest. Consequently, if the ultimate aim of the PASCAL RTE challenge is to build "generic semantic engines" then future evaluations will also have to consider the identification of em­bedded (semantic and syntactic) paraphrases across sentences.</p><p>An obvious extention of this work would be to incorporate the UCD RTE system into an MDS system, and compare its effect on sum­mary performance against a baseline semantic equivalence measure such as cosine similarity. It would also be interesting to further investi­gate how well the RTE evaluation framework simulates the process of identifying repetitive information in MDS and other applications. In a paper by Barzilay and Elhadad (2003), on sentence alignment for monolingual comparable corpora, it was shown that the effectiveness of the alignment process increased when the con­text surrounding sentences was also considered. This conclusion suggests that future RTE eval­uations should also consider evaluating the role of context in the entailment detection process, where additional context is provided by the doc­ument in which the sentence occurred.</p></section><section number="8" title="Acknowledgements"><p>The support of Enterprise Ireland and NICTA (National ICT Australia) is gratefully acknowl­edged.</p><table caption="Table 1: RTE and MDS Accuracy and CWS" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Task</b></p></td><td class="cell"><p><b>Accuracy</b></p></td><td class="cell"><p><b>CWS</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MDS</p></td><td class="cell"><p>0.5400</p></td><td class="cell"><p>0.6006</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>RTE Average</i></p></td><td class="cell"><p><i>0.5650</i></p></td><td class="cell"><p><i>0.6000</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CD</p></td><td class="cell"><p>0.7400</p></td><td class="cell"><p>0.7764</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>IE</p></td><td class="cell"><p>0.4917</p></td><td class="cell"><p>0.5260</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>IR</p></td><td class="cell"><p>0.5444</p></td><td class="cell"><p>0.6130</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PP</p></td><td class="cell"><p>0.5600</p></td><td class="cell"><p>0.5006</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MT</p></td><td class="cell"><p>0.5083</p></td><td class="cell"><p>0.5130</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>QA</p></td><td class="cell"><p>0.5385</p></td><td class="cell"><p>0.5006</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>RC</p></td><td class="cell"><p>0.5286</p></td><td class="cell"><p>0.5685</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><references><p>E. Amigo. 2004. An empirical study of informa­tion synthesis tasks. In <i>Association for Com­putational Linguistics (ACL'04).</i></p><p>R. Barzilay and N. Elhadad. 2003. Sentence alignment for monolingual comparable cor­pora. In <i>Empirical Methods in Natural Lan­guage Processing (EMNLP'03).</i></p><p>R. Barzilay and M. Lapata. 2005. Modeling local coherence: An entity-based approach. In <i>Association for Computational Linguistics (ACL'05).</i></p><p>R. Barzilay and K. McKeown. 2005a. Sentence fusion for multidocument news summariza­tion.<page local="9" global="223"/> <i>Computational Linguistics, </i>31(3).</p><p>A. Budanitsky and G. Hirst. 2001. Seman­tic distance in WordNet: An experimental, application-oriented evaluation of five mea­sures. In <i>the Workshop on WordNet and Other Lexical Resources, NAACL'01.</i></p><p>T. Chklovski and P. Pantel. 2004a. Global path-based refinement of noisy graphs applied to verb semantics. In <i>the International Joint Conference on NLP (IJCNLP-05), </i>pages 11­13.</p><p>T. Chklovski and P. Pantel. 2004b. VerbO-cean: Mining the web for fine-grained seman­tic verb relations. In <i>Empirical Methods in Natural Language Processing (EMNLP-04).</i></p><p>I. Dagan, O. Glickman, and B. Magnini (eds). 2005a. In <i>the PASCAL Recognising Textual Entailment Challenge Workshop, April 11th-13th 2005, Southampton, UK.</i></p><p>I. Dagan, O. Glickman, and B. Magnini. 2005b. The PASCAL recognising textual entailment challenge. In <i>the PASCAL Recognising Tex­tual Entailment Challenge Workshop 2005, </i>pages 1-8.</p><p>S. Deerwester, S. Dumais, G. Furnas, T. Lan­dauer, and R. Harshman. 1990. <i>Indexing by Latent Semantic Analysis. </i>Journal of the American Society for Information Science.</p><p>V. Hatzivassiloglou, J. Klavans, M. Holcombe, R. Barzilay, Min-Yen Kan, and K. McKeown. 2001. SimFinder: A flexible clustering tool for summarization. In <i>the Workshop on Au­tomatic Summarization, NAACL-01.</i></p><p>C.-Y. Lin and E. Hovy. 2004. Automatic evaluation of summaries using n-gram co-occurence statistics. In <i>the Document Under­standing Conference (DUC'04), National In­stitute ofStandards and Technology.</i></p><p>I Mani. 2001. <i>Automatic Summarization. </i>John Benjamins (Natural language processing se­ries, edited by Ruslan Mitkov, volume 3), Amsterdam.</p><p>K. McKeown, R. Barzilay, D. Evans, V. Hatzi-vassiloglou, J. Klavans, A. Nenkova, C. Sable, B. Schiffman, and S. Sigelman. 2002. Track­ing and summarizing news on a daily basis with Columbia's Newsblaster. In <i>the Human Language Technology Conference (HLT'02).</i></p><p>G. Millar. 1995. WordNet: a lexical database for english.   <i>Communications of the ACM,</i> 38(11):39-41.</p><doubt alpha="65.9" length="44" tooSmall="False" monospace="0.0">A. Nenkova and R. Passonneau. 2004. Evaluat-</doubt><p>ing content selection in summarization: The Pyramid Method. In <i>HLT-NAACL'04.</i></p><p>E. Newman, N. Stokes, J. Dunnion, and J. Carthy. 2005. UCD IIRG approach to the Textual Entailment Challenge. In <i>the PAS­CAL Recognising Textual Entailment Chal­lenge Workshop, </i>pages 53-56.</p><p>S. Patwardhan, J. Michelizzi, S. Banerjee, and T.  Pedersen.    2003. <i>WordNet:.-Similarity Perl    Module </i>http://search.cpan.org/<b> dist/wordnet-similarity/lib/wordnet/ similarity.°/0pm.</b></p><p>J.R. Quinlan. 1993. C5.0 machine learning al­gorithm. http://www.rulequest.com<b>.</b></p></references></body></article>