Multi-Topic Multi-Document Summarization
UTIYAMA Masao
Communications Research Laboratory 588-2, Iwaoka, Nishi-ku, Kobe, Hyogo 651-2492, Japan mutiy ama@crl. go. j p
HASIDA Kôiti
Electrotechnical Labor 1-1-4, Umezono, Tukuba Ibaraki 305-8568, Japan hasida@etl.go.jp
A large event consists of a number of smaller events. These component events are usually related but such relations may not be strong enough to define larger topics. For example, a war may consist of opening, battles, negotiations, and so on. These relatively independent events are considered to be topics by themselves and would accordingly be reported in multiple news articles.
Summarization of such a large event, or multiple documents about multiple topics, is the concern of this paper. Summarization of multiple documents containing multiple topics is an unexplored research issue. Some previous studies on summarization (McKeown and Radev, 1995; Barzilay et al., 1999; Mani and Bloedorn, 1999) deal with multiple documents about a single topic, but not about multiple topics
In order to summarize multiple documents with multiple topics, one needs a general, semantics-oriented method for evaluating importance. Summarization of a single document may largely exploit the document structure. As an extreme example, the first paragraph of a newspaper article often serves as a summary of the entire article. On the other hand, summarization of multiple documents in general must be more based on their semantic structures, because there is no overall consistent document structure across them.
Selection of multiple important topics (not keywords) for multiple-topic summarization has not yet been really addressed in the previous literature. The present paper proposes a method, based on spreading activation, for extracting important topics and important documents. Another method proposed which is useful for grasping the overview of multiple documents is visualization of important entities mentioned and relationships among them. Visualization of relationships among keywords has been studied in the context of information retrieval (Niwa et al., 1997; Sanderson and Croft, 1999), but to the authors' knowledge the present study is the first to address such visualization in the context of summarization. Of course a concise summary of the entire set of multiple documents can be obtained by recovering sentences from important entities and their relationships as demonstrated in section 3.3.
The present study assumes documents annotated with GDA (Global Document Annotation) tags (Hasida, 1997; Nagao and Hasida, 1998).
GDA is a project to make on-line documents machine-understandable on the basis of a linguistic tagset, while developing and spreading technologies of content-based presentation, retrieval, question-answering, summarization, translation, among others, with much higher quality than before. GDA thus proposes an integrated global platform for electronic content authoring, presentation, and reuse. The GDA tagset
Under the current state of the art, GDA-tagging is semiautomatic and calls for manual correction by human annotators; otherwise annotation would make no sense. The cost involved here pays, because annotated documents are generic information contents from which to render diverse types of presentations, potentially involving summarization, narration, visualization, translation, information retrieval, information extraction, and so forth. The present paper concerns summarization only, but the merit of GDA-tagging is not at all restricted to summarization, and that is why it is considered reasonable to assume GDA-tagged input here.
An example of a GDA-tagged sentence is shown in Figure 1. <su> means sentential unit. <np>, <v>, and <adp> stand for noun phrase, verb, and adnominal or adverbial phrase.
<su> and the tags whose name end with 'p!(such as <adp> and <vp>) are called phrasal tags. In a sentence, an element (a text span from a begin tag to the corresponding end tag) is usually a syntactic constituent. The elements enclosed in phrasal tags are called phrasal elements, which cannot be the head of larger elements. So in Figure 1 'flies' is specified to be the head of the <su> element and 'like' the head of the <adp> element.
Each element may have an identifier as the value for the id attribute. Coreferences, including identity anaphora, are annotated by the eq attribute, as follows:
<np id="jO">John</np> beats <adp eq="jO">his</adp> dog.
When the shared semantic content is not the referent but the type (kind, set, etc.) of the referents, the eq.ab attribute is used like the following:
You bought a <np id="cl">car</np>. I bought <np eq.ab="cl">one</np>, too.
A zero anaphora is encoded as follows:
Tom visited <np id="ml">Mary</np>. He had <v iob="ml">brought</v> a present.
iob="ml" means that the indirect object of brought is element whose id value is ml, that is, Mary.
Other relations, such as sub and sup, can also be encoded, sub represents subset, part, or element. An example follows:
She has <np id="bl">many books</np>.
Adventures in Wonderland''</namep> is her favorite.
sup is the inverse of sub, i.e., includer of any sort, which is superset as to subset, whole as to part, or set as to element.
Syntactic structures and coreferences are essential for the summarization method described in section 3. Further details such as semantics, coordination, scoping, illocutionary act, and so on, are omitted here.
A set of GDA-tagged documents is regarded as a network in which nodes roughly correspond to GDA elements and links represent the syntactic and semantic relations among them. This network is the tree of GDA elements plus cross-reference (via eq, eq.ab, sub, sup, and so on) links among them. Cross-reference links may encompass different documents. Figure 2 shows a schematic, graphical representation of the network.
Spreading activation is carried out in this network to assess the importance of the elements. Spreading activation has been applied to summarization of single GDA-tagged documents (Hasida et al., 1987; Nagao and Hasida, 1998). The main conjecture of the present study is that the merit of spreading activation in that it evaluates importances of semantic entities is greater in summarization of multiple documents with multiple topics, because summarization techniques using document structures do not apply here, as mentioned earlier.
To fit the semantic interpretation, activations spread under the condition that coreferent elements should have the same activation value.
The algorithm is shown in Figure 3. Here the external input c(i) to node i represents a priori importance of i, which is set on an empirical basis; for instance, an entity
where 5kj is 1 if there is a link between node k and node j, otherwise it is 0.
The score score(i) of node i is calculated by summing the activation values of all the nodes under node i in the syntactic tree structure:
where a(i) is the activation value of node i and ch(i) is the set of child nodes of node i. ch(i) is empty if node i is a leaf node, or a word. This score is regarded as the importance of node i.
Extraction of important documents is simple once the scores of the nodes in the network are obtained. Sorting the document nodes according to their scores and extracting higher-ranked ones is sufficient for the purpose.
Variables: N: number of nodes.
D: maximum out-degree of equivalence classes.
c(i): external input to node i.
w(i, j ) : weight of the link from node i to node j :
1 if connected via eq, 1/D otherwise.
a(i): activation value of node i. The initial value is 0. a(i) is the sum of all a(j,i).
a(i,j): activation value of the link from node i to node j. The initial value is 0.
Algorithm:
repeat {
a(i) = av; }■ until convergence.
Similar procedure is used to extract important sentences from an important document. Extracted sentences are pruned according to their syntactic structures. Anaphoric expressions such as he or she are substituted by their antecedents if necessary.
An experiment has been conducted to test the effectiveness of the proposed algorithm. The example set contains fifty Japanese articles about the Peru hostage incident which continued over four months from December 1996 to April 1997. They include a lot of topics such as opening, negotiation, settlement, and so on. The GDA-tagging of these articles has involved automatic morphological analysis by .11 MAX (Kurohashi and Nagao, 1998), automatic syntactic analysis by KNP (Kurohashi, 1998), and manual annotation encompassing morphology, syntax, coref-erence, and anaphora. The types of anaphora identified here are mainly plain coreference and zero anaphora. Cross-document coreferences among entities have been automatically identified by exact string matching. d They contained errors but those errors were not corrected for the experiment. Cross-document coreferences found were 'Peru'(49), 'Japan'(39), 'Peru President' (15), 'members of Tupac Amaru'(9), ... and so on, where the numbers indicate the numbers of documents which contain these expressions.
The external inputs to nodes have been defined according to the corresponding nodes: c(i) = 10 if node i's antecedent dominates sentences (e.g., a node coreferring with a paragraph). This sets a preference for nodes which summarize preceding sentences. c(i) = 5 if node i is in the title of an article, because a title is usually important. Otherwise c(i) = 1. These crude parameter values have been set by the authors on the basis of the investigation of sum-marizations of various documents.
Two important topics, the opening (first attack by Tupac Amaru) and the settlement (attack by the Peruvian government commandos), have been extracted from the four highest ranked articles, even though temporal information has not been incorporated in the algorithm. The opening article is the first article of the sample document set. However, the settlement article is the sixth last one. So mere extraction of the last article would miss the settlement.
The 25% summaries of the two articles made by extracting and pruning sentences are shown below together with their English translations:
Armed guerrillas broke into a party at Japanese ambassador's residence. Gunshots. 200 held in hostage. — Peru.
Many people from Japanese and Peruvian sides were held in hostage. The armed group consists of about twenty people, several of which broke into the ambassador's residence. It is reported that there are intermittent shootings now.
°We are planning to incorporate recent results (Bagga and Baldwin, 1998) to identify cross-document coreferences.
y At«±ÄS$f«cAo fe„ ÄfiJISraii, ff «O KM W & tft o T V > Ô 3 i: & fe.
Japanese ambassador's residence possession incident in Peru. All hostages released. Aim at recovering his power basis — President Fujimori.
President Fujimori demonstrated himself as a strong politician by resolving the Japanese ambassador's residence possession incident. He entered the residence site. This visit to the residence impressed that he was leading the operation himself. Why did he choose to resort to ? We can say that he aimed at recovering his political leadership by resolving through military power the residence incident, which is at the root of the political crisis.
The score score(i.j) of a relation between two entities i and j. is defined by:
score(i.j) where S(E(i)) is the set of sentence nodes which dominate one of the nodes in E{i) and \E{i)\ is the number of nodes in E(i). E(i). a(i). and score(s) have been defined in Section 3.1. \E(i)\a(i) is an analogy of ltf x idf \ which is a measure of term importance widely used in information retrieval.
If score(i.j) is sufficiently large, then S(E(i)) n S(E(j)) (the sentences containing both the entities) can constitute a cross-document summary concerning i and j
An entity-relation graph (E-R graph) is made of the relations highly ranked in terms of the score defined in (2). Figure 4 shows the E-R graph made of the top eleven relations extracted from the articles about Peru hostage incident. The numbers near the lines represent the ranks of the relations.
Peru hostage incident
Peruvian government
anese ambassador's residence
observers Fujimori Tupac Amaru
The top-ranked relation was the one between Peru and Japanese ambassador's residence. Three sentences extracted from the eight sentences which contained both of the entities were as follows.
According to reports from Peru, on the 17th the Japanese ambassador's residence in Lima, the capital, was attacked by an armed groups, allegedly leftist guerrillas, and many people from both Japanese and Peruvian sides were held in hostage.
2.
mx-mmit m h,
Concerning the hostage incident at the Japanese ambassador's residence caused by armed guerrilla, on the 18th the government requested the Peruvian government to assure the safety of the hostages, and sent Mr. HORIUTI Takahiko, coordinator, Division of Middle and South America. • • •
President Fujimori's political authority will recover because he succeeded in the operation to break into the Japanese ambassador's residence in Peru on the 22nd.
These sentences, extracted from different articles, have been paraphrased on the basis of
r'Coreference chains are used to summarize single documents by Azzam et al. (1999).
rThese sentences were selected manually to demonstrate the possibility of cross document summarization based on coreference.
coreferences. Since the name of the guerilla group is not identified in the beginning of the incident, the expression U !7 £ <fr tl <5
H&ktf )\>—~J"* (armed group which seems to be leftist guerrillas) is used in the first sentence there. This expression has been replaced with ^ ( h 9 • 7~?)iy (leftist guerrillas (Tupac Amaru)) by using cross-document coreferences. The equivalence of the first sentence and the first noun phrase of the second sentence, iSAS^ft' (the hostage incident caused by armed guerrillas at the Japanese ambassador's residence in Peru), were properly detected and was replaced by another expression because the equivalence of events across possibly different documents (McKeown et al., 1999; Barzilay et al., 1999) has been also detected by comparing predicate-argument structures of relevant sentences. Date expressions such as '17 0' (the 17th) have been augmented like '1996 ^ 12 M 17 0' (Dec. 17, 1996). The resulting passages are below (underlines indicating paraphrases), together with their English translations (boldface indicating paraphrases):
&n>fv7(h• 7v;i/) icgsan, a a\ Kjv - cdmmmtm^Miï ahic t^tih.
According to news from Peru, on December 17, 1996 the Japanese ambassador's residence in Lima, the capital, was attacked by leftist guerrillas (Tupac Amaru), and many people from both Japanese and Peruvian sides were held in hostage.
2. ^cDAfi*fr t'tts i2 ft is a, «K-ï&mzM
Concerning the hostage incident, the government requested the Peruvian government to assure the hostages' safety on December the 18th, and sent Mr. HORIUTI Takahiko, coordinator, Division of Middle and South America, Ministry of International Affairs, to Peru on that night.
imxßmmz mfro^t\z&&t£ôo„
President Fujimori's political authority will recover because he succeeded in the operation to break into the Japanese ambassador's residence in Peru on April 22, 1997.
Evaluation of multi-document summarization calls for far greater cost than that of single-document summarization. Testbeds for evaluation of multi-document summarization have not been developed yet. So the present evaluation is limited to the sample set of articles mentioned above, but the obtained results suggest general applicability of the proposed method and supports the conjecture that spreading activation is effective for multi-document multi-topic summarization.
As discussed in the previous section, the proposed method can extract important articles, that is, the opening and settlement articles, from fifty articles about Peru hostage incident. Also, an E-R, graph consisting of important relations among important entities, Peru, Japanese ambassadors' residence, Tupac Amaru, and so on, has been successfully constructed on this basis. The above-mentioned method also uses cross-document coreferences for replacing expressions with more concrete ones.
All these are archived essentially by using information in the GDA-tagging only, but not domain-dependent knowledge such as embedded in templates for information extraction. The proposed method is hence expected to detect important documents and sentences and create an appropriate E-R graph when applied to another set of documents about multiple topics.
The process of summarization can be decomposed into three stages (Sparck Jones, 1999):
2. source representation transformation to summary text representation, and
3. summary text generation from summary representation.
GDA-tagged documents are regarded as source text representations. The method described above focuses on the transformation stage. Its multi-linguality comes from the multi-linguality of the stage.
Summarization of multiple documents about multiple topics has been discussed in this paper. The method proposed here uses spreading activation over documents syntactically and semantically annotated with GDA tags. It is capable of:
• extraction of the opening and settlement articles from fifty articles about a hostage incident,
• creation of an entity-relation graph of important relations among important entities,
• extraction and pruning of important sentences, and
• substitution of expressions with more concrete ones using cross-document coreferences.
The method is essentially multilingual because it is based on GDA tags and the GDA tagset is designed to address multilingual coverage. Since this tagset can embed various linguistic information into documents, it could be a standard format for the study of the transformation and/or generation stage of document summarization, among other natural language processing tasks.
Saliha Azzam, Kevin Humphreys, and Robert Gaizauskas. 1999. Using coreference chains for text summarization. In ACL'99 Workshop on Coreference and Its Applications, pages 77......84.
Amit Bagga and Breck Baldwin. 1998. Entity-based cross-document coreferencing using the vector space model. In COLING-ACL'98, pages 79 85.
Regina Barzilay, Kathleen R. McKeown, and Michael Elhadad. 1999. Information fusion in the context of multi-document summarization. In ACL'99, pages 550 557.
Kôiti Hasida, Syun Ishizaki, and Hitoshi Isa-hara. 1987. A connectionist approach to the generation of abstracts. In Gerard Kempen, editor. Natural Langauge Generation: New Results in Artificial Intelligence, Psychology, and Linguistics, pages 149 156. Martinus Ni-jhoff.
Kôiti Hasida. 1997. Global Document Annotation. In NLPRS'97, pages 505 508.
Sadao Kurohashi and Makoto Nagao. 1998. Japanese morphological analysis system JUKI AN manual.
Sadao Kurohashi. 1998. Japanese syntactic analysis system KNP manual.
Inderjeet Mani and Eric Bloedorn. 1999. Summarizing similarities and differences among related documents. In Inderjeet Mani and Mark T. Maybury, editors, ADVANCES IN AUTOMATIC TEXT SUMMARIZATION chapter 23, pages 357 379. The MIT Press.
Mark T. Maybury. 1999. Generating summaries from event data. In Inderjeet Mani and Mark T. Maybury, editors, ADVANCES IN AUTOMATIC TEXT SUMMARIZATION chapter 17, pages 265 281. The MIT Press.
Kathleen McKeown and Dragomir R. Radev. 1995. Generating summaries of multiple news articles. In SIGIR'95, pages 74 82.
Kathleen R. McKeown, Judith L. Klavans, Vasileios Hatzivassiloglou, Regina Barzilay, and Elezar Eskin. 1999. Towards multi-document summarization by reformulation: Progress and prospects. In AAAI-99, pages 453460.
Katashi Nagao and Kôiti Hasida. 1998. Automatic Text Summarization Based on the Global Document Annotation. In COLING-ACL'98, pages 917 921.
Yoshiki Niwa, Shingo Nishioka, Makoto Iwayama, Akihiko Takano, and Yosihiko Nitta. 1997. Topic graph generation for query naviagation: Use of frequency classes for topic extraction. In NLPRS'97, pages 95100.
Mark Sanderson and Bruce Croft. 1999. Deriving concept hierarchies from text. In SI-GIR'99, pages 206 213.
Karen Sparck Jones. 1999. Automatic summarizing: factors and directions. In Inderjeet Mani and Mark T. Maybury, editors, ADVANCES IN A UTOMATIC TEXT SUMMARIZATION chapter 1, pages 1 12. The MIT Press.