This paper describes the automated creation of what we call topic signatures, constructs that can play a central role in automated text summarization and information retrieval. Topic signatures can be used to identify the presence of a complex concept—a concept that consists of several related components in fixed relationships. Restaurant-visit, for example, involves at least the concepts menu, eat, pay, and possibly waiter, and Dragon Boat Festival (in Tai­wan) involves the concepts calamus (a talisman to ward off evil), moxa (something with the power of preventing pestilence and strengthening health), pic­tures of Chung Kuei (a nemesis of evil spirits), eggs standing on end, etc. Only when the concepts co-occur is one licensed to infer the complex concept; eat or moxa alone, for example, are not sufficient. At this time, we do not consider the interrelationships among the concepts.

Since many texts may describe all the compo­nents of a complex concept without ever explic­itly mentioning the underlying complex concept—a topic—itself, systems that have to identify topic (s), for summarization or information retrieval, require a method of inferring complex concepts from their component words in the text.

In late 1970's, DeJong (DeJong, 1982) developed a system called FRUMP (Fast Reading Understand­ing and Memory Program) to skim newspaper sto­ries and extract the main details. FRUMP uses a data structure called sketchy script to organize its world knowledge. Each sketchy script is what FRUMP knows about what can occur in particu­lar situations such as demonstrations, earthquakes, labor strikes, and so on. FRUMP selects a partic­ular sketchy script based on clues to styled events in news articles. In other words, FRUMP selects an empty template whose slots will be filled on the fly-as FRUMP reads a news article. A summary is gen­erated based on what has been captured or filled in the template.

The recent success of information extraction re­search has encouraged the FRUMP approach. The SUMMONS (SUMMarizing Online NewS articles) system (McKeown and Radev, 1999) takes tem­plate outputs of information extraction systems de­veloped for MUC conference and generating sum­maries of multiple news articles. FRUMP and SUM­MONS both rely on prior knowledge of their do­mains. However, to acquire such prior knowledge is labor-intensive and time-consuming. For exam­ple, the University of Massachusetts CIRCUS sys­tem used in the MUC-3 (SAIC, 1998) terrorism do­main required about 1500 person-hours to define ex­traction patterns (Riloff, 1996). In order to make them practical, we need to reduce the knowledge en­gineering bottleneck and improve the portability of FRUMP or SUMMONS-like systems.

Since the world contains thousands, or perhaps millions, of complex concepts, it is important to be able to learn sketchy scripts or extraction patterns automatically from corpora—no existing knowledge base contains nearly enough information. (Riloff and Lorenzen, 1999) present a system AutoSlog-TS that generates extraction patterns and learns lexical con­straints automatically from preclassified text to al­leviate the knowledge engineering bottleneck men­tioned above. Although Riloff applied AutoSlog-TS to text categorization and information extraction, the concept of relevancy signatures introduced by her is very similar to the topic signatures we pro­posed in this paper. Relevancy signatures and topic signatures are both trained on preclassified docu­ments of specific topics and used to identify the presence of the learned topics in previously unseen documents. The main differences to our approach are: relevancy signatures require a parser. They are sentence-based and applied to text categorization. On the contrary, topic signatures only rely on cor­pus statistics, are document-based and used in text summarization.

In the next section, we describe the automated text summarization system SUMMARIST that we used in the experiments to provide the context of discussion. We then define topic signatures and de­tail the procedures for automatically constructing topic signatures. In Section 5, we give an overview of the corpus used in the evaluation. In Section 6 we present the experimental results and the possibility of enriching topic signatures using an existing ontol­ogy. Finally, we end this paper with a conclusion.

SUMMARIST (Hovy and Lin, 1999) is a system designed to generate summaries of multilingual in­put texts. At this time, SUMMARIST can process English, Arabic, Bahasa Indonesia, Japanese, Ko­rean, and Spanish texts. It combines robust natural language processing methods (morphological trans­formation and part-of-speech tagging), symbolic world knowledge, and information retrieval tech­niques (term distribution and frequency) to achieve high robustness and better concept-level generaliza­tion.

The core of SUMMARIST is based on the follow­ing 'equation':

summarization = topic identification + topic interpretation + generation.

These three stages are:

Topic Identification: Identify the most important (central) topics of the texts. SUMMARIST uses positional importance, topic signature, and term frequency. Importance based on discourse structure will be added later. This is the most developed stage in SUMMARIST.

Topic Interpretation: To fuse concepts such as waiter, menu, and food into one generalized concept restaurant, we need more than the sim­ple word aggregation used in traditional infor­mation retrieval. We have investigated concept I ABCMEWS.com : Delay in Handing Flight 990 Probe to FBI

Figure 1: A Nov. 16 1999 ABC News page summary-generated by SUMMARIST.

counting and topic signatures to tackle the fu­sion problem.

Summary Generation: SUMMARIST can pro­duce keyword and extract type summaries.

Figure 1 shows an ABC News page summary about EgyptAir Flight 990 by SUMMARIST. SUM­MARIST employs several different heuristics in the topic identification stage to score terms and sen­tences. The score of a sentence is simply the sum of all the scores of content-bearing terms in the sen­tence. These heuristics are implemented in separate modules using inputs from preprocessing modules such as tokenizer, part-of-speech tagger, morpholog­ical analyzer, term frequency and tfidf weights cal­culator, sentence length calculator, and sentence lo­cation identifier. We only activate the position mod­ule, the tfidf module, and the topic signature module for comparison. We discuss the effectiveness of these modules in Section 6.

Before addressing the problem of world knowledge acquisition head-on, we decided to investigate what type of knowledge would be useful for summariza­tion. After all, one can spend a lifetime acquir­ing knowledge in just a small domain. But what is the minimum amount of knowledge we need to enable effective topic identification as illustrated by the restaurant-visit example? Our idea is simple. We would collect a set of terms that were typi­cally highly correlated with a target concept from a preclassified corpus such as TREC collections, and then, during summarization, group the occurrence of the related terms by the target concept. For exam­ple, we would replace joint instances of table, menu, waiter, order, eat, pay, tip, and so on, by the single phrase restaurant-visit, in producing an indicative summary. We thus defined a topic signature as a family of related terms, as follows:

where topic is the target concept and signature is a vector of related terms. Each ti is an term highly-correlated to topic with association weight w, . The number of related terms n can be set empirically-according to a cutoff associated weight. We describe how to acquire related terms and their associated weights in the next section.

4. select a confidence level from the x distribution table; determine the cutoff associated weight and the number of terms to be included in the signatures

The training data derives from the Question and Answering summary evaluation data provided by TIPSTER-SUMMAC (Mani et al., 1998) that is a subset of the TREC collections. The TREC data is a collection of texts, classified into various topics, used for formal evaluations of information retrieval sys­tems in a series of annual comparisons. This data set contains essential text fragments (phrases, clauses, and sentences) which must be included in summaries to answer some TREC topics. These fragments are each judged by a human judge. As described in Sec­tion 3, SUMMARIST employs several independent modules to assign a score to each sentence, and then combines the scores to decide which sentences to ex­tract from the input text. One can gauge the efficacy

TREC Topic Description_ Test Questions

Table 1: TREC topic description for topic 151, test questions expected to be answered by relevant doc­uments, and a sample document with answer keys.

of each module by comparing, for different amounts of extraction, how many 'good' sentences the module selects by itself. We rate a sentence as good simply if it also occurs in the ideal human-made extract, and measure it using combined recall and precision (F-score). We used four topics of total 6,194 doc­uments from the TREC collection. 138 of them are relevant documents with TIPSTER-SUMMAC pro­vided answer keys for the question and answering evaluation. Model extracts are created automati­cally from sentences containing answer keys. Table 1 shows TREC topic description for topic 151, test questions expected to be answered by relevant doc­uments, and a sample relevant document with an­swer keys markup.

topic 271: Solar Power, 1555 texts, 59 relevant.

In order to assess the utility of topic signatures in text summarization, we follow the procedure de­scribed at the end of Section 4.1 to create topic signature for each selected TREC topic. Docu­ments are separated into relevant and nonrelevant sets according to their TREC relevancy judgments for each topic. We then run each document through a part-of-speech tagger and convert each word into its root form based on the WordNet lexical database. We also collect individual root word (unigram) fre­quency, two consecutive non-stopword (bigram) fre­quency, and three consecutive non-stopwords (tri-gram) frequency to facilitate the computation of the —2logX value for each term. We expect high rank­ing bigram and trigram signature terms to be very-informative. We set the cutoff associated weight at 10.83 with confidence level a = 0.001 by looking up a x statistical table.

Table 2 shows the top 10 unigram, bigram, and tri­gram topic signature terms for each topic . Several conclusions can be drawn directly. Terms with high —2logX are indeed good indicators for their corre­sponding topics. The —2logX values decrease as the number of words in a term increases. This is rea­sonable, since longer terms usually occur less often than their constituents. However, bigram terms are more informative than unigram terms as we can ob­serve: jail/prison overcrowding of topic 151, tobacco industry of topic 257, computer security of topic 258, and solar energy/power of topic 271. These auto­matically generated signature terms closely resemble or equal the given short TREC topic descriptions. Although trigram terms shown in the table, such as federal court order, philip morris rjr, jet propul­sion laboratory, and mobile telephone system are also meaningful, they do not demonstrate the closer term relationship among other terms in their respective topics that is seen in the bigram cases. We expect that more training data can improve the situation.

We notice that the —2logX values for topic 258 are higher than those of the other three topics. As indicated by (Mani et al., 1998) the majority of rel­evant documents for topic 258 have the query topic as their main theme; while the others mostly have the query topics as their subsidiary themes. This implies that it is too liberal to assume all the terms in relevant documents of the other three topics are relevant. We plan to apply text segmentation algo­rithms such as TextTiling (Hearst, 1997) to segment documents into subtopic units. We will then per­form the topic signature creation procedure only on the relevant units to prevent inclusion of noise terms.

In this paper we presented a procedure to automati­cally acquire topic signatures and valuated the effec­tiveness of applying topic signatures to extract topic relevant sentences against two other methods. The topic signature method outperforms the baseline and the tfidf methods for all test topics. Topic signatures can not only recognize related terms (topic identifi­cation), but group related terms together under one target concept (topic interpretation). Topic identi­fication and interpretation are two essential steps in a typical automated text summarization system as we present in Section 3.

Topic signatures can also been viewed as an in­verse process of query expansion. Query expansion intends to alleviate the word mismatch problem in information retrieval, since documents are normally-written in different vocabulary. How to automati­cally identify highly correlated terms and use them to improve information retrieval performance has been a main research issue since late 1960's. Re­cent advances in the query expansion (Xu and Croft, 1996) can also shed some light on the creation of topic signatures. Although we focus the use of topic signatures to aid text summarization in this paper, we plan to explore the possibility of applying topic signatures to perform query expansion in the future.

The results reported are encouraging enough to allow us to continue with topic signatures as the ve­hicle for a first approximation to world knowledge. We are now busy creating a large number of signa­tures to overcome the world knowledge acquisition problem and use them in topic interpretation.

We thank the anonymous reviewers for very use­ful suggestions. This work is supported in part by DARPA contract N66001-97-9538.