<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="733"/><title>A Multi-Document Multi-Lingual Automatic Summarization System</title><author surname="Honarpisheh" givenname="Mohamad Ali"><org  name="Sharif University of Technology" country="Iran" city="Tehran"/></author><author surname="Ghassem-Sani" givenname="Gholamreza"><org  name="Sharif University of Technology" country="Iran" city="Tehran"/></author><author surname="Mirroshandel" givenname="Ghassem"><org  name="Sharif University of Technology" country="Iran" city="Tehran"/></author></firstpageheader><frontmatter><p><b>A Multi-Document Multi-Lingual Automatic Summarization System</b></p><p>Mohamad Ali Honarpisheh, Gholamreza Ghassem-Sani, Ghassem Mirroshandel Sharif University of Technology,</p><p>Department of Computer Engineering, Tehran, Iran,</p><p>Honarpisheh@ce.sharif.edu<b>,   </b>Sani@sharif.ir<b>,  </b>Mirroshandel@ce.sharif.edu</p></frontmatter><abstract><b>Abstract. </b>In this paper, a new multi-document multi-lingual text summarization technique, based on singular value decom­position and hierarchical clustering, is pro­posed. The proposed approach relies on only two resources for any language: a word segmentation system and a dictionary of words along with their document fre­quencies. The summarizer initially takes a collection of related documents, and trans­forms them into a matrix; it then applies singular value decomposition to the re­sulted matrix. After using a binary hierar­chical clustering algorithm, the most im­portant sentences of the most important clusters form the summary. The appropri­ate place of each chosen sentence is deter­mined by a novel technique. The system has been successfully tested on summariz­ing several Persian document collections. </abstract></header><body><section number="1" title="Introduction"><p>With the advent of the Internet, different newspa­pers and news agencies regularly upload their news in their sites. This let users to access different viewpoints and quotes about a single event. At the same time of this explosive growth of the amount of textual information, the need of people for quick access to information has dramatically increased. The solution proposed for dealing with this huge amount of information is using Text Summarizers. Several systems have been developed with respect to this solution (McKeown et. al., 2002; Radev et. al., 2001).</p><p>Generally in the process of multi-document text summarization, a collection of input documents about a particular subject is received from the user and a coherent summary without redundant infor­mation is generated. However, several challenges exist in this process the most important of which are removing redundant information from the input sentences and ordering them properly in the output summary. In a new approach to multi-document summarization proposed in this paper, Singular Value Decomposition (SVD) is used to find the most important dimensions and also to remove noisy ones. This process makes clustering of simi­lar sentences easer. In order to determine the level of importance of different clusters, the generated singular values and singular vector of the SVD have been used in a fashion similar to that (Steinberger and., Jezek, 2004). To evaluate gener­ated summaries the SVD-based method proposed in the same paper is used.</p></section><section number="2" title="Text Summarization approaches"><p>There are different features with which we can classify text summarization systems. In (Sparck-Jones, 1999) these features are divided according to the input, purpose, and output of system. With respect to this categorization, our proposed system is a general multi-document multi-lingual text summarizer which generates extracts a summary from the input documents.</p><p>Different approaches to text summarization can be categorized in different ways according to vari­ous features of text summarization systems. With respect to the output of the system, there are two categories of extracting and abstracting methods.<page local="2" global="734"/> Extraction-based summarization methods are also divided into three classes.</p><p>The first group of Extraction-based methods is statistical. These methods statistically assign sig­nificance score to different textual units. The very first developed summarization methods were of this category (Edmundson and Wyllys, 1961). Scoring policy in these systems was based on dif­ferent features, such as term frequency and place of the sentences. Vector Space Models (Salton et. al., 1994), compression of sentences with Auto­matic Translation approaches (Knight and Marcu, own, 2000), Topic Signatures based methods (Lin and Hovy, 2000, Lacatusu et al., 2006) are among the most popular techniques that have been used in the summarization systems of this category.</p><doubt alpha="66.7" length="42" tooSmall="False" monospace="0.0">2000), Hidden Markov Model (Jing and McKe-</doubt><p>The second groups of extraction-based methods, shallow understanding approaches use some in­formation about words or textual units and their dependencies to improve the performance of ex­traction. The understanding can be induced using dependencies between words (Barzilay and Elha-dad, 1997), rhetorical relations (Paice and Johns, 1993), events (Filatova and Hatzivassiloglou, 2004). In all of these methods, the most focused dependencies are used as a measure for saliency of each textual unit.</p><p>The third group, knowledge-based approaches, uses particular domain knowledge in discriminat­ing the important parts of input documents. This knowledge is usually taking some assumptions about the working domain. Centrifuger (Elhadad et al., 2005) is a good example of systems in this category, which operates in medical domains.</p><p>The new approach proposed in this paper uses SVD and hierarchical clustering methods. It can therefore be categorized in the statistical based methods.</p></section><section number="3" title="SVD based methods"><p>Methods that use SVD to find salient information in the input document are a member of Vector Space Models. In such models, each textual unit is represented by a vector. Each component of this vector is filled with a value which represents both the local and global importance of each word.</p><p>The idea of using SVD in summarization was first introduced in (Gong and Liu, 2001). In this model, the input document is transformed into an <i>m x n </i>sparse matrix, A, where <i>m </i>is the number of words and <i>n </i>is the number of the sentences of input document.</p><p>The SVD of this <i>m x n </i>matrix, with the assump­tion <i>m f n</i>, is defined as follows:</p><doubt alpha="30.8" length="13" tooSmall="False" monospace="0.0">A = U S V,(1)</doubt><p>where <i>U=[uij] </i>is an <i>m x n </i>column-orthogonal ma­trix with columns named as the left singular vec­tors, <i>S = diag</i><b><i>(&lt;J1,a2,...,an</i></b>) is an <i>n x n </i>diagonal matrix with non-negative diagonal elements in de­scending order, and <i>V=[vij] </i>is an <i>n x n </i>row-orthogonal matrix with rows named as the right singular vectors (figure 1 demonstrate application of SVD to <i>A).</i><i> </i>The number of non-zero elements in S is equal to the rank, r, of matrix <i>A</i>.</p><page local="3" global="735"/><p>There are two viewpoints about the result of per­forming SVD on sentence by the word matrix of document (Gong and Liu, 2001). From transforma­tion viewpoint, SVD for each sentence reduces the dimensions from <i>m </i>to r. The salience degree of the reduced dimension decreases from the first to the <i>r</i>th dimension. From the semantic viewpoint, SVD derives the hidden latent structure of the input document. This structure is represented in <i>r </i>line­arly independent base vectors (i.e. concepts). SVD can capture and model the interrelations between concepts and hidden topics, which can be used to cluster semantically related words and sentences.</p><p>In SVD-summarization method, for each salient concept (singular vector) of matrix V, the sentence with the highest value in that dimension is chosen. This technique helps us to choose the sentences of the summary that best represent the most important concepts. For example, the most important sen­tence of a document in this summarization method is the sentence that has the highest value in the first row of <i>V.</i></p><p>However, this method faces two significant problems. First, the number of dimensions should be less than or at most equal to the number of dis­covered. Second, in this method just individual concept is used to determine their saliency, not their combinations. This strategy obviously works poor when a sentence that is not the most impor­tant of any dimension alone may contain concept that in combination make it important enough.</p><p>These problems led to the introduction of a new summarization method (Steinberger and Jezek, 2004). This new approach uses summation of the weighted components of singular vectors instead of each individual concept alone. The weight of each vector's component of each sentence is its corre­sponding singular value. The reason for such weighting is to increase the effect of more impor­tant singular vectors. Formally, the degree of sali­ence of each sentence can be computed by using the following formula:</p><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">i=1</doubt><p>where <i>sk </i>is the salience degree of <i>kth </i>sentence in the modified latent vector space, and <i>r </i>is the num­ber of important dimensions of the new space. Cor­responding value of each <i>r </i>dimensions is greater than half of the first singular value.</p><p>Both of above strategies for text summarization were proposed for single document summarization only. These approaches do not utilize the clustering power of SVD in discovering sentences with close meanings. On the other hand, their pure reliance on SVD, which does not depend on the characteristics of any language, makes them appropriate to be ap­plied to any language.</p></section><section number="4" title="Multi-Document SVD based Summari­zation"><p>In this paper a new version of the SVD based summarization is introduced. After transforming all documents into sentence by word matrix, SVD is applied to the resultant matrix. To remove re­dundant sentences from the summary, a hierarchi­cal bottom-up clustering algorithm is applied to the <i>r </i>most important extracted concepts of the input sentences. After extracting the clusters, the sali-ency score of each cluster is determined, and the most important clusters are selected. At the same time, using a simple ordering method, the appro­priate place of each sentence in the sorted collec­tion is determined. In the following sections, each of these processes is described in more details.</p><p><b>Matrix Construction and Application of SVD</b></p><p>Given a collection of input documents that are re­lated to a particular subject, the system decom­poses the documents into sentences and their cor­responding words. In addition, it computes the to­tal number of occurrences of each word in all documents as the Term Frequency (TF).</p><p>The developed system works on Persian lan­guage. It is assumed that words are separated by spaces. However, some words in Persian are com­pound words .However this did not cause any problem for the developed system; because the most meaningful part of such words is usually less common than others and thus have an Inverse Document Frequency (IDF) that is higher than that of more common less meaningful parts. IDF repre­sents the amount of meaning stored in each word. It can be used to reduce the impact of less impor­tant constituents such as stop words, which usually have a high TF but contains little meaning.The formula for calculating IDF is as follows:</p><doubt alpha="58.8" length="17" tooSmall="False" monospace="0.0">IDF (term)= log(-</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(3)</doubt><doubt alpha="100.0" length="6" tooSmall="False" monospace="0.0">NUMDOC</doubt><p><i>NUMDOC(term)</i><page local="4" global="736"/></p><p>where <i>NUMDOC </i>represents the total number of document processed to create IDF dictionary and <i>NUMDOC(term) </i>is the number of documents in which the <i>term </i>appeared.</p><p>After decomposition, the input sentences along with their corresponding words are arranged as a matrix. Two different weighting schemes have been applied to each element of this matrix: 1) a constant value and, 2) each word's associated TF*IDF (Salton and Buckley, 1988).</p><p>After constructing the Sentence by word matrix, SVD is applied to the resultant matrix. Applying SVD removes the effect of unimportant words and highlights more salient dimensions. It also reduces the number of dimensions for each sentence, re­sulting in an easer clustering process. This im­proves the performance of sentence clustering by making it faster and less sensible to unimportant differences between sentences.</p><p><b>Clustering</b></p><p>To cluster reduced dimension sentences, a binary hierarchical agglomerative clustering algorithm with average group linkage is used. In this algo­rithm, at first, each sentence is considered as a cluster. At each step, two closest clusters are com­bined into a single cluster. The dimension of this new cluster is the average dimensions of the two combining ones. These steps are repeated until we have only one cluster. So the result of this algo­rithm is a binary tree.</p><p>The question that needs to be answered at this step is "how clusters containing similar sentences can be extracted from this binary tree?" Two prop­erties are required to propose a sub-tree as a possi­ble cluster of similar sentences:</p><p>1. The number of existing sentences at the current node (cluster) should be less than or equal to the total number of input documents; because it is assumed that there is not much redundant information in each document. This assump­tion is valid with respect to the news docu­ments in which there might be only little re­dundancy.</p><p>2. The distance between two children of the cur­rent cluster should be less than or equal to the distance between current cluster and its sibling node. This condition has been found empiri­cally.</p><p>Using these two heuristics, similar clusters are ex­tracted from the binary tree.</p><p><b>Finding Salient units</b></p><p>To select important clusters from the set of ex­tracted clusters, different clusters are scored based on two diffrent methods. In the first method, the average of TF*IDF of different words in the each sentence in the current cluster are used. The second approach was the latest SVD-based approach which was proposed by (Steinberger and Jezek, 2004) and was described in the section 3. In the latter, score of each cluster is found using the fol­lowing formula:</p><p><u>Z</u><i>score(s)</i></p><doubt alpha="54.5" length="22" tooSmall="False" monospace="0.0">score(cluster) = -,(4)</doubt><doubt alpha="63.6" length="11" tooSmall="False" monospace="0.0">| cluster |</doubt><p>where <i>\cluster\ </i>represent the number of sentences in the current cluster.</p><p><b>Selecting and ordering sentences:</b></p><p>In this step, the final representation of the sum­mary will be generated. To this end, from each im­portant cluster, a sentence should be selected. At the same time, the proper order of selected sen­tences should be determined. To find this order, a Representative Document (RD) is selected. The RD is the document which includes most sentences of the most important clusters. After selecting RD the following steps are performed:</p><p>1. Starting from the most important cluster, while the number of processed words of summary sentences does not exceed form the specified number:</p><p>a.   If no sentence from the current cluster was not added to the summary:</p><p>i. If there is a sentence from the RD in this cluster, choose this sentence;</p><p>ii. Otherwise find the most important sen­tence, the current cluster: To find out the place of the selected sentence in the summary, a search is performed for clus­ters that contain both sentences from RD and neighbors of the selected sentence. The place of the sentences from RD that co- clustered with neighbors of the se­lected sentence is chosen as the selected sentence boundary.</p><p>iii. If any place has been found for the se­lected sentence, add it to summary in the specified location, and mark that cluster as having a sentence in the summary.</p><page local="5" global="737"/><p>2. If it remains any unplaced sentence which should be presented in the summary, go to step 1 with the remaining number of words.</p></section><section number="5" title="Experiments"><subsection number="5.1" title="Testing Collection"><p>The proposed summarizer is originally developed for the Persian language. In Persian like many other languages there is not a standard test collec­tion, to evaluate the summarizers. To overcome the lake of a test collection in Persian, an unsupervised approach of evaluating summaries is selected (i.e. SVD-based evaluation method proposed in (Steinberger and Jezek, 2004)). In addition to an evaluator, a collection of documents was also re­quired. For this purpose different texts related to a single event were collected. The properties of these collections are presented in table 1</p></subsection><subsection number="5.2" title="Results and Discussions:"><p>To find out which term weighting and distance measure causes the highest increase in the SVD-Scores, various combinations of these approaches has been used in the summarization approach. To find the distance between clusters, Euclidian, Hamming, and Chebyshev distances and to deter­mine the saliency of different clusters, TFIDF and SVD-based methods were used. The gained SVD-</p><p>Based score using different configurations are rep­resented in table 2</p><p>As it can be seen in table 2, TFIDF-based meth­ods score higher than SVD-based methods. Also, the most promising distance was the hamming dis­tance. It can also be seen that the performance de­creases substantially when instead of a constant value <i>tf*idf </i>scores were used. It was observed that using various distance measure the SVD-Score of each collection would be different. The SVD-Scores are in favor of using the boosting methods for classification of sentences with different dis­tance measure for each classifier. Comparing these results with the ones proposed in (Steinberger and Jezek, 2004), a significant decrease in evaluated SVD-Based scores is observed. One of the reasons for this phenomenon is that the distinct words ap­pearing in multi-documents are more extensive than the words appear in a single document.</p></subsection></section><section number="6" title="Conclusions"><p>This paper presents a new SVD-based multi­lingual multi-document summarization method using agglomerative clustering. In this method, first, using SVD, the most important concepts rep­resenting sentences are extracted into a word by a sentence matrix. Then, similar sentences are clus­tered using a new heuristic method. Finally, impor-</p><p><b>Table 1. </b>Testing Collections -Details <b>Table 3.</b><b> </b>Using TF-IDF for each element of the matrix<page local="6" global="738"/></p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Average number of distinct words in documents</p></td><td class="cell"><p>1474</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Average number of sentences in documents</p></td><td class="cell"><p>32</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Average number of sentences in subjects</p></td><td class="cell"><p>643</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Maximum Number of distinct words in subjects</p></td><td class="cell"><p>2189</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Minimum Number of distinct words in subjects</p></td><td class="cell"><p>485</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Number of subjects</p></td><td class="cell"><p>14</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Euclidian</p></td><td class="cell"><p>Hamming</p></td><td class="cell"><p>Chebyshev</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Avg</p></td><td class="cell"><p>Max</p></td><td class="cell"><p>Min</p></td><td class="cell"><p>Avg</p></td><td class="cell"><p>Max</p></td><td class="cell"><p>Min</p></td><td class="cell"><p>Avg</p></td><td class="cell"><p>Max</p></td><td class="cell"><p>Min</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>TFIDF</p></td><td class="cell"><p>0.450</p></td><td class="cell"><p>0.572</p></td><td class="cell"><p>0.286</p></td><td class="cell"><p><b>0.518</b></p></td><td class="cell"><p>0.596</p></td><td class="cell"><p>0.343</p></td><td class="cell"><p>0.475</p></td><td class="cell"><p>0.605</p></td><td class="cell"><p>0.264</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SVD-based</p></td><td class="cell"><p>0.466</p></td><td class="cell"><p>0.632</p></td><td class="cell"><p>0.313</p></td><td class="cell"><p>0.472</p></td><td class="cell"><p>0.650</p></td><td class="cell"><p>0.322</p></td><td class="cell"><p>0.449</p></td><td class="cell"><p>0.620</p></td><td class="cell"><p>0.309</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Table 2. </b>Using a constant value for word-sentence matrix</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Euclidian</p></td><td class="cell"><p></p></td><td class="cell"><p>Hamming</p></td><td class="cell"><p></p></td><td class="cell"><p>Chebyshev</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Avg</p></td><td class="cell"><p>Max</p></td><td class="cell"><p>Min</p></td><td class="cell"><p>Avg</p></td><td class="cell"><p>Max</p></td><td class="cell"><p>Min</p></td><td class="cell"><p>Avg</p></td><td class="cell"><p>Max</p></td><td class="cell"><p>Min</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>TFIDF</p></td><td class="cell"><p>0.364</p></td><td class="cell"><p>0.549</p></td><td class="cell"><p>0.109</p></td><td class="cell"><p>0.269</p></td><td class="cell"><p>0.512</p></td><td class="cell"><p>0.113</p></td><td class="cell"><p>0.406</p></td><td class="cell"><p>0.512</p></td><td class="cell"><p>0.283</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SVD-based</p></td><td class="cell"><p>0.309</p></td><td class="cell"><p>0.134</p></td><td class="cell"><p>0.563</p></td><td class="cell"><p>0.319</p></td><td class="cell"><p>0.499</p></td><td class="cell"><p>0.112</p></td><td class="cell"><p>0.367</p></td><td class="cell"><p>0.518</p></td><td class="cell"><p>0.235</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>in the summary are extracted from the resulting clusters. Different weighting schemes, distance metrics and scoring methods have been experi­mented. According to our experiments constant weighting scheme along with hamming distance is superior to other combinations. Since this method only needs determination of words and their in­verse document frequency, it can be applied to any language providing these resources. We are now trying to improve the performance of the proposed algorithm. It seems that applying Principle Direc­tion Partitioning (Blei, 2002) algorithm in the clus­tering phase and using Latent Dirichlet Allocation method (Boley, 1998) instead of the SVD based ones to model sentences can improve the score of the proposed method.</p></section><references><p>Barzilay, R. and Elhadad. M. 1997. "Using lexical chains for text summarization." In Proceedings of the ACL/EACL'97 Summarization Workshop, Madrid, Spain.</p><p>Blei, D., Ng, A., and Jordan M., Latent Dirichlet alloca­tion. Journal of Machine Learning Research, 3:993­1022, January 2003. (A shorter version appeared in NIPS 2002).</p><p>Boley, D.L.: Principal Direction Divisive Partitioning. Data Mining and Knowledge Discovery, Vol. 2(4):325-344, Dec. 1998.</p><p>Edmundson, H.P. and Wyllys, R.E., Automatic abstract­ing and indexing - Survey and recommendations, Communications of the ACM, Vol. 4, (1961) 226­234</p><p>Elhadad, N., Kan, M.Y., Klavans, J., McKeown, K.: Customization in a unified framework for summariz­ing medical literature, Artificial Intelligence in Medi­cine Vol. 33(2): (2005) 179-198.</p><p>Filatova, E., Hatzivassiloglou, V.: Event-based Extrac­tive summarization, In: Proceedings of ACL 2004 Workshop on Summarization, (2004) 104-111.</p><p>Gong, Y., Liu, X.: Generic Text Summarization Using Relevance Measure and Latent Semantic Analysis. Proceedings of the 24th ACM SIGIR conference on Research and development in information retrieval, New Orleans, Louisiana, United States (2001) 19-25</p><p>Jing, H., McKeown, K.: Cut and paste based text sum­marization. Proceedings of the 1st Conference of the North American Chapter of the Association for Computational Linguistics (NAACL'00), (2000), Se­attle-Washington</p><p>Knight, K., Marcu, D.: Statistics based summarization .step one: Sentence compression, Proceeding of the 17th National Conference of the American Associa­tion for Artificial Intelligence (2000)703-710.</p><doubt alpha="60.9" length="110" tooSmall="False" monospace="0.0">Lacatusu, F., Hickl, A., Roberts, K., Shi, Y., Bensley, J., Rink, B., Wang, P., Taylor, L.: LCC's GISTexter at</doubt><p>DUC 2006: Multi-Strategy Multi-Document Summa­rization,   Document   Understanding Conference</p><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">(2006)</doubt><p>Lin, C.Y., Hovy, E.: From single to multi-document summarization: A prototype system and its evalua­tion. Proceedings of the ACL, pages 457—164, 2002</p><p>McKeown, K.R., Barzilay, R., Evans, D., Hatzivassi-loglou, V., Klavans, J.L., Nenkova, A., Sable, C., Schiffman, B., Sigelman, S.: Tracking and summariz­ing news on a daily basis with Columbia's newsblas-ter. Proceedings of 2002 Human Language Technol­ogy Conference (HLT), San Diego, CA, 2002</p><p>Paice, C. D., Johns, A. P.: The identification of impor­tant concepts in highly structured technical papers, In: Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Develop­ment in Information Retrieval (1993)</p><p>Radev, D. R., Blair-Goldensohn, S., Zhang, Z., Ragha-van R.S.: Newsinessence: A system for domain-independent, real-time news clustering and multi-document summarization. Proceedings of 2001 Hu­man Language Technology Conference (Demo Ses­sion), San Diego, CA, 2001</p><p>Salton, G., Allan, J., Buckley, C., Singhal, A.: Auto­matic analysis, theme generation, and summarization of machine readable texts, Science, Vol. 264(5164),</p><doubt alpha="0.0" length="16" tooSmall="False" monospace="0.0">(1994) 1421-1426</doubt><p>Salton, G. and Buckley, C.. Term weighting approaches in automatic text retrieval. Information Processing and Management, (1988), 24(5):513523</p><p>Sparck-Jones, K.: Automatic summarizing: factors and directions. Mani I, Maybury MT, editors. Advances in automatic text summarization. (1999) 10-12 [chap­ter 1]</p><p>Steinberger, J., Jezek, K. : Text Summarization and Singular Value Decomposition, Lecture Notes in Computer Science.Advances in Information Systems, Vol. 3261/2004, Springer-Verlag, Berlin Heidelberg</p><doubt alpha="30.4" length="23" tooSmall="False" monospace="0.0">New York (2004) 245-254</doubt></references></body></article>