<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="66"/><title>Summarizing Contrastive Viewpoints in Opinionated Text</title><pubinfo>Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing,pages 66-76, MIT, Massachusetts, USA, 9-11 October 2010. ©2010 Association for Computational Linguistics</pubinfo><author surname="Paul" givenname="Michael"><org  name="University of Iasi" country="Romania" city="Iasi"/></author><author surname="Zhai" givenname="ChengXiang"><org  name="University of Iasi" country="Romania" city="Iasi"/></author><author surname="Girju" givenname="Roxana"><org  name="University of Iasi" country="Romania" city="Iasi"/></author></firstpageheader><frontmatter><p><b>Summarizing Contrastive Viewpoints in Opinionated Text</b></p><p><b>Michael J. Paul*                ChengXiang Zhai Roxana Girju</b></p><p>University of Illinois             University of Illinois University of Illinois</p><p>Urbana, IL 61801, USA Urbana, IL 61801, USA Urbana, IL 61801, USA</p><p>mjpaul2@illinois.edu czhai@cs.uiuc.edu girju@illinois.edu</p></frontmatter><abstract>This paper presents a two-stage approach to summarizing multiple contrastive viewpoints in opinionated text. In the first stage, we use an unsupervised probabilistic approach to model and extract multiple viewpoints in text. We experiment with a variety of lexical and syntactic features, yielding significant perfor­mance gains over bag-of-words feature sets. In the second stage, we introduce <i>Compara­tive LexRank, </i>a novel random walk formula­tion to score sentences and pairs of sentences from opposite viewpoints based on both their representativeness of the collection as well as their contrastiveness with each other. Exper­imental results show that the proposed ap­proach can generate informative summaries of viewpoints in opinionated text. </abstract></header><body><section number="1" title="Introduction"><p>The amount of opinionated text available online has been growing rapidly, increasing the need for sys­tems that can summarize opinions expressed in such text so that a user can easily digest them. In this pa­per, we study how to summarize opinionated text in a such a way that highlights contrast between multi­ple viewpoints, which is a little-studied task.</p><p>Usually, online opinionated text is generated by multiple people, and thus often contains multi­ple viewpoints about an issue or topic. A view­point/perspective refers to "a mental position from which things are viewed" (cf. WordNet). An opin­ion is usually expressed in association with a partic­ular viewpoint, even though the viewpoint is usually</p><p>*Now at Johns Hopkins University (mpaul@cs.jhu.edu).</p><p>not explicitly given; for example, a blogger that is in favor of a policy would likely look at the positive aspects of the policy (i.e., positive viewpoint), while someone against the policy would likely empha­size the negative aspects (i.e., negative viewpoint). Moreover, in an opinionated text with diverse opin­ions, the multiple viewpoints taken by opinion hold­ers are often "contrastive", leading to opposite po­larities. Indeed, such contrast in opinions may be a main driving force behind many online discussions.</p><p>Futhermore, opinions regarding news events and other short-term issues may quickly emerge and dis­appear. Such opinions may reflect many differ­ent types of viewpoints which cannot be modeled by current systems. For this reason, we believe that a viewpoint summarization system would ben­efit from the ability to extract unlabeled viewpoints without supervision. Even if such clustering has in­accuracies, it could still be a useful starting point for human editors to select representative excerpts.</p><p>Thus, given a set of opinionated documents about a topic, we aim at automatically extracting and sum­marizing the multiple contrastive viewpoints implic­itly expressed in the opinionated text to facilitate digestion and comparison of different viewpoints. Specifically, we will generate two types of multi-view summaries: macro multi-view summary and micro multi-view summary. A macro multi-view summary would contain multiple sets of sentences, each representing a different viewpoint; these differ­ent sets of sentences can be compared to understand the difference of multiple viewpoints at the "macro level." A micro multi-view summary would con­tain a set of pairs of contrastive sentences (each pair consists of two sentences representing two different viewpoints), making it easy to understand the differ­ence between two viewpoints at the "micro level.<page local="2" global="67"/>"</p><p>Although opinion summarization has been exten­sively studied (e.g., (Liu et al., 2005; Hu and Liu, 2004; Hu and Liu, 2006; Zhuang et al., 2006)), ex­isting work has not attempted to generate our envi­sioned contrastive macro and micro multi-view sum­maries in an unsupervised way, which is the goal of our work. For example, Hu and Liu (2006) rank sen­tences based on their dominant sentiment according to the polarity of adjectives occuring near a product feature in a sentence. A contradiction occurs when two sentences are highly unlikely to be simultane­ously true (cf. (Marneffe et al., 2008)). Although little work has been done on contradiction detection, there are a few notable approaches (Harabagiu et al., 2006; Marneffe et al., 2008; Kim and Zhai, 2009).</p><p>The closest work to ours is perhaps that of Ler-man and McDonald (2009) who present an approach to contrastive summarization. They add an objective to their summarization model such that the summary model for one set of text is different from the model for the other set. The idea is to highlight the key differences between the sets, however this is a dif­ferent type of contrast than the one we study here -our goal is instead to make the summaries <i>similar </i>to each other, to contrast how the same information is conveyed through different viewpoints.</p><p>In this paper, we propose a two-stage approach to solving this novel summarization problem, which will be explained in the following two sections.</p></section><section number="2" title="Modeling Viewpoints"><p>The first challenge to be solved in order to generate a contrastive summary of multiple viewpoints is to model and extract these viewpoints which are hidden in text. In this paper we propose to solve this chal­lenge by employing the Topic-Aspect Model (TAM) (Paul and Girju, 2010), which is an extension of the Latent Dirichlet Allocation (LDA) model (Blei et al., 2003) for jointly modeling topics and viewpoints in text. While most existing work on such topic models (including TAM) has taken a topic model as a gen­erative model for word tokens in text, we propose to take TAM as a generative model for more complex linguistic features extracted from text.  These are more discriminative than single word tokens and can improve the accuracy of extracting multiple view­points as we will show in the experimental results' section. Below we first give a brief introduction to TAM and then present the proposed set of features.</p><subsection number="2.1" title="Topic-Aspect Model (TAM)"><p>LDA-style probabilistic topic models of document content (Blei et al., 2003) have been shown to offer state-of-the-art summarization quality. Such mod­els also provide a framework for adding additional structure to a summarization model (Haghighi and Vanderwende, 2009). In our case, we want to add more structure to a model to incorporate the notion of viewpoint/perspective into our summaries.</p><p>When it comes to extracting viewpoints, recent re­search suggests that it may be beneficial to model both topics and perspectives, as sentiment may be expressed differently depending on the issue in­volved (Brody and Elhadad, 2010; Paul and Girju, 2010). For example, let's consider a set of product reviews for a home theater system. Content topics in this data might include things like sound qual­ity, usability, etc., while the viewpoints might be the positive and negative sentiments. A word like <i>speakers, </i>for instance depends on the sound topic but not a viewpoint, while <i>good </i>would be an exam­ple of a word that depends on a viewpoint but not any particular topic. A word like <i>loud </i>would depend on both (since it would be considered positive senti­ment only in the context of the sound quality topic), while a word like <i>think </i>depends on neither.</p><p>We make use of a recent model, the Topic-Aspect Model (Paul and Girju, 2010), which can model such behavior with or without supervision. Under this model, a document has a mixture over topics as well as a mixture over viewpoints. The two mix­tures are drawn independently of each other, and thus can be thought of as two separate clustering di­mensions. A word is associated with variables de­noting its topic and viewpoint assignments, as well as two binary variables to denote if the word de­pends on the topic and if the word depends on the viewpoint. A word may depend on the topic, the viewpoint, both, or neither, as in the above example.</p><p>The generative process for a document d under this model can be briefly described as follows. For each word in a document:</p><page local="3" global="68"/><p>1. Sample a topic z from P(z|d) and a viewpoint v from P (v|d).</p><p>2. Sample a "level" £ <i>e </i>{0,1} from P(£|d). This determines if the word will depend on the topic (topical level) or not (background level).</p><p>3. Sample a "route" r e {0,1} from P(r|£, z). This determines if the word will depend on the viewpoint.</p><p>4. Sample a word w from P(w|z, v, r, £).</p><p>The probabilities are multinomial/binomial dis­tributions with Dirichlet/Beta priors, and thus this model falls under the standard LDA framework. The number of topics and number of viewpoints are pa­rameters that must be specified. Inference can be done with Gibbs sampling (Paul and Girju, 2010).</p><p>TAM naturally gives us a very rich output to use in a viewpoint summarization application. If we are doing unsupervised viewpoint extraction, we can use the output of the model to compute P(v|<i>sentence) </i>which could be used to generate summaries that contain only excerpts that strongly highlight one viewpoint over another. Similarly, we could use the learned topic mixtures to generate topic-specific summaries. Futhermore, the variables r and £ tell us if a word is dependent on the view­point and topic, and we could use this information to focus on sentences that contain informative con­tent words. Note that without supervision, TAM's clustering is based only on co-occurrences and the patterns it captures may or may not correspond with the viewpoints we wish to extract. Nonetheless, we show in this research that it can indeed find mean­ingful viewpoints with reasonable accuracy on cer­tain data sets. Although we do not explore this in this paper, additional information about the view­points could be added to TAM by defining priors on the distributions to further improve the accuracy of viewpoint discovery.</p></subsection><subsection number="2.2" title="Features"><p>Previous work with TAM used only bag of words features, which may not be the best features for cap­turing viewpoints. For example, "Israel attacked Palestine" and "Palestine attacked Israel" are iden­tical excerpts in an exchangable bag of words rep­resentation, yet one is more likely to come from the perspective of a Palestinian and the other from an Is­raeli. In this subsection, we will propose a variety of feature sets. We evaluate the utility of these features to the task of modeling viewpoints by measuring the accuracy of unsupervised clustering.</p><subsubsection number="2.2.1" title="Words"><p>We have experimented with simple bag of words features as baseline approaches, both with and with­out removing stop words, and found that the accu­racy of clustering by viewpoint is better when re­taining all words. This supports the observation that common function words may have important psychological properties (Chung and Pennebaker, 2007). Thus, we do not do any stop word removal for any of our other feature sets. We find that we get better results by stemming the words, so we apply Porter's stemmer to all of our features described.</p></subsubsection><subsubsection number="2.2.2" title="Dependency Relations"><p>It has been shown that using syntactic information can improve the accuracy of sentiment models (Joshi and Rose, 2009). Thus, instead of representing doc­uments as a bag of words, we will experiment with using features returned by a dependency parser. For this, we used the Stanford parser<footnote anchor="1"/>, which returns de­pendency tuples of the form rel(a, <i>b) </i>where rel is some dependency relation and <i>a </i>and b are tokens of a sentence. We can use these specific tuples as fea­tures, referred here as the <i>full-tuple </i>representation.</p><p>One problem with this representation is that we are using very specific information and it is harder for learning algorithms to find patterns due to the lack of redundancy. One solution is to generalize these features and rewrite a tuple rel (a, b) as two tuples: rel (a, *) and rel(*, b) (Greene and Resnik, 2009; Joshi and Rose, 2009). We will refer to this as the <i>split-tuple </i>representation.</p></subsubsection><subsubsection number="2.2.3" title="Negation"><p>If a word <i>W</i><i>i </i>appears in the head of a neg rela­tion, then we would like this to be reflected in other dependency tuples in which wi occurs. For a tuple rel(wi, <i>W</i><i>j</i>), if either wi or <i>W</i><i>j </i>is negated, then we simply rewrite it as -rel(wi, Wj).</p><p>An alternative would be to rewrite the individual word wi as -wi. However in our experiments this representation produced worse accuracies, perhaps because this produces less redundancy.</p><footnote label="1"> http://nlp.stanford.edu/software/</footnote><page local="4" global="69"/></subsubsection><subsubsection number="2.2.4" title="Polarity"><p>We also hypothesize that lexical polarity informa­tion may improve our model. If we are using the <i>full-tuple </i>representation, then a tuple becomes more general by replacing the specific word with a + or —. In the case that both words are polarity words, we use two tuples, replacing only one word at a time rather than replacing both words with their po­larity signs. To determine the polarity of a word, we simply use the Subjectivity Clues lexicon (Wil­son et al., 2005) and as polarity values, <i>positive </i>(+), <i>negative </i>(-), and <i>neutral </i>(*). Under our <i>split-tuple </i>representation, this becomes more specific by re­placing the * with the polarity sign. For example, the tuple amod(idea, <i>good) </i>would be represented as amod(idea, +) and amod(*,<i>good). </i>We collapse negated features to flip the polarity sign such that -rel(a, +) becomes rel(a, —).</p></subsubsection><subsubsection number="2.2.5" title="Generalized Relations"><p>We also experimented with backing off the rela­tions themselves. Since the Stanford dependencies can be organized in a hierarchy<footnote anchor="2"/>, we will represent the relations at more generalized levels in the hi­erarchy. For example, both a direct object and an indirect object are a type of object. For a relation rel, we define R<b>rei </b>as the relation above rel in the hierarchy - for example, R<b>dobj </b>= obj. We make an exception for neg which has its own im­portant properties that we wish to retain, so we let Rneg = neg. Thus, when using these features, we rewrite rel(a, b) as R<b>rei</b>(a, b).</p></subsubsection></subsection></section><section number="3" title="Multi-Viewpoint Summarization"><p>As a computation problem, extractive multi-viewpoint summarization would take as input a set of candidate excerpts<footnote anchor="3"/> X = {x1,x2, <i>■-,x\x</i>|} with k viewpoints and generate two types of multi-view contrastive summaries: 1) A macro contrastive sum­mary <i>Smacro </i>consists of k disjoint sets of excerpts, Xi, X2,Xk C X with each Xi containing repre­sentative sentences of the i-th view (i.e., <i>Smacro </i>= (X1,Xk)). The number of excerpts in each Xican be empirically set based on application needs.</p><footnote label="2">The complete hierarchy can be found in the Stanford de­pendencies manual (Marneffe and Manning, 2008).</footnote><footnote label="3">An "excerpt" refers to the smallest unit of text that will make up our summary such as a sentence.</footnote><p>2) A micro contrastive summary <i>Smicro </i>consists of a set of excerpt pairs, each containing two ex­cerpts from two different viewpoints, i.e., S<i>micro </i>= {(s1, t1),<i>(sn, tn)} </i>where si e X and ti e X are two comparable excerpts representing two different viewpoints. n is the length of the summary, which can be set empirically based on application needs. Note that both macro and micro summaries can re­veal contrast between different viewpoints, though at different granularity levels.</p><p>To generate macro and micro summaries based on the probabilistic assignment of excerpts to view­points given by TAM, we propose a novel extension to the LexRank algorithm (Erkan and Radev, 2004), a graph-based method for scoring representative ex­cerpts to be used in a summary. Our key idea is to modify the definition of the jumping probability in the random walk model so that it would favor ex­cerpts that represent a viewpoint well and encour­age jumping to an excerpt comparable with the cur­rent one but from a different viewpoint. As a re­sult, the stationary distribution of the random walk model would capture representative contrastive ex­cerpts and allow us to generate both macro and mi­cro contrastive summaries within a unified frame­work. We now describe this novel summarization algorithm (called Comparative LexRank) in detail.</p><subsection number="3.1" title="Comparative LexRank"><p>LexRank is a PageRank-like algorithm (Page et al., 1998), where we define a random walk model on top of a graph that has sentences to be summarized as nodes and edges placed between two sentences that are similar to each other. We can then score all the sentences based on the expected probability of a random walker visiting each sentence. We use the short-hand P (xj |xi) to denote the probability of being at node x<i>j </i>at a time t given that the walker was at x<i>i </i>at time t — 1. The jumping probability from node x<i>i </i>to node x<i>j </i>is given by:</p><doubt alpha="50.0" length="24" tooSmall="False" monospace="0.0">P(xj|xi)=sim(xi,(xj))(1)</doubt><p>where sim is a content similarity function defined on two sentence/excerpt nodes.</p><p>Our extension is mainly to modify this jumping probability in two ways so as to favor visiting con-trastive representative opinions from multiple viewpoints.<page local="5" global="70"/> The first modification is to make it favor jumping to a good representative excerpt x of any viewpoint v (i.e., with high probability p(v|x) ac­cording to the TAM model). The second modifica­tion is to further favor jumping between two excerpts that can potentially form a good contrastive pair for use in generating a micro contrastive summary.</p><p>Specifically, under our model, the random walker first decides whether to jump to a sentence of the same viewpoint or to a sentence of a different view­point. We define this decision as a binary variable z <i>e </i>{0,1}. Intuitively, if we can force the ran­dom walker to move back and forth between view­points, then the final scores will favor sentences that are similar across both viewpoints.</p><p>We define two different modified similarity func­tions for the two possible values of z. The first one, sim<b>o </b>(corresponding to z = 0) scales the similarity by the likelihood that the two x's represent the same viewpoint, and the second one, sim<b>1 </b>(for z = 1) scales the similarity by the likelihood that the x's come from different viewpoints.</p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">k</doubt><doubt alpha="54.9" length="51" tooSmall="False" monospace="0.0">sirno(xi,Xj) =sim(xi,xj)^ ^P(v = rn\xi)P(v = rn\xj)</doubt><doubt alpha="20.0" length="5" tooSmall="False" monospace="0.0">m = 1</doubt><doubt alpha="50.0" length="30" tooSmall="False" monospace="0.0">sim1(xi,Xj)    =   sim(xi,Xj)x</doubt><doubt alpha="40.7" length="27" tooSmall="False" monospace="0.0">y ^P(v = m1\xi)P(v = m2\xj)</doubt><doubt alpha="33.3" length="18" tooSmall="False" monospace="0.0">m1&gt;m2£[1&gt;k],mi =m2</doubt><p>where P (v|x) denotes the probability that the ex­cerpt x belongs to the viewpoint v, and in general, can be obtained through any multi-viewpoint model. A special case of this is when the labels for view­points are known, in which case P (v|x) = 1 for the correct label and 0 for the others.</p><p>In our experiments, P(v|x) comes from the out­put of TAM, and we define sim(xi, xj ) as the cosine between the vectors x<i>i</i><i> </i>and x<i>j,</i><i> </i>although again any similarity function could be used. The conditional transition probability from xi to xj given z is then:</p><doubt alpha="22.2" length="18" tooSmall="False" monospace="0.0">P(xj ^z)=—-;---(2)</doubt><doubt alpha="57.9" length="19" tooSmall="False" monospace="0.0">2-^j'exsimz(xi,xj')</doubt><p>Using A to denote P (z = 0) and marginalizing across z, we have the transition probability:</p><doubt alpha="39.6" length="48" tooSmall="False" monospace="0.0">P (xj|xi)=AP (xj|xi,z= 0) + (1— A)P (xj|xi,z= 1)</doubt><p>The stationary distribution of the random walk gives us a scoring of the excerpts to be used in our summary. It is also possible to score <i>pairs </i>of ex­cerpts that contrast each other. We define the score for a pair (xi, xj) as the probability of being at xi and transitioning to xj or vice versa, where xi and xj are of opposite viewpoints. Specifically:</p><doubt alpha="41.9" length="43" tooSmall="False" monospace="0.0">P(xi)P(xj|xj,z= 1) +P(xj)P(xj|xj, z= 1) (3)</doubt></subsection><subsection number="3.2" title="Summary Generation"><p>The final summary should be a set of excerpts that have a high relevance score according to our scoring algorithm, but are not redundant among each other. Many techniques could be used to accomplish this (Carbonell and Goldstein, 1998; McDonald, 2007), but we use a simple greedy approach: at each step of the summary generation algorithm, we add the excerpt with the highest relevance score as long as the excerpt's redundancy score - the cosine similar­ity between the candidate and the current summary - is under some threshold 5. This is repeated until the summary reaches a user-supplied length limit.</p><p><b>Macro contrastive summarization: </b>A macro-level summary consists of independent summaries for each viewpoint, which we generate by first using the random walk stationary distribution across all of the data to rank the excerpts. We then separate the top-ranked excerpts into two disjoint sets according to their viewpoint based on whichever gives a greater value of P(v|x), and finally remove redundancy and produce the summary according to our method de­scribed above. We refer to this as <i>macro</i><i> contrastive summarization, </i>because the summaries will contrast each other in that they have related content, but the excerpts in the summaries are not explicitly aligned with each other.</p><p><b>Micro contrastive summarization: </b>A candidate excerpt for a micro-level summary will consist of a pair ( xi, xj) with the pairwise relevance score de­fined in Equation 3. We can then rank these pairs and remove redundancy. It is possible that both xi and xj in a high-scoring pair may belong to the same view­point; such a case would be filtered out since we are mainly interested in including contrastive pairs in our summary. We refer to this as <i>micro</i><i> contrastive summarization, </i>because the summaries will allow us to see contrast at the level of individual excerpts from different viewpoints.</p><page local="6" global="71"/></subsection></section><section number="4" title="Experiments and Evaluation"><subsection number="4.1" title="Experimental Setup"><p>Evaluation of multi-view summarization is challeng­ing as there is no existing data set we can use. We leverage the resources on the Web and created two data sets in the domain of political opinion.</p><p>Our first dataset is a set of 948 verbatim responses to a Gallup® phone survey about the 2010 U.S. healthcare bill (Jones, 2010), conducted March 4-7, 2010. Responses in this set tend to be short and of­ten incomplete or otherwise ill-formed and informal sentences. Respondants indicate if they are 'for' or 'against' the bill, and there is a roughly even mix of the two viewpoints (45% for and 48% against).</p><p>We also use the Bitterlemons corpus, a collection of 594 editorials about the Israel-Palestine conflict. This dataset is fully described in (Lin et al., 2006) and has been used in other perspective modeling lit­erature (Lin et al., 2008; Greene and Resnik, 2009). The style of this data differs substantially from the healthcare data in that documents in this set tend to be long and verbose articles with well-formed sen­tences. It again contains a fairly even mixture of two different perspectives: 312 articles from Israeli au­thors and 282 articles from Palestinian authors.</p><p>Moreover, for the healthcare data set, manually extracted opinion polls are available on the Web, which we further leverage to construct gold stan­dard summaries to evaluate our method quantita­tively.    The data and test sets are available at</p><p>http://apfel.ai.uiuc.edu/resources.html.</p></subsection><subsection number="4.2" title="Stage One: Modeling Viewpoints"><p>The main research question we want to answer in modeling viewpoints is whether richer feature sets would lead to better accuracy than word features. We used our various feature sets as input to TAM and measured the accuracy of clustering documents by viewpoint. This evaluation serves both to mea­sure how accurately this type of clustering can be done, as well as to measure which types of features are important for modeling viewpoints.</p><p>We found that the clustering accuracy is improved if we measure the accuracy of only the subset of documents such that P (v|doc) is greater than some threshold (we used 0.8). Thus, the accuraries pre­sented in this section are measured using this confidence threshold. We will use this approach for the summarization task as well, as it ensures we are only summarizing documents where we have high confi­dence about their viewpoint membership.</p><p>There are several parameters to set for TAM. Since our focus is on comparing linguistic features with word features, we simply set these parame­ters to some reasonable values: We used Dirich-let pseudo-counts of 80.0 for P= 0), 20.0 for P= 1), uniform pseudo-counts of 5.0 for P(x), 0.1 for the topic and aspect mixtures, and 0.01 for the word distributions. We tell the model to use 2 viewpoints as well as 5 topics for the healthcare cor­pus and 8 topics for the Bitterlemons corpus.</p><p>There is high variance in the accuracies depend­ing on how the Gibbs samplers were initialized. We thus repeated the experiments many times to obtain relatively confident measures - 200 times for the healthcare set and 50 times for the Bitterlemons set, with 2000 iterations each time. A natural way to se­lect a model is to choose the model that gives the highest likelihood to its input. To evaluate how well this selection strategy would work, we measured the correlation between accuracy and likelihood.</p><p>The results are shown in Table 1. We can make several observations. (1) In all cases, the proposed linguistic features yield higher accuracy than the word features, supporting our hypothesis that for viewpoint modeling, applying TAM to these features improves performance over using simple word fea­tures. Since virtually all existing work on topic mod­els assumes word tokens as data to be modeled, our results suggest that it would be interesting to explore applying generative topic models to complex fea­tures for other tasks as well. This may be because by adding additional complex features to the observed data, we artificially inflate the data likelihood to em­phasize modeling co-occurrences of such features, which effectively biases the model to capture a cer­tain perspective of co-occurrences.</p><p>(2) The increase is substantially greater for the Bitterlemons corpus, which may be due to the fact that the parsing accuracy is likely better because the language is formal. The <i>split-tuple </i>representation is very significantly better for the healthcare corpus, but it is not clear which is better for the Bitterlemons corpus. It is also not clear how the generalized rela­tions affect the performance.</p><page local="7" global="72"/><p>Table 1: The clustering accuracy with TAM using a variety of feature sets. These results were averaged over 200 randomly-initialized Gibbs sampling procedures for the healthcare set, and 50 procedures for the Bitterlemons set. The 95% confidence interval using a standard t-test is also given. <i>Max </i>refers to the maximum accuracy obtained over the 200 or 50 instances. <i>MaxLL </i>refers to the clustering accuracy using the model that yielded the highest corpus log-likelihood as defined by TAM. <i>Corr </i>refers to the Pearson correlation coefficient between accuracy and log-likelihood.</p><p>(3) It appears that adding polarity helps the <i>full-tuple </i>features (by making them more general) but hurts the <i>split-tuple </i>features (by making them more specific). Negation significantly improves the <i>full-tuple </i>features in the Bitterlemons corpus, but it is not clear if it helps in the other cases. It should be noted that capturing negation and polarity is a very complex and difficult task, and it is not expected that our simple approaches will accurately capture these properties. Nonetheless, it seems that these simple features may help in certain cases.</p></subsection><subsection number="4.3" title="Stage Two: Summarizing Viewpoints"><p>For the second stage (i.e., the Comparative LexRank algorithm), we mainly want to evaluate the quality of the generated contrastive multi-viewpoint sum­mary and study the effectiveness of our extension to the standard LexRank. Below we present exten­sive evaluation of our summarization method on the healthcare data. We do not have an evaluation set with which to compute quantitative metrics on the Bitterlemons corpus, so we will instead perform a simple qualitative evaluation in the last subsection.</p><subsubsection number="4.3.1" title="Gold Standard Summaries"><p>The responses to the Gallup healthcare poll are described in an article<footnote anchor="4"/> which gives a table of the main responses found in the data along with their prominence in the data. In a way, this represents an expert human-generated summary of our database, and we will use this as a gold standard macro con-trastive summary against which the representativeness of a multi-viewpoint contrastive summary can be evaluated. The reasons given in this table will be used verbatim as our reference set, excluding the other/no-reason/no-opinion reasons. A sample of this table is shown in Table 2.</p><footnote label="4">http://www.gallup.com/poll/126521/Favor-Oppose- Obama-Healthcare-Plan.aspx</footnote><p>We also want to develop a reference set for micro contrastive summaries, where we are mainly inter­ested in evaluating contrastiveness. To do this, we asked 3 annotators to identify contrastive pairs in the "main reasons" table described above. Each pair must contain one reason from the 'for' side and one reason from the 'against' side, though we do not re­quire a one-to-one alignment; that is, multiple pairs may contain the same reason. We take the set of pairs that were identified as being contrastive by at least 2 annotators to be our gold set of contrastive pairs. Because these pairs come from the gold sum­mary, they are still representative of the collection as a whole, rather than fine-grained contrasts.</p><p>The macro reference set contains 9 'for' reasons and 15 'against' reasons. The micro reference set contains 13 annotator-identified pairs composed of 9 unique 'for' reasons and 8 unique 'against' reasons.</p></subsubsection><subsubsection number="4.3.2" title="Baseline Approaches"><p><b>Graph-based algorithms: </b>The standard LexRank algorithm can also be used to score pairs of sen­tences according to Equation 3. We will thus com­pare our new LexRank extension to the unmodified form of this algorithm. When A = 1, the random walk model only transitions to sentences within the same viewpoint, and thus in this case our modified algorithm produces the same ranking as the unmod­ified LexRank. This will be our first baseline.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><i>Healthcare Corpus</i></p></td><td class="cell"><p><i>Bitterlemons Corpus</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Feature Set</b></p></td><td class="cell"><p><b>Mean</b></p></td><td class="cell"><p><b>Med</b></p></td><td class="cell"><p><b>Max</b></p></td><td class="cell"><p><b>MaxLL</b></p></td><td class="cell"><p><b>Corr</b></p></td><td class="cell"><p><b>Mean</b></p></td><td class="cell"><p><b>Med</b></p></td><td class="cell"><p><b>Max</b></p></td><td class="cell"><p><b>MaxLL</b></p></td><td class="cell"><p><b>Corr</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>bag of words - no stopwords</p></td><td class="cell"><p>61.12+/-0.76% 60.58 +/- 0.79%</p></td><td class="cell"><p>61.01 60.50</p></td><td class="cell"><p>72.17 72.18</p></td><td class="cell"><p>52.92 62.58</p></td><td class="cell"><p>0.187 0.154</p></td><td class="cell"><p>68.22+/- 3.31% 61.29+/- 3.05%</p></td><td class="cell"><p>69.26 57.69</p></td><td class="cell"><p>88.27 91.34</p></td><td class="cell"><p>84.94 82.91</p></td><td class="cell"><p>0.39 0.33</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>full-tuples + negation + neg. + polarity</p></td><td class="cell"><p>62.42 +/- 0.88% 63.67 +/- 0.81% 63.16+/-0.94%</p></td><td class="cell"><p>62.47 64.54 64.46</p></td><td class="cell"><p>74.04 74.07 74.05</p></td><td class="cell"><p>63.37 69.25 67.8</p></td><td class="cell"><p>0.201 0.338 0.455</p></td><td class="cell"><p>80.89 +/- 3.45% 80.60 +/- 3.88% 82.53 +/- 3.55%</p></td><td class="cell"><p>85.40 88.07 86.64</p></td><td class="cell"><p>94.07 95.61 94.44</p></td><td class="cell"><p>92.10 91.32 91.16</p></td><td class="cell"><p>0.34 0.66 0.31</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>gen. full-tuples</p></td><td class="cell"><p>63.80 +/- 0.73%</p></td><td class="cell"><p>64.35</p></td><td class="cell"><p>73.29</p></td><td class="cell"><p>71.70</p></td><td class="cell"><p>0.254</p></td><td class="cell"><p>76.62 +/- 4.09%</p></td><td class="cell"><p>84.56</p></td><td class="cell"><p>94.53</p></td><td class="cell"><p>84.56</p></td><td class="cell"><p>0.25</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>split-tuples + negation + neg. + polarity</p></td><td class="cell"><p>68.32 +/- 0.90% 68.00 +/- 0.91% 65.11 +/- 1.05%</p></td><td class="cell"><p>70.74 69.11 65.35</p></td><td class="cell"><p>77.80 79.73 78.59</p></td><td class="cell"><p>76.57 76.14 67.22</p></td><td class="cell"><p>0.646 0.187 0.159</p></td><td class="cell"><p>77.14 +/- 3.64% 83.53 +/- 3.05% 81.24+/- 3.37%</p></td><td class="cell"><p>81.29 87.71 83.44</p></td><td class="cell"><p>92.99 95.00 95.03</p></td><td class="cell"><p>88.13 95.00 88.55</p></td><td class="cell"><p>0.30 0.12 0.08</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>gen. split-tuples</p></td><td class="cell"><p>69.31 +/- 0.83%</p></td><td class="cell"><p>70.69</p></td><td class="cell"><p>77.90</p></td><td class="cell"><p>73.90</p></td><td class="cell"><p>0.653</p></td><td class="cell"><p>76.69 +/- 4.36%</p></td><td class="cell"><p>83.78</p></td><td class="cell"><p>93.60</p></td><td class="cell"><p>91.67</p></td><td class="cell"><p>0.09</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="8" global="73"/><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">k</doubt><p>Table 2: Some of the top reasons given along with their prominence in the healthcare data, as analyzed by Gallup. This is a sample of what will serve as our gold set. The highlighted cells show an example of a contrastive pair identified by our annotators.</p><p><b>Model-based algorithms: </b>We will also compare against the approach of Lerman and McDonald (2009) who introduce their contrastiveness objective into a model-based summarization algorithm. The basic form of this algorithm is to select a set of sen­tences <i>Sm </i>to minimize the KL-divergence between the models of the summary Sm and the entire collec­tion <i>Xm </i>for a viewpoint m. The objective function is: - Em<b>=i </b>KL(L(Sm)||L(Xm)) where <i>L </i>is an ar­bitrary language model. We define <i>L(A) </i>simply as the unigram distribution over words in the collection A, a method also evaluated by Haghighi and Vander-wende (2009). This is the fairest comparison to our LexRank experiments, where sentences are also rep­resented as unigrams. (We do not do any modeling with TAM in our quantitative evaluation.)</p><p>Lerman and McDonald introduce an additional term to maximize the KL-divergence between the summary of one viewpoint and the collection of the opposite viewpoint, so that each viewpoint's sum­mary is dissimilar to the other viewpoints. We bor­row this idea but instead do the opposite so that the viewpoints' summaries are more (rather than less) similar to each other. This contrastive version of our model-based baseline is formulated as:</p><doubt alpha="45.8" length="24" tooSmall="False" monospace="0.0">- ]TKL(L(Smi)\\L(Xmi)) +</doubt><doubt alpha="33.3" length="6" tooSmall="False" monospace="0.0">mi = 1</doubt><doubt alpha="22.6" length="31" tooSmall="False" monospace="0.0">( k — 1      m2 G [1,k] ,mi =m2</doubt><doubt alpha="47.4" length="19" tooSmall="False" monospace="0.0">KL(L(Smi)\\L(Xm2 ))</doubt><p>Our summary generation algorithm is to iteratively add excerpts to the summary in a greedy fash­ion, selecting the excerpt with the highest score in each iteration. Note that this approach only gen­erates macro-level summaries, leaving us with the LexRank baseline for micro-level summaries.</p></subsubsection><subsubsection number="4.3.3" title="Metrics"><p>We will evaluate our summaries using a variant of the standard ROUGE evaluation metric (Lin, 2004). Recall that we have two different evaluation sets - one that contains all of the reasons for each viewpoint, and one that consists only of aligned pairs of excerpts. Since the same excerpt may appear in mul­tiple pairs, there would be significant redundancy in our reference summary if we were to include every pair. Thus, we will restrict a contrastive reference summary to exclude overlapping pairs, and we will have many reference sets for all possible combina­tions of pairs. There is only one reference set for the representativeness criterion.</p><p>Our reference summaries have a unique property in that the summaries have already been annotated with the prominence of the different reasons in the data. A good summary should capture the more prominent statements, so we will include this in our scoring function. We thus augment the basic ROUGE n-gram recall score by weighting the n-gram counts in the reference summary according to this percent­age. This is a generalization of the standard ROUGE formula where this percentage would be uniform.</p><p>For evaluating the macro-level summaries, we will score the summaries for the two viewpoints sep­arately, given a reference set <i>Refi </i>and a candidate summary <i>Ci </i>for a viewpoint <i>v = i. </i>The final score is a combination of the scores for both viewpoints, i.e. <i>S</i><i>rep </i><i>= </i>0.5S<i>(Ref</i><i>j,</i><i> </i><i>C)+0.5S(Ref</i><i>j</i><i>, C</i><i>j</i><i>) </i>where <i>S(Ref, C) </i>is our RouGE-based scoring metric. It would also be interesting to measure how well a viewpoint's summary matches the gold summary of the <i>opposite </i>viewpoint, which will give insights into how well the Comparative LexRank algorithm makes the two summaries similar to each other. We will measure this as the inverse of the above metric, i.e. <i>S</i><i>opp </i><i>= 05S(Ref</i><i>j,</i><i> </i><i>C</i><i>j</i><i>) </i>+ <i>0.5S(Ref3, C</i><i>j).</i></p><p>Finally, to score the micro-level comparative sum­maries (recall that this gives explicitly-aligned pairs of excerpts), we will concatenate each pair <i>(x</i><i>i,</i><i> </i><i>X</i><i>j</i><i>) </i>as a single excerpt, and use these as the excerpts in our reference and candidate summaries. The scor­ing function is then <i>S</i><i>p </i><i>= S(Ref</i><i>pairs, </i><i>C</i><i>pairs). </i>Note that we have multiple reference summaries for the<page local="9" global="74"/></p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>For</i></p></td><td class="cell"><p><i>Against</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>People need health insurance/Too many uninsured System is broken/Needs to be fixed Costs are out of control/Would help control costs Moral responsibility to provide/Obligation/Fair</p></td><td class="cell"><p>29% 18% 12% 12%</p></td><td class="cell"><p>Will raise costs of insurance/Make it less affordable Does not address real problems</p><p>Need more information/clarity on how system would work Against big government/Too much government involvement</p></td><td class="cell"><p>20% 19% 8% 8%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Table 3: Our evaluation scores for various values of A. Smaller val­ues of A favor greater contrastiveness. Note that A =1 should be con­sidered a baseline, because at this value the algorithm ignores the con-trastiveness and it becomes a standard summarization problem. MB and MC refer to our model-based baselines described in Subsection 4.3.2. Bold scores are significant over all baselines according to a paired t-test.</p><p>micro-level evaluation due to overlapping pairs in the evaluation set. In this case, the Rouge score is defined as the maximum score among all possible reference summaries (Lin, 2004).</p><p>We measure both unigram (removing stop words, denoted S-1) and bigram (retaining stop words, de­noted S-2) recall, stemming words in all cases.</p></subsubsection><subsubsection number="4.3.4" title="Evaluation Results"><p>In order to evaluate our Comparative LexRank algorithm by itself, in this subsection we will not use the output of TAM as part of our summariza­tion input, and will assign excerpts fixed values of <i>P(v\x) </i><i>= 1 </i>for the correct label and 0 otherwise. We constructed our sentence vectors with unigrams (removing stop words) and no IDF weighting.</p><p>We set the PageRank damping factor (Erkan and Radev, 2004) to 0.01 and tried combinations of the redundancy threshold ö <i>G </i>{0.01, 0.05, 0.1, 0.2} with different values of A, the parameter which con­trols the level of contrastiveness. For each value of A, we optimized ö on the original data set according to Srep x <i>S</i><i>opp </i>so that we can directly compare these scores, and then we tuned ö separately for Sp. The summary length is 6 excerpts. To obtain more ro­bust results, we repeated the experiment 100 times on random half-size subsets of our data. The scores shown in Table 3 are averaged across these trials.</p><p>In general, increasing A increases Srep, which suggests that tuning A behaves as expected, and high- and mid-range A values indeed produce sum­maries where the summaries of the two viewpoints are more similar to each other. Similarly, mid-range A values produce substantially highervalues ofSp-1, the unigram Rouge scores for the micro contrastive summary, although there is not a large difference be­tween the bigram scores. An example of our micro­level output is shown in Table 4.</p><p>As for our model-based baseline, we show results for both the basic algorithm (denoted MB) in addi­tion to the contrastive modification (denoted MC). We see that the contrastive modification behaves as expected and produces much higher scores for <i>Sopp</i>, however, this method does not outperform our LexRank algorithm. It is interesting to note that in almost all cases where a contrastive objective is in­troduced, the scores for the opposite viewpoint Sopp increase without decreasing the <i>Srep </i>scores, sug­gesting that contrastiveness can be introduced into a multi-view summarization problem without dimin­ishing the overall quality of the summary. It is admittedly difficult to make generalizations about these methods from experiments with only one data set, but we have at least some evidence that our al­gorithm works as intended.</p></subsubsection></subsection><subsection number="4.4" title="Unsupervised Summarization"><p>So far we have focused on evaluating our viewpoint clustering models and our multi-view summariza­tion algorithms separately. We will finally show how these two stages might work in tandem in unsuper-vised summarization of the Bitterlemons corpus.</p><p>Without a gold set, it is difficult to perform an extensive automatic evaluation as we did with the healthcare data. Instead we will perform a sim­ple qualitative evaluation to see if the algorithm ap­pears to achieve its goal. Thus, we asked 8 people to guess if each viewpoint's summary was written by Israeli or Palestinian authors. To diversify the summaries, for each annotator we randomly split each summary into two equal-sized subsets of the sentence set. Thus each person was asked to label four different summaries, which were presented in a random order. If humans can correctly identify the viewpoints, then this would suggest both that the TAM accurately clustered documents by viewpoint and the summarization algorithm is selecting sen­tences that coherently represent the viewpoints.</p><p>We first ran TAM on our data using the same pro­cedure and parameters as in Subsection 4.2 using the <i>full-tuple </i>features. We repeated this 10 times and used the model that gave the highest data likelihood as our model for summarization input. We then gen-<page local="10" global="75"/></p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>A</p></td><td class="cell"><p><b>Srep-1</b></p></td><td class="cell"><p><b>Sopp-1</b></p></td><td class="cell"><p><b>Srep-2</b></p></td><td class="cell"><p><i>Sopp</i><b>-2</b></p></td><td class="cell"><p>S<b>p-1</b></p></td><td class="cell"><p>S<b>p-2</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>0.0</p></td><td class="cell"><p><b>.425</b></p></td><td class="cell"><p><b>.416</b></p></td><td class="cell"><p>.083</p></td><td class="cell"><p>.060</p></td><td class="cell"><p>.309</p></td><td class="cell"><p>.036</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>0.2</p></td><td class="cell"><p><b>.410</b></p></td><td class="cell"><p><b>.423</b></p></td><td class="cell"><p>.082</p></td><td class="cell"><p><b>.065</b></p></td><td class="cell"><p>.285</p></td><td class="cell"><p>.044</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>0.5</p></td><td class="cell"><p><b>.419</b></p></td><td class="cell"><p><b>.434</b></p></td><td class="cell"><p>.085</p></td><td class="cell"><p><b>.072</b></p></td><td class="cell"><p><b>.386</b></p></td><td class="cell"><p>.044</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>0.8</p></td><td class="cell"><p><b>.410</b></p></td><td class="cell"><p>.324</p></td><td class="cell"><p><b>.095</b></p></td><td class="cell"><p>.028</p></td><td class="cell"><p><b>.367</b></p></td><td class="cell"><p><b>.062</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1.0</p></td><td class="cell"><p>.354</p></td><td class="cell"><p>.240</p></td><td class="cell"><p>.070</p></td><td class="cell"><p>.006</p></td><td class="cell"><p>.322</p></td><td class="cell"><p>.057</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MB</p></td><td class="cell"><p>.362</p></td><td class="cell"><p>.246</p></td><td class="cell"><p>.089</p></td><td class="cell"><p>.003</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MC</p></td><td class="cell"><p>.347</p></td><td class="cell"><p>.350</p></td><td class="cell"><p>.054</p></td><td class="cell"><p>.059</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Table 4: An example of our <i>micro-level </i>contrastive summarization output on the healthcare data, using 5 = 0.05 and A = 0.5.</p><p>erated macro contrastive summaries of our data for the two viewpoints with 6 sentences per viewpoint. We used unigram sentence vectors with IDF weight­ing. We used A = 0.5 and ö = 0.1, which gave the highest score at this A value on the healthcare data.</p><p>Only one of these sentences was clustered incor­rectly by TAM. The human judges correctly labeled 78% of the summary sets, suggesting that our sys­tem accurately selected some sentences that could be recognized as belonging to the viewpoints, but is not perfect. Unsupervised micro-level summaries were less coherent. Many of the sentences are mis-labeled, and the ones that are correctly labeled are not representative of the collection.</p><p>This is not surprising, and indeed exposes the challenge inherent in our problem definition: clus­tering documents based on similarity and then high­lighting sentences with high similarity but opposite cluster membership are almost conflicting objectives for an unsupervised learner. Such contrastive pairs are perhaps the most difficult data points to model. A good test of a viewpoint model may be whether it can capture the nuanced properties of the viewpoints needed to contrast them at the micro level.</p></subsection></section><section number="5" title="Discussion"><p>The properties of the text which we attempt to sum­marize in our work are related to the concept of <i>framing </i>from political science (Chong and Druck-man, 2010), which is defined as "an interpretation or evaluation of an issue, event, or person that empha­sizes certain of its features or consequences" focus­ing on "certain features and implications of the issue - rather than others." For example, someone in favor of the healthcare bill might focus on the benefits and someone against the bill might focus on the cost. However, our approach is different in that our contrastive objective encourages the summaries to include each point as addressed by all viewpoints, rather than each viewpoint selectively emphasizing only certain points. In a sense, this makes our sum­mary more like a live debate, where one side must directly respond to a point raised by the other side. For example, someone in favor of healthcare reform might cite the high cost of the current system, but someone against this might counter-argue that the proposed system in the new bill has its own high costs (as seen in the last row of Table 4). The idea is to show how both sides address the same issues.</p><p>Thus, we can say that we are summarizing the key arguments/issues/points from different opinions. Futhermore, our models and algorithms are defined very generally, and while we tested their viability in the domain of political opinion, they may also be useful for many other comparative tasks.</p><p>In conclusion, we have presented steps toward a two-stage system that can automatically extract and summarize viewpoints in opinionated text. First, we have shown that accuracy of clustering documents by viewpoint can be enhanced by using simple but rich dependency features. This can be done within the framework of existing probabilistic topic models without altering the models simply by using a "bag of features" representation of documents.</p><p>Second, we have introduced <i>Comparative LexRank, </i>an extension of the LexRank algorithm that aims to generate contrastive summaries both at the macro and micro level. The algorithm presented is general enough that it can be applied to any number of viewpoints, and can accomodate input where the viewpoints are either given fixed labels, or given probabilistic assignments. The tradeoff between contrast and representation can flexibly be tuned to an application's needs.</p><table class="main" frame="box" rules="all" border="1" regular="True"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>For the Healthcare Bill</i></p></td><td class="cell"><p><i>Against the Healthcare Bill</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>the government already provides half of the healthcare dollars in the united states [...] [they] might as well spend their dollars smarter</p></td><td class="cell"><p>government is too much involvement.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>my kids are uninsured.</p></td><td class="cell"><p>a lot of people will be getting it that should be getting it on their own, and my kids will be paying a lot of taxes.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>so everybody would have it and afford it.</p></td><td class="cell"><p>we cannot afford it.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>because of my family.</p></td><td class="cell"><p>i don't know enough about it and i don't know where exactly it's going to put my family.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>because i have no health insurance and i need it.</p></td><td class="cell"><p>because i have health insurance.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>cost of healthcare is so high.</p></td><td class="cell"><p>high costs.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="11" global="76"/></section><references><p>David M. Blei, Andrew Y. Ng, and Michael I. Jordan.</p><p>2003. Latent dirichlet allocation. <i>Journal of Machine</i> vised aspect-sentiment model for online reviews. In <i>NAACL '10.</i></p><doubt alpha="65.0" length="80" tooSmall="False" monospace="0.0">Learning Research,3:993-1022. Samuel Brody and Noemie Elhadad. 2010. An unsuper-</doubt><p>Jaime Carbonell and Jade Goldstein. 1998. The use of mmr, diversity-based reranking for reordering docu­ments and producing summaries. In <i>SIGIR '98, </i>pages 335-336.</p><p>Dennis Chong and James N. Druckman. 2010. Identi­fying frames in political news. In Erik P. Bucy and R. Lance Holbert, editors, <i>Sourcebook for Political Communication Research: Methods, Measures, and Analytical Techniques. </i>Routledge.</p><p>Cindy Chung and James W. Pennebaker. 2007. The psy­chological function of function words. <i>Social Commu­nication: Frontiers ofSocial Psychology, </i>pages 343­359.</p><p>Gunes Erkan and Dragomir R. Radev. 2004. Lexrank: graph-based lexical centrality as salience in text sum­marization. <i>J. Artif. Int. Res., </i>22(1):457-479.</p><p>Stephan Greene and Philip Resnik. 2009. More than words: syntactic packaging and implicit sentiment. In <i>NAACL 09, </i>pages 503-511.</p><p>Aria Haghighi and Lucy Vanderwende. 2009. Exploring content models for multi-document summarization. In <i>NAACL 09, </i>pages 362-370.</p><p>Sanda Harabagiu, Andrew Hickl, and Finley Lacatusu. 2006. Negation, contrast and contradiction in text pro­cessing.</p><p>Minqing Hu and Bing Liu. 2004. Mining opinion fea­tures in customer reviews. In <i>Proceedings ofAAAI, </i>pages 755-760.</p><p>Minqing Hu and Bing Liu. 2006. Opinion extraction and summarization on the Web. In <i>Proceedings of the 21st National Conference on Artificial Intelligence (AAAAI-2006), Nectar Paper Track, </i>Boston, MA.</p><p>Jeffrey M. Jones. 2010. "in u.s., 45% favor, 48% oppose obama healthcare plan", March.</p><p>Mahesh Joshi and Carolyn Penstein Rose. 2009. Gen­eralizing dependency features for opinion mining. In <i>ACL-IJCNLP 09:</i><i> Proceedings ofthe ACL-IJCNLP</i></p><p><i>2009 Conference Short Papers, </i>pages 313-316. Hyun Duk Kim and ChengXiang Zhai. 2009. Generating comparative summaries of contradictory opinions in text. In <i>CIKM '09: Proceeding of the 18th ACM con­ference on Information and knowledge management,</i> pages 385-394, New York, NY, USA. ACM.</p><p>Kevin Lerman and Ryan McDonald. 2009. Contrastive summarization: an experiment with consumer reviews.</p><p>In <i>NAACL '09, </i>pages 113-116, Morristown, NJ, USA. Association for Computational Linguistics.</p><p>Wei-Hao Lin, Theresa Wilson, Janyce Wiebe, and Alexander Hauptmann. 2006. Which side are you on?: identifying perspectives at the document and sentence levels. In <i>CoNLL-X '06: Proceedings of the Tenth Conference on Computational Natural Lan­guage Learning, </i>pages 109-116.</p><p>Wei-Hao Lin, Eric Xing, and Alexander Hauptmann. 2008. A joint topic and perspective model for ideo­logical discourse. In <i>ECML PKDD '08: Proceedings of the European conference on Machine Learning and Knowledge Discovery in Databases - Part II, </i>pages 17-32, Berlin, Heidelberg. Springer-Verlag.</p><p>Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Stan Szpakowicz Marie-Francine Moens, editor, <i>Text Summarization Branches Out: Proceedings oftheACL-04 Workshop, </i>pages 74­81, Barcelona, Spain, July.</p><p>Bing Liu, Minqing Hu, and Junsheng Cheng. 2005. Opinion observer: analyzing and comparing opinions on the web. In <i>WWW '05: Proceedings of the 14th international conference on World Wide Web, </i>pages 342-351, New York, NY, USA. ACM Press.</p><p>Marie-Catherine De Marneffe and Christopher Manning. 2008. Stanford typed dependencies manual. Techni­cal report, Stanford University.</p><p>Marie-Catherine De Marneffe, Anna Rafferty, and Christopher Manning. 2008. Finding contradictions in text. In <i>Proceedings ofthe Association for Compu­tational Linguistics Conference (ACL).</i></p><p>Ryan McDonald. 2007. A study of global infer­ence algorithms in multi-document summarization. In <i>ECIR'07: Proceedings of the 29th European confer­ence on IR research, </i>pages 557-564, Berlin, Heidel­berg. Springer-Verlag.</p><p>Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1998. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford Digital Library Technologies Project.</p><p>Michael Paul and Roxana Girju. 2010. A two-dimensional topic-aspect model for discovering multi-faceted topics. In <i>AAAI-2010: Twenty-Fourth Confer­ence on Artificial Intelligence.</i></p><p>Theresa Wilson, Janyce Wiebe, and Paul Hoffmann.</p><p>2005. Recognizing contextual polarity in phrase­level sentiment analysis. In <i>HLT 05: Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing,</i> pages 347-354.</p><p>Li Zhuang, Feng Jing, Xiao-yan Zhu, and Lei Zhang.</p><p>2006. Movie review mining and summarization. In <i>Proceedings ofthe ACM SIGIR Conference on Infor­mation and Knowledge Management (CIKM).</i><i></i></p></references></body></article>