<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="180"/><title>Sentiment Analysis of Conditional Sentences</title><author surname="Narayanan" givenname="Ramanathan"><org  name="Northwestern University" country="USA" city="Evanston"/></author><author surname="Liu" givenname="Bing"><org  name="Northwestern University" country="USA" city="Evanston"/></author><author surname="Choudhary" givenname="Alok"><org  name="Northwestern University" country="USA" city="Evanston"/></author></firstpageheader><frontmatter><p><b>Sentiment Analysis of Conditional Sentences</b></p><p><b>Ramanathan Narayanan Bing Liu Alok Choudhary</b></p><p>Dept. of EECS Dept. of Computer Science Dept. of EECS</p><p>Northwestern University     Univ. of Illinois at Chicago       Northwestern University</p><p>ramanathan.an@gmail.com liub@cs.uic.edu alokchoudhary01@gmail.com</p></frontmatter><abstract>This paper studies sentiment analysis of condi­tional sentences. The aim is to determine whether opinions expressed on different topics in a conditional sentence are positive, negative or neutral. Conditional sentences are one of the commonly used language constructs in text. In a typical document, there are around 8% of such sentences. Due to the condition clause, sentiments expressed in a conditional sentence can be hard to determine. For example, in the sentence, <i>if your Nokia phone is not good, buy this great Samsung phone, </i>the author is posi­tive about <i>"Samsung phone" </i>but does not ex­press an opinion on <i>"Nokia phone" </i>(although the owner of the <i>"Nokia phone" </i>may be nega­tive about it). However, if the sentence does not have <i>"if", </i>the first clause is clearly nega­tive. Although <i>"if" </i>commonly signifies a con­ditional sentence, there are many other words and constructs that can express conditions. This paper first presents a linguistic analysis of such sentences, and then builds some super­vised learning models to determine if senti­ments expressed on different topics in a condi­tional sentence are positive, negative or neu­tral. Experimental results on conditional sen­tences from 5 diverse domains are given to demonstrate the effectiveness of the proposed approach. </abstract></header><body><section number="1" title="Introduction"><p>Sentiment analysis (also called opinion mining) has been an active research area in recent years. There are many research directions, e.g., senti­ment classification (classifying an opinion doc­ument as positive or negative) (e.g., Pang, Lee and Vaithyanathan, 2002; Turney, 2002), subjec­tivity classification (determining whether a sen­tence is subjective or objective, and its associated opinion) (Wiebe and Wilson, 2002; Yu and Hat-zivassiloglou, 2003; Wilson et al, 2004; Kim and</p><p>* This work was done when Bing Liu was on sabbatical leave at Northwestern University.</p><p>Hovy, 2004; Riloffand Wiebe, 2005), fea­ture/topic-based sentiment analysis (assigning positive or negative sentiments to topics or prod­uct features) (Hu and Liu 2004; Popescu and Et-zioni, 2005; Carenini et al., 2005; Ku et al., 2006; Kobayashi, Inui and Matsumoto, 2007; Titov and McDonald. 2008). Formal definitions of different aspects of the sentiment analysis problem and discussions of major research direc­tions and algorithms can be found in (Liu, 2006; Liu, 2009). A comprehensive survey of the field can be found in (Pang and Lee, 2008).</p><p>Our work is in the area of topic/feature-based sentiment analysis or opinion mining (Hu and Liu, 2004). The existing research focuses on solving the general problem. However, we argue that it is unlikely to have a one-technique-fit-all solution because different types of sentences ex­press sentiments/opinions in different ways. A divide-and-conquer approach is needed, e.g., fo­cused studies on different types of sentences. This paper focuses on one type of sentences, i.e., conditional sentences, which have some unique characteristics that make it hard to determine the orientation of sentiments on topics/features in such sentences. By <i>sentiment orientation, </i>we mean positive, negative or neutral opinions. By <i>topic, </i>we mean the target on which an opinion has been expressed. In the product domain, a top­ic is usually a <i>product feature </i>(i.e., a component or attribute). For example, in the sentence, <i>I do not like the sound quality, but love the design of this MP3 player, </i>the product features (topics) are <i>"sound quality" </i>and <i>"design" </i>of the MP3 player as opinions have been expressed on them. The sentiment is positive on <i>"design" </i>but negative on <i>"sound quality".</i></p><p>Conditional sentences are sentences that de­scribe implications or hypothetical situations and their consequences. In the English language, a variety of conditional connectives can be used to form these sentences. A conditional sentence contains two clauses: the condition clause and</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">180</doubt><p><i>Proceedings of the 2009 Conference on Empirica</i><i>l </i><i>Methods in Natura</i><i>l </i><i>Language Processing, </i>pages 180-189, Singapore, 6-7 August 2009. ©2009 ACL and AFNLP<page local="2" global="181"/></p><p>the consequent clause, that are dependent on each other. Their relationship has significant im­plications on whether the sentence describes an opinion. One simple observation is that senti­ment words (also known as opinion words) (e.g., <i>great, beautiful, bad) </i>alone cannot distinguish an opinion sentence from a non-opinion one. A conditional sentence may contain many senti­ment words or phrases, but express no opinion.</p><p><b>Example 1: </b><i>If someone makes a beautiful and reliable car, I will buy it </i>expresses no sentiment towards any particular car, although <i>"beautiful" </i>and <i>"reliable" </i>are positive sentiment words.</p><p>This, however, does not mean that a condition­al sentence cannot express opinions/sentiments.</p><p><b>Example 2: </b><i>If your Nokia phone is not good, buy this great Samsung phone </i>is positive about the <i>"Samsung phone" </i>but does not express an opinion on the <i>"Nokia phone" </i>(although the owner of the <i>"Nokia phone" </i>may be negative about it). Clearly, if the sentence does not have <i>"if, </i>the first clause is negative. Hence, a method for determining sentiments in normal sentences will not work for conditional sentences. The ex­amples below further illustrate the point.</p><p>In many cases, both the condition and conse­quent together determine the opinion.</p><p><b>Example 3: </b><i>If you are looking for a phone with good voice quality, don't buy this Nokia phone </i>is negative about the <i>"voice quality" </i>of the "Nokia phone", although there is a positive sen­timent word <i>"good" </i>in the conditional clause modifying <i>"voice quality". </i>However, in the fol­lowing example, the opinion is just the opposite.</p><p><b>Example 4: </b><i>If you want a phone with good voice quality, buy this Nokia phone </i>is positive about the <i>"voice quality" </i>of the "Nokia phone".</p><p>As we can see, sentiment analysis of condi­tional sentences is a challenging problem.</p><p>One may ask whether there is a large percen­tage of conditional sentences to warrant a fo­cused study. Indeed, there is a fairly large pro­portion of such sentences in evaluative text. They can have a major impact on the sentiment analy­sis accuracy. Table 1 shows the percentage of conditional sentences (sentences containing the words <i>if, unless, assuming, </i>etc) and also the total number of sentences from which we computed the percentage in several user-forums. The fig­ures definitely suggest that there is considerable benefit to be gained by developing techniques that can analyze conditional sentences.</p><table caption="Table 1: Percent of conditional sentences"></table><p>To the best of our knowledge, there is no fo­cused study on conditional sentences. This paper makes such an attempt. Specifically, we deter­mine whether a conditional sentence (which is also called a <i>conditional </i>in the linguistic litera­ture) expresses positive, negative or neutral opi­nions on some topics/features. Since our focus is on studying how conditions and consequents af­fect sentiments, we assume that <i>topics </i>are given, which are <i>product attributes </i>since our data sets are user comments on different products.</p><p>Our study is conducted from two perspectives. We start with the linguistic angle to gain a good understanding of existing work on different types of conditionals. As conditionals can be expressed with other words or phrases than <i>if, </i>we will study how they behave compared to <i>if. </i>We will also show that the distribution of these conditionals based on our data sets.</p><p>With the linguistic knowledge, we perform a computational study using machine learning. A set of features for learning is designed to capture the essential determining information. Note that the features here are data attributes used in learn­ing rather than product attributes or features. Three classification strategies are designed to study how to best perform the classification task due to the complex situation of two clauses and their interactions in conditional sentences. These three classification strategies are <i>clause-based, consequent-based </i>and <i>whole-sentence-based. </i>Clause-based classification classifies each clause separately and then combines their results. Con­sequent-based classification only uses conse­quents for classification as it is observed that in conditional sentences, it is often the consequents that decide the opinion. Whole-sentence-based classification treats the entire sentence as a whole in classification. Experimental results on condi­tional sentences from diverse domains demon­strate the effectiveness of these classification models. The results indicate that the whole-sentence-based classifier performs the best.</p><p>Since this paper only studies conditional sen­tences, a natural question is whether the pro­posed technique can be easily integrated into an overall sentiment analysis or opinion mining sys­tem. The answer is yes because a large propor­tion of conditional sentences can be detected us­ing conditional connectives. Keyword search is thus sufficient to identify such sentences for spe­cial handling using the proposed approach.<page local="3" global="182"/> There are, however, some subtle conditionals which do not use normal conditional connectives and will need an additional module to identify them, but such sentences are very rare as Table 2 indicates.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">181</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Source</b></p></td><td class="cell"><p><b>% of cond. (total #. of sent.)</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Cellphone</p></td><td class="cell"><p>8.6 (47711)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Automobile</p></td><td class="cell"><p>5.0 (8113)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>LCD TV</p></td><td class="cell"><p>9.92 (258078)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Audio Systems</p></td><td class="cell"><p>8.1 (5702)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Medicine</p></td><td class="cell"><p>8.29 (160259)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><section number="2" title="The Problem Statement"><p>The paper follows the <i>feature-based sentiment analysis </i>model in (Hu and Liu 2004; Popescu and Etzioni, 2005). We are particularly interested in sentiments on products and services, which are called objects or entities. Each object is de­scribed by its <i>parts </i>and <i>attributes, </i>which are col­lectively called <i>features </i>in (Hu and Liu, 2004; Liu, 2006). For example, in the sentence, <i>If this camera has great picture quality, I will buy it, "picture quality" </i>is a feature of the camera. For formal definitions of objects and features, please refer to (Liu, 2006; Liu, 2009). In this paper, we use the term <i>topic </i>to mean <i>feature </i>as the feature here can confuse with the feature used in ma­chine learning. The term topic has also been used by some researchers (e.g., Kim and Hovy, 2004; Stoyanov and Cardie, 2008).</p><p>Our objective is to predict the sentiment orientation (positive, negative or neutral) on each topic that has been commented on in a sentence.</p><p>The problem of automatically identifying fea­tures or topics being spoken about in a sentence has been studied in (Hu and Liu, 2004; Popescu and Etzioni, 2005; Stoyanov and Cardie, 2008). In this work, we do not attempt to identify such topics automatically. Instead, we assume that they are given because our objective is to study how the interaction of the condition and conse­quent clauses affects sentiments. For this pur­pose, we manually identify all the topics.</p></section><section number="3" title="Conditional Sentences"><p>This section presents the linguistic perspective of conditional sentences.</p><table caption="Table 2: Percentage of sentences with some main conditional connectives"></table><subsection number="3.1" title="Conditional Connectives"><p>A large majority of conditional sentences are introduced by the subordinating conjunction <i>If. </i>However, there are also many other conditional connectives, e.g., <i>even if, unless, in case, assum­ing/supposing, as long as, </i>etc. Table 2 shows the distribution of conditional sentences with various connectives in our data. Detailed linguistic dis­cussions of them are beyond the scope of this paper. Interested readers, please refer to (Dec-lerck and Reed, 2001). Below, we briefly discuss some important ones and their interpretations.</p><p><i>If: </i>This is the most commonly used conditional connective. In addition to its own usage, it can also be used to replace other conditional connec­tives, except some semantically richer connec­tives (Declerck and Reed, 2001). Most (but not all) conditional sentences can be logically ex­pressed in the form 'If <i>P </i>then Q', where <i>P </i>is the condition clause and <i>Q </i>is the consequent clause. For practical purposes, we can automatically segment the condition and consequent clauses using simple rules generated by observing grammatical and linguistic patterns.</p><p><i>Unless: </i>Most conditional sentences containing <i>unless </i>can be replaced with equivalent sentences with an <i>if </i>and a <i>not. </i>For example, the sentence <i>Unless you need clarity, buy the cheaper model </i>can be expressed with <i>If you don't need clarity, buy the cheaper model.</i></p><p><i>Even if: </i>Linguistic theories claim that <i>even if </i>is a special case of a conditional which may not always imply an if-then relationship (Gauker 2005). However, in our datasets, we have ob­served that the usage of <i>even if </i>almost always translates into a conditional. Replacing <i>even if </i>by <i>if </i>will yield a sentence that is semantically simi­lar enough for the purpose of sentiment analysis.</p><p><i>Only if, provided/providing that, on condition that: </i>Conditionals involving these phrases typi­cally express a necessary condition, e.g., <i>I will buy this camera only if they can reduce the price. </i>In such sentences, <i>only </i>usually does not affect whether the sentence is opinionated or not.</p><p><i>In case: </i>Conditional sentences containing <i>in case </i>usually describe a precaution <i>(I will close the window in case it rains), </i>prevention <i>(I wore sunglasses in case I was recognized), </i>or a relev­ance conditional <i>(In case you need a car, you can rent one). </i>Identifying the conditional and conse­quent clauses is not straightforward in many cas­es. Further, in these instances, replacing <i>in case </i>with <i>if </i>may not convey the intended meaning of the conditional. We have ignored these cases in our analysis as we believe that they need a sepa­rate study, and also such sentences are rare.<page local="4" global="183"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">182</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Conditional Connective</b></p></td><td class="cell"><p><b>% of sentences</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>If</p></td><td class="cell"><p>6.42</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Unless</p></td><td class="cell"><p>0.32</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Even if</p></td><td class="cell"><p>0.17</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Until</p></td><td class="cell"><p>0.10</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>As (so) long as</p></td><td class="cell"><p>0.09</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Assuming/supposing</p></td><td class="cell"><p>0.04</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>In case</p></td><td class="cell"><p>0.04</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Only if</p></td><td class="cell"><p>0.03</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p><i>As (so) long as: </i>Sentences with these connec­tives behave similarly to <i>if </i>and can usually be replaced with <i>if.</i></p><p><i>Assuming/Supposing: </i>These are a category of conditionals that behave quite differently. The participles <i>supposing </i>and <i>assuming </i>create condi­tional sentences where the conditional clause and the consequent clause can be syntactically inde­pendent. It is quite difficult to distinguish those conditional sentences which contain an explicit consequent clause and fit within our analysis framework. In our data, most of such sentences have no consequent, thus representing assump­tions rather than opinions. We omit these sen­tences in our study (they are also rare).</p></subsection><subsection number="3.2" title="Types of Conditionals"><p>There are extensive studies of conditional sen­tences (also known as <i>conditionals) </i>in linguis­tics. Various theories have led to a number of classification systems. Popular types of condi­tionals include <i>actualization conditionals, infe­rential conditionals, implicative conditionals, </i>etc (Declerck and Reed, 2001). However, these clas­sifications are mainly based on semantic mean­ings which are difficult to recognize by a com­puter program. To build classification models, we instead exploit <i>canonical tense patterns </i>of conditionals, which are often used in pedagogic grammar books. They are defined based on tense and are associated with general meanings. How­ever, as described in (Declerck and Reed, 2001), their meanings are much more complex and nu­merous than their associated general meanings. However, the advantage of this classification is that different types can be detected easily be­cause they depend on tense which can be pro­duced by a part-of-speech tagger. As we will see in Section 5, canonical tense patterns help senti­ment classification significantly. Below, we in­troduce the four canonical tense patterns.</p><p><b>Zero Conditional: </b>This conditional form is used to describe universal statements like facts, rules and certainties. In a zero conditional, both the condition and consequent clauses are in the simple present tense. An example of such sen­tences is: <i>If you heat water, it boils.</i></p><p><b>First Conditional: </b>Conditional sentences of this type are also called potential or indicative conditionals. They are used to express a hypo­thetical situation that is probably true, but the truth of which is unverified. In the first conditional, the condition is in the simple present tense, and the consequent can be either in past tense or present tense, usually with a modal aux­iliary verb preceding the main verb, e.g., <i>If the acceleration is good, I will buy it.</i><i></i></p><p><b>Second Conditional: </b>This is usually used to describe less probable situations, for stating pre­ferences and imaginary events. The condition clause of a second conditional sentence is in the past subjunctive (past tense), and the consequent clause contains a conditional verb modifier (like <i>would, should, might), </i>in addition to the main verb, e.g., <i>If the cell phone was robust, I would consider buying it.</i></p><p><b>Third conditional: </b>This is usually used to de­scribe contrary-to-fact (impossible) past events. The past perfect tense is used in the condition clause, and the consequent clause is in the present perfect tense, e.g., <i>If I had bought the a767, I would have hated it.</i></p><p>Based on the above definitions, we have devel­oped approximate part-of-speech (POS) tags <footnote anchor="1"/> for the condition and the consequent of each pattern (Table 3), which do not cover all sentences, but overall they cover a majority of the sentences. For those not covered cases, the problem is mainly due to incomplete sentences and wrong grammars, which are typical for informal writ­ings in forum postings and blogs. For example, the sentence, <i>Great car if you need powerful ac­celeration, </i>does not fall into any category, but it actually means <i>It is a great car if you need po­werful acceleration, </i>which is a zero conditional. To handle such sentences, we designed a set of rules to assign them some default types:</p><p>If condition contains VB/VBP/VBZ — 0 conditional If consequent contains VB/VBP/VBS — 0 conditional If condition contains VBG — 1st conditional If condition contains VBD — 2nd conditional If conditional contains VBN — 3rd conditional.</p><table caption="Table 3: Tenses for identifying conditional types"></table><p>The list of Part-Of-Speech (POS) tags can be found at: http://www.ling.upenn.edu/courses/Fall_2003/ling001/ penn_treebank_pos.html<page local="5" global="184"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">183</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Type</p></td><td class="cell"><p>Linguistic Rule</p></td><td class="cell"><p>Condition POS tags</p></td><td class="cell"><p>Consequent POS tags</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>0</p></td><td class="cell"><p>If + simple present — simple present</p></td><td class="cell"><p>VB/VBP/VBZ</p></td><td class="cell"><p>VB/VBP/ VBZ</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p>If + simple present — will + bare infinitive</p></td><td class="cell"><p>VB/VBP/VBZ /VBG</p></td><td class="cell"><p>MD + VB</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2</p></td><td class="cell"><p>If + past tense — would + infinitive</p></td><td class="cell"><p>VBD</p></td><td class="cell"><p>MD + VB</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3</p></td><td class="cell"><p>If + past perfect — present perfect</p></td><td class="cell"><p>VBD+VBN</p></td><td class="cell"><p>MD + VBD</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>By using these rules, we can increase the sen­tence coverage from 73% to 95%.</p></subsection></section><section number="4" title="Sentiment Analysis of Conditionals"><p>We now describe our computational study. We take a machine learning approach to predict sen­timent orientations. Below, we first describe fea­tures used and then classification strategies.</p><subsection number="4.1" title="Feature construction"><p>I. <i>Sentiment words/phrases and their locations: </i>Sentiment words are words used to express positive or negative opinions, which are in­strumental for sentiment classification for ob­vious reasons. We obtained a list of over 6500 sentiment words gathered from various sources. The bulk of it is from http://www.cs.pitt.edu/mpqa. We also added some of our own. Our list is mainly from the work in (Hu and Liu, 2004; Ding, Liu and Yu, 2008). In addition to words, there are phrases that describe opinions. We have identified a set of such phrases. Although obtaining these phrases was time-consuming, it was only a one-time effort. We will make this list availa­ble as a community resource. It is possible that there is a better automated method for finding such phrases, such as the methods in (Kanayama and Nasukawa, 2006; Breck, Choi and Cardie, 2007). However, automatically generating sentiment phrases has not been the focus of this work as our objective is to study how the two clauses interact to determine opinions given the sentiment words and phrases are known. Our list of phrases is by no means complete and we will continue to expand it in the future.</p><p>For each sentence, we also identify wheth­er it contains sentiment words/phrases in its condition or consequent clause. It was ob­served that the presence of a sentiment word/phrase in the consequent clause has more effect on the sentiment of a sentence.</p><p>II. <i>POS tags of sentiment words: </i>Sentiment words may be used in several contexts, not all of which may correspond to an opinion. For example, <i>I trust Motorola </i>and <i>He has a trust fund </i>both contain the word <i>trust. </i>But only the former contains an opinion. In such cases, the POS tags can provide useful information.</p><p>III. <i>Words indicating no opinion: </i>Similar to how sentiment words are related to opinions, there are also a number of words which imply the opposite. Words like <i>wondering, thinking, de-</i> <i>bating </i>are used when the user is posing a question or expressing doubts. Thus such phrases usually do not contribute an opinion, especially if they are in the vicinity of the <i>if </i>connective. We search a window of 3 words on either side of <i>if </i>to determine if there is any such word. We have compiled a list of these words as well and use it in our experiments.</p><p>IV. <i>Tense patterns: </i>These are the canonical tense patterns in Section 3.2. They are used to gen­erate a set of features. We identify the first verb in both the condition and consequent clauses by searching for the relevant POS tags in Table 3. We also search for the words pre­ceding the main verb to find modal auxiliary verbs, which are also used as features.</p><p>V. <i>Special characters: </i>The presence or absence of '?' and '!'.</p><p>VI. <i>Conditional connectives: </i>The conditional connective used in the sentence <i>(if, even if, unless, only if, </i>etc) is also taken as a feature.</p><p>VII. <i>Length of condition and consequent clauses: </i>Using simple linguistic and punctuation rules, we automatically segment a sentence into condition and consequent clauses. The num­bers of words in the condition and consequent clauses are then used as features. We ob­served that when the condition clause is short, it usually has no impact on whether the sen­tence expresses an opinion.</p><p>VIII. <i>Negation words: </i>The use of negation words like <i>not, don't, never, </i>etc, often alter the sen­timent orientation of a sentence. For example, the addition of <i>not </i>before a sentiment word can change the orientation of a sentence from positive to negative. We consider a window of 3-6 words before an opinion word, and search for these kinds of words.</p><p>The following two features are singled out for easy reference later. They are only used in one classification strategy. The first feature is an in­dicator, and the second feature has a parameter (which will be evaluated separately).</p><p>(1) . <i>Topic location: </i>This feature indicates whether the topic is in the conditional clause or the consequent clause.</p><p>(2) . <i>Opinion weight: </i>This feature considers only sentiment words in the vicinity of the topic, since they are more likely to influence the opinion on the topic. A window size is used to control what we mean by vicinity. The fol­lowing formula is used to assign a weight to each sentiment word, which is inversely pro­portional to the distance <i>(Dop) </i>of the senti­ment word to the topic mention. Sentiment value is +1 for a positive word and -1 for a negative word.<page local="6" global="185"/> <i>Sentwords </i>are the set of known sentiment words and phrases. „ ± 1</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">184</doubt><doubt alpha="64.3" length="28" tooSmall="False" monospace="0.0">weight =2^-,\/opg{sentwords}</doubt><doubt alpha="100.0" length="5" tooSmall="False" monospace="0.0">opDop</doubt></subsection><subsection number="4.2" title="Classification Strategies"><p>Since we are interested in topic-based sentiment analysis, how to perform classification becomes an interesting issue. Due to the two clauses, it may not be sufficient to classify the whole sen­tence as positive or negative as in the same sen­tence, some topics may be positive and some may be negative. We propose three strategies.</p><p><b>Clause-based classification: </b>Since there are two clauses in a conditional sentence, in this case we build two classifiers, one for the condition and one for the consequent. <b><i>Condition classifier: </i></b>This method classifies the condition clause as expressing positive, nega­tive or neutral opinion.</p><p><i>Training data: </i>Each training sentence is represented as a feature vector. Its class is posi­tive, negative or neutral depending on whether the consequent clause is positive, negative or neutral while considering both clauses.</p><p><i>Testing: </i>For each test sentence, the resulting classifier predicts the opinion of the conse­quent clause.</p><p><i>Topic class prediction: </i>To predict the opi­nion on a topic, if the topic is in the consequent clause, it takes the predicted class of the clause.</p><p><b>Consequent classifier: </b>This classifier classi­fies the consequent clause as expressing posi­tive, negative or neutral opinion.</p><p>The combination of these two classifiers is called the <i>clause-based classifier. </i>It works as follows: If a topic is in the conditional clause, the condition classifier is used, and if a topic is in the consequent clause, the consequent clas­sifier is used.</p><p><b>Consequent-based classification: </b>It is observed that in most cases, the condition clause con­tains no opinion whereas the consequent clause reflects the sentiment of the entire sentence. Thus, this method uses (in a different way) on­ly the above consequent classifier. If it classi­fies the consequent of a testing conditional sentence as positive, all the topics in the whole sentence are assigned the positive orientation, and likewise for negative and neutral.</p><p><b>Whole-sentence-based classification: </b>In this case, a single classifier is built to predict the opinion on each topic in a sentence.</p><p><i>Training data: </i>In addition to the normal fea­tures, the two features (1) and (2) in Section 4.1 are used for this classifier. If a sentence contains multiple topics, multiple training in­stances of the same sentence are created in the training data. Each instance represents one specific topic. The class of the instance de­pends on whether the opinion on the topic is positive, negative or neutral.</p><p><i>Testing: </i>For each topic in each test sentence, the resulting classifier predicts its opinion.</p><p><i>Topic class prediction: </i>This is not needed as the prediction has been done in testing.</p></subsection></section><section number="5" title="Results and Discussions 5.1    Data sets"><p>Our data consists of conditional sentences from 5 different user forums: Cellphone, Automobile, LCD TV, Audio systems and Medicine. We ob­tained user postings from these forums and ex­tracted the conditional sentences. We then ma­nually annotated 1378 sentences from this cor­pus. We also annotated the conditional and con­sequent clauses and identified the topics (or product features) being commented upon, and their sentiment orientations. In our annotation, we observed that sentences with no sentiment words or phrases almost never express opinions, i.e., only around 3% of them express opinions. There are around 26% sentences containing no sentiment words or phrases in our data. To make the problem challenging, we restrict our attention to only those sentences that contain at least one sentiment word or phrase. We have annotated topics from around 900 such sentences. Table 4 shows the class distributions of this data. At the clause level (topics are not considered), we ob­serve that conditional clauses contain few opi­nions. At the topic-level, 43.5% of the topics have positive opinions, 26.4% of the topics have negative opinions, and the rest have no opinions.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">185</doubt><page local="7" global="186"/><table caption="Table 4: Distribution of classes"></table><p>For the annotation of data, we assume that topics are known. One student annotated the top­ics first. Then two students annotated the senti­ments on the topics. If a student found that a top­ic annotation is wrong, he will let us know. Some mistakes and missing topics were found but there were mainly due to oversights rather than disa­greements. The agreement on sentiment annota­tions were computed using the Kappa score. We achieved the Kappa score of 0.63, which indi­cates strong agreements. The conflicting cases were then solved through discussion to reach consensus. We did not find anything that the an-notators absolutely disagree with each other.</p><subsection number="5.2" title="Experimental results"><p>We now present the results for different combi­nations of features and classification strategies. For model building, we used Support Vector Machines (SVM), and the LIBSVM implementa­tion (Chang and Lin, 2001) with a Gaussian ker­nel, which produces the best results. All the re­sults are obtained via 10-fold cross validation.</p><p><b>Two-class classification: </b>We first discuss the results for a simpler version of the problem that involves only sentences with positive or negative orientations on some topics (at least one of the clauses must have a positive/negative opinion on a topic). Neutral sentences are not used (~28% of the total). The results of all three classifiers are given in Table 5. The feature sets have been de­scribed in Section 4.1. For all the experiments below, features (1) and (2) are only used by the whole-sentence-based classifier, but not used by the other two classifiers for obvious reasons.</p><p>{I+II}: This setting uses sentiment words and phrases, their positions and POS tags as features (we used Brill's POS tagger). This can be seen as the baseline. We observe that both the conse­quent-based and whole-sentence-based classifiers perform dramatically better than the clause-based classifier. The consequent-based classifier and the whole-sentence-based classifier perform si­milarly (with the latter being slightly better). The precision, recall, and F-score are computed as the average of the two classes.</p><p>{I+II+III}: In this setting, the list of special non-sentiment related words is added to the feature set. All three classifiers improve slightly.</p><p>{I+II+III+IV}: This setting includes all the ca­nonical tense based features. We see marked im­provements for the consequent-based and whole-sentence-based classifiers both in term of accura­cy and F-score, which are statistically significant compared to those of {I+II+III} at the 95% con­fidence level based on paired t-test.</p><p>All: When all the features are used, the results of all the classifiers improve further.</p><p>Two main observations worth mentioning:</p><p>1. Both the consequent-based and whole-sentence-based classifiers outperform the clause-based classifier dramatically. This con­firms our observation that the consequent usually plays the key role in determining the sentiment of the sentence. This is further rein­forced by the fact that the consequent-based classifier actually performs similarly to the whole-sentence-based classifier. The condi­tion clause seems to give no help.</p><p>2. The second observation is that the linguistic knowledge of canonical tense patterns helps significantly. This shows that the linguistic knowledge is very useful.</p><p>We also noticed that many misclassifications are caused by grammatical errors, use of slang phrases and improper punctuations, which are typical of postings on the Web. Due to language irregularities (e.g., wrong grammar, missing punctuations, sarcasm, exclamations), the POS tagger makes many mistakes as well causing some errors in the tense based features.</p><p><b>Three-class classification: </b>We now move to the more difficult and realistic case of three classes: positive, negative and neutral (no-opinion). Ta­ble 6 shows the results. The trend is similar ex­cept that the whole-sentence-based classifier now performs markedly better than the consequent-based classifier. We believe that this is because the neutral class needs information from both the condition and consequent clauses. This is evident from the fact that there is little or no improve­ment after {I+II} for the consequent-based clas­sifier. We also observe that the accuracies and F-scores for the three-class classification are lower than those for the two-class classification. This is understandable due to the difficulty of determin­ing whether a sentence has opinion or not. Again, statistical test shows that the canonical tense-based features help significantly.</p><p>As mentioned in Section 4.1, the whole-sentence-based classifier only considers those sentiment words in the vicinity of the topic under<page local="8" global="187"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">186</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Positive</p></td><td class="cell"><p>Negative</p></td><td class="cell"><p>Neutral</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Condition</p></td><td class="cell"><p>6.9%</p></td><td class="cell"><p>6.7%</p></td><td class="cell"><p>86.4%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Consequent</p></td><td class="cell"><p>49.3%</p></td><td class="cell"><p>16.5%</p></td><td class="cell"><p>34%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Topic-level</p></td><td class="cell"><p>43.5%</p></td><td class="cell"><p>26.4%</p></td><td class="cell"><p>29.9%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">1</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">2</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">3</doubt><table caption="Table 5: Two-class classification - positive and negative"></table><p>Window size investigation. For this, we search a window of <i>n </i>words on either side of the topic mention. To study the effect of varying <i>n, </i>we performed an experiment with various values of the window size and measured the overall accuracy for each case. Table 7 shows how the accuracy changes as we increase the window size. We found that a window size of 6-10 yielded good accuracies. This is because lower values of <i>n </i>lead to loss of information regarding sentiment words as some sentiment words could be far from the topic. We finally used 8, which gave the best results.</p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">10</doubt><doubt alpha="8.1" length="99" tooSmall="False" monospace="0.0">Accuracy    | 66.1 |  62.6  |  64.1   |  64.8  |  65.3   |  65.7  |   66.3   |67.3|   66.9   | 66.8</doubt><p>We also investigated ways of using the nega­tion word in the sentence to correctly predict the sentiment. One method is to use the negation word as a feature, as described in Section 4.1. Another technique is to reverse the orientation of the prediction for those sentences which contain negation words. We found that the former tech­nique yielded better results. The results reported so far are based on the former approach.</p></subsection></section><section number="6" title="Related Work"><p>There are several research directions in sentiment analysis (or opinion mining). One of the main directions is sentiment classification, which clas­sifies the whole opinion document (e.g., a prod­uct review) as positive or negative (e.g., Pang et al, 2002; Turney, 2002; Dave et al, 2003; Ng et al. 2006; McDonald et al, 2007). It is clearly different from our work as we are interested in con­ditional sentences.</p><p>Another important direction is classifying sentences as subjective or objective, and classify­ing subjective sentences or clauses as positive or negative (Wiebe et al, 1999; Wiebe and Wilson, 2002, Yu and Hatzivassiloglou, 2003; Wilson et al, 2004; Kim and Hovy, 2004; Riloffand Wiebe, 2005; Gamon et al 2005; McDonald et al, 2007). Although these works deal with sen­tences, they aim to solve the general problem. This paper argues that there is unlikely a one-technique-fit-all solution, and advocates dealing with specific types of sentences differently by exploiting their unique characteristics. Condi­tional sentences are the focus of this paper. To the best of our knowledge, there is no focused study on them.</p><doubt alpha="64.8" length="193" tooSmall="False" monospace="0.0">Several researchers also studied feature/topic-based sentiment analysis (e.g., Hu and Liu, 2004; Popescu and Etzioni, 2005; Ku et al, 2006; Care-nini et al, 2006; Mei et al, 2007; Ding, Liu and</doubt><doubt alpha="53.3" length="45" tooSmall="False" monospace="0.0">Yu, 2008; Titov and R. McDonald, 2008; Stoya-</doubt><p>nov and Cardie, 2008; Lu and Zhai, 2008). Their objective is to extract topics or product features in sentences and determine whether the senti­ments expressed on them are positive or nega­tive. Again, no focused study has been made to handle conditional sentences. Effectively han­dling of conditional sentences can help their ef­fort significantly.</p><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">4</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">5</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">6</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">7</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">9</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">8</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">187</doubt><table caption="Table 7: Accuracy of the whole-sentence-based classifier with varying window sizes(n)" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b><i>Clause-based classifier</i></b></p></td><td class="cell"><p><b><i>Consequent-based classifier</i></b></p></td><td class="cell"><p><b><i>Whole-sentence-based classifier</i></b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Acc. Prec. Rec. <b>F</b></p></td><td class="cell"><p>Acc.   Prec.   Rec. <b>F</b></p></td><td class="cell"><p>Acc. Prec. Rec. <b>F</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>I+II (senti. words+POS)</p></td><td class="cell"><p>39.9   42.8   34.0 <b>37.9</b></p></td><td class="cell"><p>69.1    72.9   67.1 <b>69.8</b></p></td><td class="cell"><p>68.9   73.7 68.13 <b>70.8</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>I+II+III (+ non-senti. words)</p></td><td class="cell"><p>41.5   44.9   37.1 <b>40.6</b></p></td><td class="cell"><p>69.3    73.9   66.3 <b>69.9</b></p></td><td class="cell"><p>69.2   73.7   63.5 <b>71.0</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>I+II+III+IV (+ tenses)</p></td><td class="cell"><p>42.7   45.2   38.5 <b>41.6</b></p></td><td class="cell"><p>72.7    76.4   72.0 <b>74.1</b></p></td><td class="cell"><p>71.1   77.9   72.2 <b>74.9</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>All</p></td><td class="cell"><p>43.2   46.1   38.9 <b>42.2</b></p></td><td class="cell"><p>73.3    77.0   72.7 <b>74.8</b></p></td><td class="cell"><p>72.3   77.8   73.6 <b>75.6</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Table 6: Three-class classification - positive, negative and neutral (no opinion)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b><i>Clause-based classifier</i></b></p></td><td class="cell"><p><b><i>Consequent-based classifier</i></b></p></td><td class="cell"><p><b><i>Whole-sentence-based classifier</i></b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Acc. Prec. Rec. <b>F</b></p></td><td class="cell"><p>Acc.   Prec.   Rec. <b>F</b></p></td><td class="cell"><p>Acc. Prec. Rec. <b>F</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>I+II (senti. words+POS)</p></td><td class="cell"><p>45.2   41.3   35.1 <b>37.9</b></p></td><td class="cell"><p>54.6    57.7   52.9 <b>55.2</b></p></td><td class="cell"><p>59.1   58.1   56.4 <b>57.2</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>I+II+III (+ non-senti. words)</p></td><td class="cell"><p>46.9   42.8   37.8 <b>40.1</b></p></td><td class="cell"><p>55.3    60.0   51.3 <b>55.3</b></p></td><td class="cell"><p>61.4   60.1   60.8 <b>60.4</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>I+II+III+IV (+ tenses)</p></td><td class="cell"><p>50.3   48.7   40.9 <b>44.5</b></p></td><td class="cell"><p>57.3    64.0   50.0 <b>56.1</b></p></td><td class="cell"><p>64.6   63.3   63.9 <b>63.6</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>All</p></td><td class="cell"><p>53.3   49.8   44.1 <b>46.8</b></p></td><td class="cell"><p>58.7    64.5   50.1 <b>56.4</b></p></td><td class="cell"><p>67.8   66.9   65.1 <b>66.0</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="9" global="188"/><p>In this work, we used many sentiment words and phrases. These words and phrases are usually compiled using different approaches (Hatzivassi-loglou and McKeown, 1997; Kaji and Kitsure-gawa, 2006; Kanayama and Nasukawa, 2006; Esuli and Sebastiani, 2006; Breck et al, 2007; Ding, Liu and Yu. 2008; Qiu et al, 2009). There are several existing lists produced by researchers. We used the one from the MPQA corpus (http://www.cs.pitt.edu/mpqa) with added phras­es of our own from (Ding, Liu and Yu. 2008). In our work, we also assume that the topics are known. (Hu and Liu, 2004; Popescu and Etzioni, 2005; Kobayashi, Inui and Matsumoto, 2007; Stoyanov and Cardie, 2008) have studied top­ic/feature extraction.</p><p>One existing focused study is on comparative and superlative sentences (Jindal and Liu, 2006; Bos and Nissim, 2006; Fiszman et al, 2007; Ga-napathibhotla and Liu, 2008). Their work identi­fies comparative sentences, extracts comparative relations in the sentences and analyzes compara­tive opinions (Ganapathibhotla and Liu, 2008). An example comparative sentence is <i>"Honda looks better than Toyota". </i>As we can see, com­parative sentences are entirely different from conditional sentences. Thus, their methods can­not be directly applied to conditional sentences.</p></section><section number="7" title="Conclusion"><p>To perform sentiment analysis accurately, we argue that a divide-and-conquer approach is needed, i.e., focused study on each type of sen­tences. It is unlikely that there is a one-size-fit-all solution. This paper studied one type, i.e., condi­tional sentences, which have some unique cha­racteristics that need special handling. Our study was carried out from both the linguistic and computational perspectives. In the linguistic study, we focused on canonical tense patterns, which have been showed useful in classification. In the computational study, we built SVM mod­els to automatically predict whether opinions on topics are positive, negative or neutral. Experi­mental results have shown the effectiveness of the models.</p><p>In our future work, we will further improve the classification accuracy and study related problems, e.g., identifying topics/features. Al­though there are some special conditional sen­tences that do not use easily recognizable condi­tional connectives and identifying them are use­ful, such sentences are very rare and spending time and effort on them may not be cost-effective at the moment.</p></section><section title="Acknowledgements"><p>This work was supported in part by DOE SCI-DAC-2: Scientific Data Management Center for Enabling Technologies (CET) grant DE-FC020621443, NSF SDCI OCI-0724599, and NSF ST-HEC CCF-0444405.</p><doubt alpha="63.2" length="38" tooSmall="False" monospace="0.0">07ER25808, DOE FASTOS award number DE-</doubt><doubt alpha="43.2" length="37" tooSmall="False" monospace="0.0">FG02-08ER25848,    NSF    HECURA CCF-</doubt></section><references><p>J. Bos, and M. Nissim. 2006. An Empirical Ap­proach to the Interpretation of Superlatives.</p><p>EMNLP-2006.</p><p>E. Breck, Y. Choi, and C. Cardie. 2007. Identify­ing expressions of opinion in context, IJCAI2007.</p><doubt alpha="52.4" length="42" tooSmall="False" monospace="0.0">C.-C. Chang and C.-J. Lin. 2001. LIBSVM: a</doubt><p>library for support vector machines. http://www.csie.ntu.edu.tw /~cjlin/libsvm</p><p>G. Carenini, R. Ng, and A. Pauls. 2006. Interac­tive Multimedia Summaries of Evaluative</p><p>Text. IUI-2006.</p><p>C. Gauker. 2005. Conditionals in Context. MIT</p><p>Press.</p><p>D. Dave, A. Lawrence, and D. Pennock. 2003. Mining the Peanut Gallery: Opinion Extrac­tion and Semantic Classification of Product</p><p>Reviews. WWW-2003.</p><p>R. Declerck, and S. Reed. 2001. Conditionals: A Comprehensive Empirical Analysis. Berlin: Mouton de Gruyter.</p><p>X. Ding, B. Liu, and P. S. Yu. 2008. A holistic lexicon-based approach to opinion mining.</p><p>WSDM-2008.</p><p>A. Esuli, and F. 2006. Sebastiani. Determining term subjectivity and term orientation for opi­nion mining, EACL-2006.</p><p>M. Fiszman, D. Demner-Fushman, F. Lang, P. Goetz, and T. Rindflesch. 2007. Interpreting Comparative  Constructions in Biomedical</p><p>Text. BioNLP-2007.</p><p>M. Gamon, A. Aue, S. Corston-Oliver, S. and E. Ringger. 2005. Pulse: Mining customer opi­nions from free text. IDA-2005.</p><p>G. Ganapathibhotla and B. Liu. 2008. Identifying Preferred Entities in Comparative Sentences.</p><p>COLING-2008.</p><p>V. Hatzivassiloglou, and K. McKeown, K. 1997.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">188</doubt><page local="10" global="189"/><p>2007.</p><p>Predicting the Semantic Orientation of Adjec­tives. ACL-EACL-1997.</p><p>M. Hu and B. Liu. 2004. Mining and summariz­ing customer reviews. KDD-2004.</p><p>N. Jindal, and B. Liu. 2006. Mining Comparative Sentences and Relations. AAAI-2006.</p><p>N. Kaji, and M. Kitsuregawa. 2006. Automatic construction of polarity-tagged corpus from</p><p>HTML documents. ACL-2006.</p><p>H. Kanayama, and T. Nasukawa. 2006. Fully Automatic Lexicon Expansion for Domain-Oriented Sentiment Analysis. EMNLP-2006.</p><p>S. Kim and E. Hovy. 2004. Determining the Sen­timent of Opinions. COLING-2004.</p><p>N. Kobayashi, K. Inui and Y. Matsumoto. 2007. Extracting Aspect-Evaluation and Aspect-of Relations in Opinion Mining. EMNLP-2007.</p><doubt alpha="45.5" length="44" tooSmall="False" monospace="0.0">L.-W. Ku, Y.-T. Liang, and H.-H. Chen. 2006,</doubt><p>Opinion Extraction, Summarization and Tracking in News and Blog Corpora. AAAI-</p><p>CAAW.</p><p>B. Liu. 2006. Web Data Mining: Exploring Hyperlinks, Content and Usage Data. Sprin­ger.</p><p>B. Liu. 2009. Sentiment Analysis and Subjectivi­ty. To appear in <i>Handbook of Natural Lan­guage Processing, </i>Second Edition, (editors: N. Indurkhya and F. J. Damerau), 2009 or 2010.</p><p>Y. Lu, and C. X. Zhai. 2008. Opinion integration through   semi-supervised  topic modeling.</p><doubt alpha="59.3" length="54" tooSmall="False" monospace="0.0">WWW-2008. R. McDonald, K. Hannan, T. Neylon, M. Wells,</doubt><p>and J. Reynar. 2007. Structured models for fine-to-coarse sentiment analysis. ACL-2007</p><p>Q. Mei, X. Ling, M. Wondra, H. Su, and C. X.</p><p>Zhai. 2007. Topic Sentiment Mixture: Model­ing Facets and Opinions in Weblogs. WWW-</p><p>V. Ng, S. Dasgupta, and S. M. Niaz Arifin. 2006.</p><p>Examining the role of linguistic knowledge sources in the automatic identification and classification of reviews. ACL-2006.</p><p>B. Pang and L. Lee. 2008. Opinion Mining and Sentiment Analysis. Foundations and Trends in Information Retrieval 2(1-2), pp. 1-135, 2008.</p><p>B. Pang, L. Lee. and S. Vaithyanathan. 2002.</p><p>Thumbs up? Sentiment Classification Using Machine   Learning   Techniques. EMNLP2002.</p><p>A-M. Popescu, and O. Etzioni. 2005. Extracting Product Features and Opinions from Reviews.</p><p>EMNLP-2005.</p><p>G. Qiu, B. Liu, J. Bu and C. Chen. 2009. Ex­panding Domain Sentiment Lexicon through</p><p>Double Propagation. IJCAI-2009.</p><p>E. Riloff, and J. Wiebe. 2003. Learning extrac­tion  patterns  for  subjective expressions.</p><p>EMNLP-2003.</p><p>V. Stoyanov, and C. Cardie. 2008. Topic Identi­fication for fine-grained opinion analysis.</p><doubt alpha="60.3" length="58" tooSmall="False" monospace="0.0">COLING-2008. I. Titov and R. McDonald. 2008. A Joint Model</doubt><p>of Text and Aspect Ratings for Sentiment Summarization. ACL-2008. P. Turney. 2002. Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsuper-vised Classification of Reviews. ACL-2002.</p><p>J. Wiebe, R. Bruce, and T. O'Hara. 1999. Devel­opment and use of a gold standard data set for subjectivity classifications. ACL-1999.</p><p>J. Wiebe, and T. Wilson. 2002. Learning to Dis-ambiguate Potentially Subjective Expressions.</p><doubt alpha="54.2" length="59" tooSmall="False" monospace="0.0">CoNLL-2002. T. Wilson, J. Wiebe. and R. Hwa. 2004. Just how</doubt><p>mad are you? Finding strong and weak opinion clauses. AAAI-2004.</p><p>H. Yu, and Y. Hatzivassiloglou. 2003. Towards answering opinion questions: Separating facts from opinions and identifying the polarity of opinion sentences. EMNLP-2003.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">189</doubt></references></body></article>