<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="1446"/><title>Construction of a Blog Emotion Corpus for Chinese Emotional Expression Analysis</title><author surname="Quan" givenname="Changqin"><org  name="Tokushima University" country="Japan" city="Tokushima"/></author><author surname="Ren" givenname="Fuji"><org  name="Tokushima University" country="Japan" city="Tokushima"/></author></firstpageheader><frontmatter><p><b>Construction of a Blog Emotion Corpus for Chinese Emotional</b></p><p><b>Expression Analysis</b></p><p><b>Changqin Quan Fuji Ren</b></p><p>Faculty of Engineering Faculty of Engineering</p><p>University of Tokushima University of Tokushima</p><p>2-1 Minamijosanjima Tokushima Japan 2-1 Minamijosanjima Tokushima Japan</p><p>quan-c@is.tokushima-u.ac.jp ren@is.tokushima-u.ac.jp</p></frontmatter><abstract>There is plenty of evidence that emotion analysis has many valuable applications. In this study a blog emotion corpus is con­structed for Chinese emotional expression analysis. This corpus contains manual an­notation of eight emotional categories (ex­pect, joy, love, surprise, anxiety, sorrow, angry and hate), emotion intensity, emo­tion holder/target, emotional word/phrase, degree word, negative word, conjunction, rhetoric, punctuation and other linguistic expressions that indicate emotion. An­notation agreement analyses for emotion classes and emotional words and phrases are described. Then, using this corpus, we explore emotion expressions in Chi­nese and present the analyses on them. </abstract></header><body><section number="1" title="Introduction"><p>Textual emotion analysis is becoming increasingly important due to augmented communication via computer mediated communication (CMC) inter­net sources such as weblogs, email, websites, fo­rums, and chat rooms. Especially, blogspace con­sists of millions of users who maintain an online diary, containing frequently-updated views and personal remarks about a range of issues.</p><p>Despite the increased focus on analysis of web content, there has been limited emotion analy­sis of web contents, with the majority of studies focusing on sentiment analysis or opinion min­ing. Classifying the mood of a single text is a hard task; state-of-the-art methods in text classi­fication achieve only modest performance in this domain (Mishne, 2005). In this area, some of the hardest problems involve acquiring basic re­sources. Corpora are fundamental both for devel­oping sound conceptual analyses and for training these emotion-oriented systems at different lev­els: to recognize emotions, to express appropriate emotions, to anticipate emotions, and other emo­tion processing applications.</p><p>In this study we propose a relatively fine­grained annotation scheme, annotating emotion in text at three levels: document, paragraph, and sen­tence. We select eight emotion classes (expect, joy, love, surprise, anxiety, sorrow, angry and hate) for this annotation, and explore various linguis­tic expressions that indicate emotion in Chinese. The annotation scheme has been employed in the manual annotation of a corpus containing 1,487 documents, with 11,255 paragraphs, 35,096 sen­tences, and 878,164 Chinese words. Then, using this corpus, we explore and present data analy­ses on emotions, involving emotion states, accom­panying emotions, transfer emotions, independent emotions in texts.</p><p>The remainder of this paper is organized as fol­lows. Section 2 describes the emotion corpus an­notation scheme. Section 3 presents the inter-annotator agreement study. Section 4 describes the analysis of emotion expressions. Section 5 presents a review of current emotion corpora for textual emotion analysis. Section 6 concludes this study with closing remarks and future directions.</p></section><section number="2" title="Blog Emotion Corpus Annotation Scheme"><p>Weblogs are an increasingly popular mode of communication in the ever changing online world. Writing suits the recording of facts and the com­munication of ideas, and their textual basis makes them equally suitable for recording emotions and opinions. So, we select blogs as object and data source for this emotion corpus annotation.</p><subsection number="2.1" title="Emotional Expression in Text"><p>An important starting point in constructing this corpus is to represent emotion in text. One of the biggest questions in affect recognition is, "What</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1446</doubt><p><i>Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, </i>pages 1446-1454, Singapore, 6-7 August 2009. ©2009 ACL and AFNLP<page local="2" global="1447"/></p><p>are the couplings between affective states and their patterns of expressions" (Picard, 1997).</p><p>In this study we propose an emotional expres­sion space model to represent emotion in text, which is hierarchical in consistent with the natural structure of a document. Emotion of a document is represented by a vector</p><doubt alpha="27.6" length="29" tooSmall="False" monospace="0.0">~t =&lt;ei,e2,...,ei,...,en&gt; (1)</doubt><p>Here, <i>a </i>is a basic emotion class contained in doc­ument <i>d.</i><i> </i>The values of range from 0.0 to 1.0 (discrete), indicating the intensities of the basic emotion classes. Similar to a document, emotion of each paragraph and each sentence in a docu­ment is represented by an emotion vector.</p><p>Basic emotions may be defined in many ways. To decrease confusions on emotion categories' se­lection and to contain the most common emotion classes in blogs, we select eight emotion classes (expect, joy, love, surprise, anxiety, sorrow, angry and hate) for this manual annotation, and they are agreed by eleven annotators through a testing an­notation period. Table 1 shows the numbers of the eight emotion classes in documents, paragraphs, and sentences in this corpus.</p><p>As shown in Table 1, we have reasonably large counts for all 8 emotions in all 3 units of text. And we also can get the average value for the numbers of emotion classes in each document, each para­graph and each sentence; they are 2.83, 1.84, and 1.36 respectively.</p></subsection><subsection number="2.2" title="The Multi-level Annotation Frame"><p>The annotation frame includes 3 levels: docu­ment, paragraph, and sentence. Sentence level is the basic level for emotion annotation; the an­notation includes intensities of the eight basic emotion classes, emotion holder/target, emotional words/phrases, rhetoric, emotional punctuations, emotion objective/subjective and emotion polarity. Paragraph level is the upper level of sentence level; the annotation includes intensities of the eight ba­sic emotion classes, topic words to reflect the topic of a paragraph, and the number of topic sentence that can express the main points of this paragraph. Document level is the uppermost level; its anno­tation is similar to paragraph level. The tokenized text files are organized into XML documents. An example document is listed in Figure 1.</p><doubt alpha="66.7" length="12" tooSmall="True" monospace="0.0">- &lt;document&gt;</doubt><doubt alpha="56.2" length="224" tooSmall="False" monospace="0.0">&lt;Joy&gt;0.0&lt;/Joy&gt; &lt;Hate&gt;0.6&lt;/Hate&gt; &lt;l_ove&gt;fJ.7&lt;/l_ove&gt; &lt;Sarrrjw&gt;0.7&lt;/Sorrrjw&gt; &lt; Anxiety&gt;0.7&lt;/Anxiety&gt; &lt;Surprise&gt;0.0&lt;/Surprise&gt; &lt;AngBr&gt;0.6&lt;/Anger&gt; &lt;Expect&gt;O.S&lt;/Expect&gt; &lt;Topio#H^laaü:&lt;/Topic&gt; + &lt;title T-'#H»±m&gt;fl^"&gt; - &lt;paragraph&gt;</doubt><doubt alpha="56.5" length="23" tooSmall="False" monospace="0.0">&lt;P_no&gt;#lfë"g^feâ&lt;/P_no&gt;</doubt><doubt alpha="42.9" length="14" tooSmall="False" monospace="0.0">&lt;Joy&gt;0.0&lt;/Joy&gt;</doubt><doubt alpha="62.5" length="16" tooSmall="False" monospace="0.0">&lt;Hats&gt;O.Q&lt;/Hate&gt;</doubt><doubt alpha="50.0" length="16" tooSmall="False" monospace="0.0">&lt;Love&gt;0.8&lt;/Love&gt;</doubt><doubt alpha="60.0" length="20" tooSmall="False" monospace="0.0">&lt;Sorrow&gt;0.0&lt;/Sorrow&gt;</doubt><doubt alpha="60.9" length="23" tooSmall="False" monospace="0.0">&lt; Anxiety&gt;0.0&lt;/Anxiety&gt;</doubt><p>&lt;Surprise<b>&gt;0.rJ</b>&lt;/Surprise&gt;</p><doubt alpha="66.7" length="18" tooSmall="False" monospace="0.0">&lt;Anger&gt;O.G&lt;/Anger&gt;</doubt><doubt alpha="60.0" length="20" tooSmall="False" monospace="0.0">&lt;Expect&gt;0.7&lt;/ExpBct&gt;</doubt><doubt alpha="63.0" length="27" tooSmall="False" monospace="0.0">&lt;Topio#HT^la fl:5i:&lt;/Topic&gt;</doubt><p>&lt;Summarize_Sentences<b>&gt;2</b>&lt;/Summarize_Sentences&gt; - sentence S<b>="*IÄÄ£-^JUl *H^«m#. Ü</b></p><doubt alpha="26.9" length="26" tooSmall="False" monospace="0.0">&lt;S_n o±&amp;%1^1 t*ä &lt;/S_n □ &gt;</doubt><doubt alpha="57.9" length="114" tooSmall="False" monospace="0.0">&lt;Segmented_S&gt;Sn^/c ft/r^/v-HVmAm/tftj/u fÀ/n, /w3fcU/v^/tffil/u1k#/n,/wS/r #/v S/d fF^/v fhk/r ? /w&lt;/S8grnBnt8d_S&gt;</doubt><doubt alpha="65.2" length="23" tooSmall="False" monospace="0.0">&lt;S_Length&gt;2G&lt;/S_Length&gt;</doubt><doubt alpha="62.3" length="159" tooSmall="False" monospace="0.0">^Keywords start-'-l"position-"23"end-"-l"Surpnse="0"Sorrow="0"POS="v"Opinionhclder="0"Love='0.e"Joy-'O"Hate="0"Expect="0.7"Anxiety="0"Anger="0"&gt;tFiê&lt;/Keywords&gt;</doubt><p>&lt;degree_adv start<b>="-l" </b>position<b>="22" </b>modifier_word_position<b>="23" </b>modifier_word_length<b>="2" </b>end<b>="-l"&gt;:fE</b>&lt;/degree_adv&gt;</p><p>&lt;E_conjunction start<b>="-l" </b>position—"□" next_position<b>="0" </b>rnodifier_word_position<b>="2" </b>tnodifier_word_length<b>="17" </b>end<b>="-l"&gt;inM </b>&lt;/E_conjunction&gt; &lt;Opinion_Fact<b>&gt;opinion</b>&lt;/Opinion_Fact&gt;</p><doubt alpha="69.2" length="39" tooSmall="True" monospace="0.0">&lt;Rhetoric E_type="flB^"&gt;^fnI&lt;/Rhetoric&gt;</doubt><doubt alpha="69.8" length="43" tooSmall="True" monospace="0.0">&lt;punctuation E_type="Ärf#"&gt;? &lt;/punctuation&gt;</doubt><doubt alpha="80.0" length="25" tooSmall="True" monospace="0.0">&lt;Polarity&gt;fttt&lt;/Polarity&gt;</doubt><doubt alpha="61.2" length="49" tooSmall="False" monospace="0.0">&lt;Opinion_holder start="-l"position="2"8nd="-l"&gt;3£</doubt><doubt alpha="62.0" length="187" tooSmall="False" monospace="0.0">&lt;/Opinion_holder&gt; &lt;Joy&gt;0.0&lt;/Joy&gt; &lt;Hate&gt;O.Q&lt;/Hate&gt; &lt;Love&gt;O.D&lt;/Love&gt; &lt;Scrrow&gt;0.0&lt;/Borrcw&gt; &lt;Anxiety&gt;O.Q&lt;/Anxiety&gt; &lt;Surprise&gt;D.D&lt;/Surprise&gt; &lt;Angsr&gt;0.0&lt;/Anger&gt; &lt;Expect&gt;D.7&lt;/Expect&gt; &lt;/sentence&gt;</doubt><doubt alpha="50.0" length="34" tooSmall="True" monospace="0.0">+ &lt;sentence S="-iMÄt¥: ^sT* II&gt; "&gt;</doubt><doubt alpha="66.2" length="136" tooSmall="True" monospace="0.0">&lt;/paragraph&gt; + &lt;paragraph&gt; + &lt;paragraph&gt; + &lt;paragraph&gt; + &lt;paragraph&gt; + &lt;paragraph&gt; + &lt;paragraph&gt; + &lt;paragraph&gt; + &lt;paragraph&gt; ^/documente</doubt><p>Figure 1 : An annotated document in XML format<page local="3" global="1448"/></p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1447</doubt><table caption="Table 1: Num. of the eight emotion classes" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Emotions</b></p></td><td class="cell"><p><b>Doc</b></p></td><td class="cell"><p><b>Para</b></p></td><td class="cell"><p><b>Sen</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Expect</p></td><td class="cell"><p>656</p></td><td class="cell"><p>2,145</p></td><td class="cell"><p>4,588</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Joy</p></td><td class="cell"><p>565</p></td><td class="cell"><p>2,740</p></td><td class="cell"><p>6,211</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Love</p></td><td class="cell"><p>911</p></td><td class="cell"><p>4,991</p></td><td class="cell"><p>11,866</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Surprise</p></td><td class="cell"><p>124</p></td><td class="cell"><p>503</p></td><td class="cell"><p>1,118</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Anxiety</p></td><td class="cell"><p>732</p></td><td class="cell"><p>4,128</p></td><td class="cell"><p>10,115</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Sorrow</p></td><td class="cell"><p>693</p></td><td class="cell"><p>3643</p></td><td class="cell"><p>8,166</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Angry</p></td><td class="cell"><p>189</p></td><td class="cell"><p>900</p></td><td class="cell"><p>2,221</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Hate</p></td><td class="cell"><p>335</p></td><td class="cell"><p>1,589</p></td><td class="cell"><p>3,555</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Sum</p></td><td class="cell"><p>4,205</p></td><td class="cell"><p>20,639</p></td><td class="cell"><p>47,840</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></subsection><subsection number="2.3" title="Sentence Level Annotation"><p>Sentences are basic units for emotional expres­sion. The central aim of sentence level annotation is to explore as much linguistic expressions for re­flecting emotion in Chinese as possible.</p><p><b>a) Emotion holder/target</b></p><p>In the task of opinion analysis, the problem of opinion holder identification has also been stud­ied, (Bethard, Steven et al., 2004; Choi, Cardie, et al., 2005; Kim and Hovy, 2005). As for emo­tion holder/target identification, little research has been conducted, but we believe it is important for exploring emotional expression and emotion anal­ysis. Emotion holder is the one who holds the emotions, and an emotion target is the object of an emotion holder. For instance,</p><p>(1) <b>ft#tit^io </b>(English: I like this teacher.) In sentence (1<b>),S,"</b><b> </b>(English: I)" is the emotion holder, and ^ <b>^</b><b> </b><b>]fp</b><b> </b>"(English: this teacher.)" is the emotion target.</p><p>In this corpus, not every sentence is annotated with emotion holder or emotion target, and emo­tion holder or emotion target may not appear in pairs in one sentence. If one sentence has more than one emotion holders or emotion targets, they are all annotated.</p><p><b>b) Emotional words and phrases </b>Lexicon-based methods have received a lot of attention in opinion analysis task. There are many lexical resources for these tasks. For emotion anal­ysis tasks, the function of words is equally funda­mental. In most sentimental lexicons, the words usually bear direct emotions or opinions, such as happy or sad, good or bad. However, there are a lot of sentences can evoke emotions without direct emotional words, for example,</p><p>(2) #^£&amp;?fl]ftjHSffi, 4Iïfî]W't II o (English: Spring is in children's eyes, and in their hearts.)</p><p>In sentence (2), we may feel joy, love or ex­pect delivered by the writer. Indeed, as (Ortony, Andrew, et al., 1987) indicates, besides words di­rectly referring to emotional states and for which an appropriate lexicon would help, there are words that act only as an indirect reference to emotions depending on the context.</p><p>In this annotation scheme, direct emotional words and indirect emotional words in a sen­tence are all annotated. In sentence (2), # ^"(English: spring)", <i>Ws</i><i> </i><i>"?</i><i> </i><i>\W</i>"(English: chil­dren)" are labeled. An emotional word or phrase is represented as a vector to record its intensi­ties of the eight basic emotional classes. For in­stance, the vector for the word <b>H </b>^"(English: like)" <i>JÈ</i><i> </i><i>=</i><i> </i>(0.0,0.3,0.9,0.0,0.0,0.0,0.0,0.0) indicates the emotions of weak joy and strong love. For indirect emotional words, we anno­tate their emotion vectors according to their con­texts, for example, the possible emotion vec­tor for the word "# ^(English: spring)" <i>vï</i><i> </i><i>=</i><i> </i>(0.1,0.3,0.3,0.0,0.0,0.0,0.0,0.0) indicates the emotions of weak expect, joy and love. (The emo­tions and intensity values may be different because of different annotators).</p><p>Emotional phrases are combination of words, such as Chinese proverbs, like "tä-h^nXHl?, K§g#lp ^(English: where there is a will, there is a way) ". For an emotional phrase, the positions of its first and character in a sentence are labeled, and also for emotional words if there are Chinese word segmentation mistakes.</p><p>The statistics show that 84.9% of all emotional words have one emotion, and 14.7% have two emotions, only 0.4% have three or four emotions, but they are indispensable for expressing complex feelings in use of language.</p><p>Table 2 shows the numbers of emotional words with different POS (part-of-speech) tags. The set of POS includes 35 classes; Table 2 lists the top five classes.</p><p>As shown in Table 2, verbs, nouns, adjectives and adverbs are strong markers of emotion in Chi­nese.</p><p><b>c) Degree words, negative words, conjunc­tions</b></p><p>Degree words are associated with the intensi­ties of emotions. In Chinese, degree words ap­pear with high frequency. In this corpus, there are 1,039 different degree words annotated, the to­tal occurring number of them is 16,713, in which, 8,294 degree words modify emotional words or phrases directly. Degree words and the modifying contents are all labeled.<page local="4" global="1449"/></p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1448</doubt><table caption="Table 2: Emotional words with different POS" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>POS</b></p></td><td class="cell"><p><b>Num. of words</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(have repeat)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Verb</p></td><td class="cell"><p>37,572</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Noun</p></td><td class="cell"><p>21,308</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Adj.</p></td><td class="cell"><p>20,265</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Adv.</p></td><td class="cell"><p>4,223</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Gerund</p></td><td class="cell"><p>2,789</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Negative words can be placed almost every­where in a sentence to change the meaning, also to change the emotions. Negative words are fre­quently used in Chinese. The statistical data shows that there are 645 different negative words anno­tated in this corpus, the total occurring number of them is 13,750, in which, 3,668 negative words modify emotional words or phrases directly.</p><p>Besides, conjunctions may change the emotion of a sentence, for example,</p><p>T S^H » (Jin guan wo men xi huan zhe ge lao shi, dan ta yi jing li kai le wo men; English: Although we like this teacher, she has leaved.)</p><p>Sentence (3) uses the conjunctions 1f...fM...(jin guan...dan..., English: although)" express emotions of love and sorrow. There are 297 different conjunctions annotated in this corpus. Conjunctions and the modifying contents are all labeled. If conjunctions appear in pairs in a sentence, the position of pairing words for each conjunction are also labeled. For the above sentence (3), conjunctions are annotated as follows (Figure. 2).</p><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">Ii</doubt><doubt alpha="64.0" length="25" tooSmall="False" monospace="0.0">- position 2 position 1 -</doubt><doubt alpha="20.0" length="10" tooSmall="False" monospace="0.0">*v-T-"---'</doubt><doubt alpha="61.1" length="36" tooSmall="False" monospace="0.0">conjuctionl        ]/ conjuction 2 T</doubt><p>modifying contents 1    modifying contents 2</p><p>English: Although we like this teacher, she has leaved.</p><figure caption="Figure 2: An example of conjunctions annotation"></figure><p>Figure 3 shows the growth curve of word num­ber with document number from 300 to 1487. As can be seen from Figure 3, the increase numbers of emotional words/phrases slow down with the in­crease in the number of documents, and the num­bers of negative words, degree words and conjunc­tions basically remained stable. We can look for­ward to containing most of common emotional ex­pressions in weblogs articles.</p><p><b>d) Rhetorics, punctuations</b></p><p>Chinese rhetoric has been well studied from the view of linguistics and literature. We se­lect nine common rhetoric categories to anno­tate: tb Pfj(English: metaphor), English: ^ ^(exaggeration), ftt A(English: personifica­tion), <i>M </i>{^(English:  antithesis or parallel), <i>ffl</i></p><p><b>Number of words</b> <b>Number of documents</b></p><doubt alpha="0.0" length="35" tooSmall="True" monospace="0.0">18000 -16000 -14000 -12000 -10000 -</doubt><doubt alpha="0.0" length="4" tooSmall="True" monospace="0.0">8000</doubt><doubt alpha="0.0" length="4" tooSmall="True" monospace="0.0">6000</doubt><doubt alpha="0.0" length="4" tooSmall="True" monospace="0.0">4000</doubt><doubt alpha="0.0" length="4" tooSmall="True" monospace="0.0">2000</doubt><doubt alpha="2.3" length="43" tooSmall="True" monospace="0.0">200 400 600 800 1000 1200         1400 160C</doubt><figure caption="Figure 3: Growth curve of word number"></figure><p>tb(English: parallelism sentence), &amp; fö] (English: rhetorical question with answer), jx. ÏrI (English: rhetorical question), S Ä(English: repeat), W $IJ(English: irony). Especially, W <i>M </i>(English: irony) is a way as to imply the contrary of what one says, if a sentence is annotated with irony, its emo­tions maybe totally different from the emotions of words that it contains. We annotate rhetoric cate­gory and the corresponding emotion category.</p><p>Punctuation is the use of standard marks and signs in writing to separate words into sentences, clauses, and phrases in order to clarify meaning. Some punctuation marks can express emotions, for example, an exclamation mark (!) or a question mark (?) is used at the end of a sentence to show strong emotion. Balog, Mishne, et al. (2006) sug­gests that people relied on four strategies includ­ing punctuation to express happiness versus sad­ness. Punctuation effect is also shown in (Leshed and Kaye, 2006) to extend to emoticon placement in website text messages. We annotate punctua­tion with emotion and the corresponding emotion category.</p><p><b>e) Emotion objective/subjective, emotion po­larity</b></p><p>Distinguishing a sentence between factual and subjective information could support for many natural language processing applications. Objec­tive and subjective in our annotation scheme is to distinguish a sentence between writer's emotion and non-writer's emotion.</p><p>There is a positive side or a negative side on emotion. We call this an emotional polarity. Emo­tion polarity of a sentence is determined by inte­grating its emotions. A sentence without emotion is annotated with neutral.</p><p>An annotation tool is developed for this corpus annotation.<page local="5" global="1450"/> Input files are text files with Chinese segmentation and part-of-speech tags, the anno­tated output files are XML files.</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1449</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>■</b></p></td><td class="cell"><p>Emotinal words</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>•</b></p></td><td class="cell"><p>Emotinal phrases</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Negative words</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>—v-</p></td><td class="cell"><p>Degree words</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Conjunctions</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></subsection></section><section number="3" title="Annotation Agreement analysis"><p>Emotion annotation is a hard task because the na­ture of emotion is inherently ambiguous. In the process of annotation, annotators were encouraged to follow their "first intuition". To measure agree­ment on various aspects of the annotation scheme, three annotators independently annotated 26 doc­uments with a total of 270 paragraphs, 701 sen­tences.</p><subsection number="3.1" title="Agreement for Emotion Classes"><p>The kappa coefficient of agreement is a statistic adopted by the Computational Linguistics com­munity as a standard measure for this purpose (Carletta, 1996). We measured two agreements for emotion classes' annotation:</p><doubt alpha="61.0" length="213" tooSmall="False" monospace="0.0">Agreement (a):the agreement on classifi­cation of containing or not containing some emotions. In this case, we distinguish two classes: emotion intensityG{0.0} orG{0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9,1.0};</doubt><p><b>Agreement (b): </b>the agreement on classifica­tion of emotion intensity. In this case, we dis­tinguish four classes: G {0.0} or G {0.1, 0.2, 0.3, 0.4}, or <i>eA</i><i> </i>G {0.5, 0.6, 0.7}, or <i>eA</i><i> </i>G {0.8,0.9,1.0}.</p><table caption="Table 3 shows Agreement (a) and (b) measure on documents, paragraphs and sentences."></table><p>As shown in Table 3, it is easier for annotators to agree at the coarser levels of granularity, and it is more difficult to agree on the level of emotion intensity.</p></subsection><subsection number="3.2" title="Agreement for Emotional Words and Phrases"><p>Measuring agreement for emotional words and phrases is to verify that annotators agree on which expressions should be marked. To illustrate this agreement problem, consider the emotional words and phrases identified by annotators <b>a </b>and <b>b.</b><b> </b>This sentence was preprocessed by Chinese segmenta­tion and tagged with part-of-speech.</p><p>(4) <b>4-jt</b>/t , /w ^/d ic/v <b><i>WWn </i></b>#/u <i>mit</i><b><i>/n </i></b><b>ftlft</b>/v #®/n fcjVNg , /w <i>m</i><i>^/v</i><i> </i><b>^SM</b>/ad <b><i>M</i></b><i>/y</i><i> </i><b>äJ</b><b>/v </b>^/m <b>^M</b>/d #J/u ft <b>ft</b>/a is]i^/n , /w <b><i>M/r Wd </i></b><b>Ji</b><b>/v </b>: /w <b><i>foMHWn </i></b><b>, /w </b><b><i>£3t/n </i></b><i>m </i><b>H/z </b>o (English: This morning, when I walked to the meeting with sunshine, some wonderful words that have not been used for many years crossed my mind, which are "the autumn sky is clear, the air is crisp" and "shinning with gold color") <b>a:</b><b> </b><i>myt,iJtmjkMHM,±yt,mm; </i><b>b: </b><i>fom, myt, </i><b>ttft, </b><i>%xmhm, mm-,</i></p><p>In sentence (4), the two annotators agree that <b>"PHft", "ttft", "Äffi^H" </b>and <b>"ïlïg" </b>can ex­press emotion. In addition, annotator <b>a </b>marked the word and annotator <b>b </b>marked the word</p><p>In this task, there is no guarantee that the an­notators will identify the same set of expressions. Thus, to measure agreement we want to consider how much intersection there is between the sets of expressions identified by the annotators. We use the following voting-agreement metric to mea­sure agreement in identifying emotional words and phrases.</p><p>Metric voting-agreement is defined as follows. Let A, B and C be the sets of expressions anno­tated by annotators a, b and c respectively. The expert coder is the set of expressions that agreed by at least two annotators, see Equation 2.</p><p><i>voting -agreement = Avg{</i></p><p><i>count{ti = </i><i>ej)</i><i> count{{ti)</i></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(2)</doubt><p>In which, <i>U</i><i> </i>G <i>T,</i><i> </i><i>ej</i><i> </i>G <i>E,T</i><i> </i><i>=</i><i> </i><i>A\JB\JC,</i><i> </i><i>E</i><i> </i><i>=</i><i> </i><i>(Af]B)[j(Af]C)[j(Bf]C).</i></p><p>The agreement for emotional words and phrases is 0.785.</p></subsection></section><section number="4" title="Emotional Expressions Analysis 4.1   Emotion State"><p>"Emotion state in text" is the state of combined emotions in a text unit. An emotion state is repre­sented by 8 binary digits, each digit corresponding to a basic emotion class respectively. As an exam­ple, a document emotion state "01100000" is the state of combined emotions by joy and love.</p><p>The statistics show that, in this corpus, there are 149 different emotion states in all of the 1,487 documents, 165 different emotion states in all of the 11,255 paragraphs, and 143 different emotion states in all of the 35,096 sentences respectively.<page local="6" global="1451"/> That indicates the set of emotion state in texts is relatively small. We also found some basic emo­tions tend to combine together, such as {expect, joy, love}, {anxiety, sorrow}, {angry, hate}. How­ever, some emotions have small or scarce possibil­ity appear together, such as joy and hate, surprise and angry.</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1450</doubt><table caption="Table 3: Agreement on emotion classes" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>Agreement(a)</b></p></td><td class="cell"><p><b>Agreement(b)</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>documents</p></td><td class="cell"><p>0.831</p></td><td class="cell"><p>0.695</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>paragraphs</p></td><td class="cell"><p>0.705</p></td><td class="cell"><p>0.616</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>sentences</p></td><td class="cell"><p>0.756</p></td><td class="cell"><p>0.648</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Average</p></td><td class="cell"><p>0.764</p></td><td class="cell"><p>0.653</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><subsection number="4.2" title="Accompanying Emotions"><p>In an emotion state, some basic emotions are mixed together. When an emotion <i>ej</i><i> </i>arise, emo­tion <i>e%(% </i><i>/</i><i> </i><i>j)</i><i> </i>arise with accompany, then, <i>a</i><i> </i>is an accompanying emotion of <i>ej.</i><i> </i>To compute the probability of the accompanying emotion given an emotion <i>ej,</i><i> </i>we count the cooccurrence of and <i>ej</i><i> </i>in a text unit (a document, a paragraph, or a sentence).</p><p><i>count{ei   with </i><i>ej</i> <i>count </i><i>(ej)</i></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(3)</doubt><p>Table 4 shows the accompanying emotions with the highest probabilities for the eight basic emo­tions in documents, paragraphs and sentences.</p><p>In Table 4, the accompanying emotions has shown a high uniformity in the 3 units of text.</p></subsection><subsection number="4.3" title="Transfer Emotions"><p>When emotion change from one emotion class to another one, we call this emotion transfer. Using the context relation of paragraphs and sentences, we compute the probability <i>P(ei —&gt; </i><i>ej).</i></p><doubt alpha="63.6" length="22" tooSmall="False" monospace="0.0">count(et= eiyet+\ =ej)</doubt><doubt alpha="75.0" length="4" tooSmall="False" monospace="0.0">P(el</doubt><doubt alpha="20.0" length="5" tooSmall="False" monospace="0.0">e,- =</doubt><doubt alpha="64.3" length="14" tooSmall="False" monospace="0.0">count (et = ei</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(4)</doubt><p>in paragraph <i>t </i>+ 1 (or sentence <i>t </i>+ 1). Table 4 shows the transfer emotions with the highest prob­abilities for the eight basic emotions in paragraphs and sentences.</p><p>In which, <i>et </i>is an emotion class in paragraph <i>t </i>(or sentence <i>t),</i><i> </i>and <i>et</i><b><i>+i </i></b>is another emotion class</p><table caption="Table 5: Transfer emotions"></table><p>Similar to this, we can compute the probability of emotion state transfer <i>P(estatei —&gt; estate </i><i>j).</i><i> </i>This may help a lot for emotion prediction, for example, if we know the current emotion state is "00000110" (sorrow an angry), we can estimate the probability of this emotion state to another emotion state "00000001" (hate).</p></subsection><subsection number="4.4" title="Independent Emotion"><p>When a text unit (a document, a paragraph, or a sentence) only contains one emotion class, this emotion class is an independent emotion. The statistics show that emotion of love has high in­dependence, however, joy, surprise and angry has relative low independence. The intuition is love can be the only topic emotion in a text unit, but emotions of joy, surprise and anxiety more incline to combine with other emotions.</p></subsection></section><section number="5" title="Related work"><p>Previous approaches to textual emotion analysis have employed some different corpora. Mishne (2005) experimented mood classification in blog posts on a corpus of 815,494 blog posts from Live-journal (http://www.livejournal.com), a free we-blog service with a large community. Livejour-nal also used as data source for finding happi­ness (Mihalcea and Liu, 2006), capturing global mood levels (Mishne and De Rijke, 2006), clas­sifying mood (Jung, Park, et al., 2006; Jung, Choi, et al., 2007), discovering mood irregu­larities (Balog, Mishne, et al., 2006), recogniz­ing affect (Leshed and Kaye, 2006). A similar emotion corpus in Chinese is Yahoo !'s Chinese news (http://tw.news.yahoo.com), which is used for Chinese emotion classification of news read­ers (Lin, Yang, et al., 2007) and emotion lexi­con building (Yang, Lin, et al., 2007).<page local="7" global="1452"/> Tokuhima (2008) also use web as data resources to obtain a huge collection of emotion-provoking event in­stances for Japanese emotion classification. More and more weblogs have added mood column to record blog users' moods when they read or write a blog.</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1451</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Emotions</b></p></td><td class="cell"><p><b>Paras</b></p></td><td class="cell"><p><b>Sens</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Expect</p></td><td class="cell"><p>Love</p></td><td class="cell"><p>Expect</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Joy</p></td><td class="cell"><p>Love</p></td><td class="cell"><p>Love</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Love</p></td><td class="cell"><p>Love</p></td><td class="cell"><p>Love</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Surprise</p></td><td class="cell"><p>Love</p></td><td class="cell"><p>Love</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Anxiety</p></td><td class="cell"><p>Anxiety</p></td><td class="cell"><p>Anxiety</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Sorrow</p></td><td class="cell"><p>Sorrow</p></td><td class="cell"><p>Sorrow</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Angry</p></td><td class="cell"><p>Anxiety</p></td><td class="cell"><p>Angry</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Hate</p></td><td class="cell"><p>Hate</p></td><td class="cell"><p>Hate</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 4: Accompanying emotions" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Emotions</b></p></td><td class="cell"><p><b>Docs</b></p></td><td class="cell"><p><b>Paras</b></p></td><td class="cell"><p><b>Sens</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Expect</p></td><td class="cell"><p>Love</p></td><td class="cell"><p>Love</p></td><td class="cell"><p>Love</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Joy</p></td><td class="cell"><p>Love</p></td><td class="cell"><p>Love</p></td><td class="cell"><p>Love</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Love</p></td><td class="cell"><p>Joy</p></td><td class="cell"><p>Joy</p></td><td class="cell"><p>Joy</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Surprise</p></td><td class="cell"><p>Anxiety</p></td><td class="cell"><p>Love</p></td><td class="cell"><p>Love</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Anxiety</p></td><td class="cell"><p>Sorrow</p></td><td class="cell"><p>Sorrow</p></td><td class="cell"><p>Sorrow</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Sorrow</p></td><td class="cell"><p>Anxiety</p></td><td class="cell"><p>Anxiety</p></td><td class="cell"><p>Anxiety</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Angry</p></td><td class="cell"><p>Anxiety</p></td><td class="cell"><p>Hate</p></td><td class="cell"><p>Hate</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Hate</p></td><td class="cell"><p>Anxiety</p></td><td class="cell"><p>Sorrow</p></td><td class="cell"><p>Angry</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Two merits let them well accepted as emotion corpora: a large number of weblogs contained and moods annotated by blog users. However, there is a great inconsistency on emotion categories given by different websites. Livejournal gives a pre­defined list of 132 common moods, while Ya­hoo !'s Chinese news provides readers 8 emotion categories. Too many mood classes may confuse users, and Mishne (2005) also pointed out one ob­vious drawback of the mood "annotation" in this corpora is that they are not provided in a consistent manner; the blog writers differ greatly from each other, and their definitions of moods differ accord­ingly. In addition, some words are not fitted to be taken as emotion classes, such as "useful" in Ya­hoo !'s emotion categories. These corpora may be helpful for analyzing the global moods on a full text, but the inconsistent emotion categories is a problem, and no more labeled information can be exploited from them.</p><p>The emotion analysis on sentence level may also be important for more detailed emotion anal­ysis systems. Aim, Roth, et al. (2005) ex­plore the text-based emotion prediction problem; they annotated a corpus of 22 Grimms ' tales on sentence level with eight emotion categories (an­gry, disgusted, fearful, happy, sad, positively sur­prised, negatively surprised), contain 1580 sen­tences. Neviarouskaya, Prendinger et al. (2007) address the tasks of recognition and interpreta­tion of affect communicated through text messag­ing. They collected 160 sentences labeled with one of nine emotions categories (anger, disgust, fear, guilt, interest, joy, sadness, shame, and sur­prise) from a corpus of online diary-like blog posts and a corresponding intensity value. Aman and Szpakowicz (2007) classify emotional and non-emotional sentences based on a knowledge-based approach. They used a corpus with tags of emotion category, emotion intensity and the words/phrases that indicate emotion in text. An emotion corpus for Japanese was built for recognizing emotions and emotion estimation (Ren, 2009; Matsumoto, 2006). However, the sizes of these corpora seem not enough for large scale tex­tual emotion analysis, a lot of linguistic features are not reflected from them. A more fine-grained opinion and emotion corpus is the MPQA Corpus (Wiebe, Wilson, et al., 2005), which contains 535 news articles (10,000-sentence) from a wide va­riety of news sources, manually annotated at the sentential and subsentential level for opinions and other private states. But emotion categories are not included in it.</p><p>To the best of our knowledge, at present, there's no relatively large corpora annotated with detailed linguistic expressions for emotion in Chinese, and we believe that such corpora would support the de­velopment and evaluation of emotion analysis sys­tems.</p></section><section number="6" title="Conclusions and Future Work"><p>In this study we proposed an emotional expres­sion space model. Emotion of a document, a para­graph, a sentence, or even a word is represented by an emotional vector. Based on this model, we described a relatively fine-grained annotation scheme and annotated emotion in text. We also gave the inter-annotator agreement study on an­notation. Then, we explore the emotional expres­sions in texts.</p><p>This annotated dataset can be obtained for free with license <footnote anchor="1"/>. Eleven annotators made efforts on it spanning a period of ten months (They are Ph.D and M.S. candidates specialize in Natural Language Processing and Emotion Analysis). To ensure the quality of this dataset, each document was performed a three pass annotation, in which the first pass is annotated by one annotator and then the second and the third verification pass were performed by other two annotators. The process of this corpus annotation is easy to make mistakes because of a lot of information should be anno­tated. The verification pass is to check the an­notation mistakes (such as the start and end po­sitions of emotional phrases in sentences), but not to change the choices of emotion classes or emo­tional words which had been annotated by other annotators.</p><p>Using this corpus, we will make a more exten­sive study of textual emotion analysis in Chinese, for example, the influence of degree words, nega­tive words, or other elements on emotional expres­sion; the difference between subjective emotion and objective emotion; emotion transfer tracking.<page local="8" global="1453"/> More applications also will be explored, such as emotional summarization, emotional question an­swering; emotional topic discovering. At the same time, new research problems will arise, for exam­ples, how to acquiring more emotional words and to generate their emotional vectors automatically; how to generate emotional vectors for sentences, paragraphs and documents with known emotional elements in them? There is need to immerge fur­ther into these problems.</p><footnote label="1"> http://al  -www.is.tokushima-u.ac.jp/member/ren/Ren- CECpsl.O/Ren-CECpsl.O.html</footnote><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1452</doubt></section><section title="Acknowledgments"><p>We are grateful to our annotators: Huana Li, Ye Wu, Lei Chen, Yu Zhang, Ji Li, Ziliang Du, Yuanlu Fu, Rong Mu, Yan Sun, Cheng Wang, Yunong Wu, and other participants and support­ers. We are also grateful to Dr. Suzuki and Dr. Matsumoto for the helpful advice. This research has been partially supported by Ministry of Edu­cation, Science, Sprots and Culture, Grant-in-Aid for Challenging Exploratory Research, 21650030.</p></section><references><p>Alena Neviarouskaya, Helmut Prendinger, Mitsuru Ishizuka. 2007. Textual Affect Sensing for Social and Expressive Online Communication. <i>Proceed­ings of the 2nd international conference on Affective Computing and Intelligent Interaction, </i>pp. 218-229.</p><p>Bethard, Steven, Hong Yu, Ashley Thornton, Vasileios Hatzivassiloglou, and Dan Jurafsky. 2004. Auto­matic Extraction of Opinion Propositions and their Holders. <i>AAAI Spring Symposium on Exploring At­titude and Affect in Text: Theories and Applications </i>, pp. 133 - 136.</p><p>Changhua Yang, Kevin Hsin-Yih Lin, Hsin-Hsi Chen. 2007. Building Emotion Lexicon from Weblog Cor­pora. <i>Proceedings of the ACL 2007 Demo and Poster Sessions, </i>pp. 133 - 136.</p><p>Cecilia Ovesdotter Alm, Dan Roth, Richard Sproat. 2005. Emotions from text: Machine learning for text-based emotion prediction. <i>Proceedings of Hu­man Language Technology Conference and Con­ference on Empirical Methods in Natural Lan­guage Processing, </i>pp. 579-586, Vancouver, British Columbia, Canada.</p><p>Fuji Ren. 2009. Affective Information Processing and Recognizing Human Emotion. <i>Electronic Notes in Theoretical Computer Science, </i>225: 39-50.</p><p>Gilad Mishne. 2005. Emotions from text: Machine learning for text-based emotion prediction. <i>Proceed­ings ofStyle2005 in SIGIR'05, </i>pp. 15-19.</p><p>Gilad Mishne and Maarten de Rijke. 2006. Captur­ing global mood levels using blog posts. <i>AAAI 2006 Spring Symposium on Computational Approaches to Analysing Weblogs, </i>pp. 145-152.</p><p>Gilly Leshed and Joseph Kaye. 2006. Understanding how bloggers feel: recognizing affect in blog posts.</p><p><i>Conference on Human Factors in Computing Sys­tems CHI '06 extended abstracts on Human factors in computing systems, </i>pp. 1019- 1024.</p><p>Janyce Wiebe, Theresa Wilson, Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. <i>Language Resources and Evaluation. </i>39: 164-210.</p><p>Jean Carletta. 1996. Assessing Agreement on Classi­fication Tasks: The Kappa Statistic. <i>Computational Linguistics. </i>22(2):249-254.</p><p>Kazuyuki Matsumoto, Fuji Ren, Shingo Kuroiwa.</p><p>2006. Emotion Estimation System based on Emo­tion Occurrence Sentence Pattern. <i>Computational Intelligence, Lecture Notes in Computer Sciences, </i>pp.902-911.</p><p>Kevin Hsin-Yih Lin, Changhua Yang, Hsin-Hsi Chen.</p><p>2007. What emotions do news articles trigger in their readers? <i>Annual ACM Conference on Research and Development in Information Retrieval, </i>pp. 733- 734.</p><p>Krisztian Balog. Gilad Mishne. Maarten de Rijke. 2006. Why are they excited? identifying and ex­plaining spikes in blog mood levels. <i>Proceedings IIth Meeting of the European Chapter of the Asso­ciation for Computational Linguistics, </i>pp. 207-210.</p><p>Ortony, Andrew, Gerald L. Clore, and Mark A. Foss. 1987. The referential structure of the affective lexi­con. <i>Cognitive Science, </i>11: 341-364.</p><p>Rada Mihalcea and Hugo Liu. 2006. A corpus-based approach to finding happiness. <i>Proceedings of the AAAI Spring Symposium on Computational, </i>pp. 139-144.</p><p>Rosalind Picard. 1997. <i>Affective Computing. </i>The MIT Press, MA, USA.</p><p>Ryoko Tokuhisa, Kentaro. Inui, and Yuji. Matsumoto.</p><p>2008. Emotion Classification Using Massive Exam­ples Extracted from the Web. <i>Proceedings of COL-ING2008,pp. </i>881-888.</p><p>Saima Aman and Stan Szpakowicz. 2007. Identifying Expressions of Emotion in Text. <i>Lecture Notes in Computer Science. </i>4629: 196-205.</p><p>Soo-Min Kim and Eduard Hovy. 2005. Identifying Opinion Holders for Question Answering in Opinion Texts. <i>Proceedings of AAAI-05 Workshop on Ques­tion Answering in Restricted Domains, </i>pp. 1367-1373.</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1453</doubt><page local="9" global="1454"/><p>Yejin Choi, Claire Cardie, Ellen Riloff, and Siddharth Patwardhan. 2005. Identifying Sources of Opinions with ConditionalRandom Fields and Extraction Pat­terns. <i>Proceedings ofHLT/EMNLP-05, </i>pp. 355-362.</p><p>Yuchul Jung, Hogun Park, Sung Hyon Myaeng. 2006. A Hybrid Mood Classification Approach for Blog Text. <i>Lecture Notes in Computer Science, </i>pp. 1099-1103.</p><p>Yuchul Jung, Yoonjung Choi, Sung-Hyon Myaeng. 2007. Determining Mood for a Blog by Combin­ing Multiple Sources of Evidence. <i>IEEE/WIC/ACM</i> <i>International Conference on Web Intelligence, </i>pp. 271-274.</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1454</doubt></references></body></article>