<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="422"/><title>UPAR7: A knowledge-based system for headline sentiment tagging</title><pubinfo>Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007),pages 422-425, Prague, June 2007. ©2007 Association for Computational Linguistics</pubinfo><author surname="Chaumartin" givenname="François-Régis"><org  name="Lattice/Talana - Université Paris" country="France"/></author></firstpageheader><frontmatter><p><b>UPAR7: A knowledge-based system for headline sentiment tagging</b></p><p><b>François-Régis Chaumartin</b></p><p>Lattice/Talana - Université Paris 7 30, rue du château des rentiers - 75013 Paris - France</p><p>fchaumartin@linguist.jussieu.fr / frc@proxem.com</p></frontmatter><abstract>For the Affective Text task at SemEval-2007, University Paris 7's system first evaluates emotion and valence on all words of a news headline (using enriched versions of SentiWordNet and a subset of WordNet-Affect). We use a parser to find the head word, considering that it has a major im­portance. We also detect contrasts (be­tween positive and negative words) that shift valence. Our knowledge-based system achieves high accuracy on emotion and va­lence annotation. These results show that working with linguistic techniques and a broad-coverage lexicon is a viable ap­proach to sentiment analysis of headlines. </abstract></header><body><section number="1" title="Introduction"><subsection number="1.1" title="Objectives"><p>The detection of emotional connotations in texts is a recent task in computational linguistics. Its economic stakes are promising; for example, a company could detect, by analyzing the blo-gosphere, people's opinion on its products.</p><p>The goal of the SemEval task is to annotate news headlines for emotions (using a predefined list: anger, disgust, fear, joy, sadness &amp; surprise), and for valence (positive or negative). A specific difficulty here is related to the small number of words available for the analysis.</p></subsection><subsection number="1.2" title="Overall architecture"><p>Our system is mainly rule-based and uses a lin­guistic approach. From a macroscopic point of view, we follow the hypothesis that, in a news title, all the words potentially carry emotions. If linguis­tic resources make it possible to detect these emo­tions individually, how can we deal with headlines where bad and good emotions appear at once?</p><p>Our objective is to identify the expression which carries the main topic of the title. One can consider that this expression has a primary importance.</p><p>We also seek to lay down rules for detecting specific emotions. For instance, surprise some­times comes from the contrast between good and bad news. And sometimes, simple lexical elements are characteristic of an emotion; a negation or a modal auxiliary in a title may be a relevant indica­tor of surprise.</p><p>We describe here the techniques we imple­mented to address all these points.</p></subsection></section><section number="2" title="Components &amp; resources used"><p>The system we employed for the Affective Text evaluation consists of the following components<footnote anchor="1"/>:</p><p>• The SS-Tagger (a Part-of-Speech tagger)<footnote anchor="2"/>,</p><p>• The Stanford Parser.</p><p>We also used several lexical resources:</p><doubt alpha="63.6" length="22" tooSmall="False" monospace="0.0">• WordNet version 2.1,</doubt><p>• A subset of WordNet-Affect,</p><p>• SentiWordNet.</p><p>As the SS-Tagger is straightforward, we will not say more about it here. We will, however, discuss the remaining components and resources below.</p><p>We used them through the Antelope NLP framework (www.proxem.com), which makes them easy to use. <footnote anchor="2"/> This fast PoS tagger uses an extension of Maximum Entropy Markov Models. See (Tsuruoka, Tsujii, 2005).</p><page local="2" global="423"/><subsection number="2.1" title="Choice of the Stanford Parser"><p>We wished to use a syntactic parser for this task. We hesitated between two parsers producing a dependency graph, the <i>Link Grammar Parser </i>(Sleator, Temperley, 1991) and the <i>Stanford Parser </i>(Manning, Klein, 2002).</p><p>As a news title is sometimes reduced to a nomi­nal group, without a verb, our experiments showed that we should modify the title to make it "gram­matically correct". Such a step is essential to obtain accurate results with a rule-based analyzer such as the Link Grammar Parser. On the other hand, a statistical analyzer like the Stanford Parser is more tolerant with constructions which are not grammatically correct. That is why we chose it.</p></subsection><subsection number="2.2" title="WordNet"><p>We used WordNet (Miller, 1995) as a semantic lexicon. This well-known project, started in 1985 at Princeton, offers a broad-coverage semantic network of the English language, and is probably one of the most popular NLP resources.</p><p>In WordNet, words are grouped into sets of synonyms. Various semantic relations exist be­tween these synsets (for example, hypernymy and hyponymy, antonymy, derivation...).</p></subsection><subsection number="2.3" title="WordNet-Affect"><p>WordNet-Affect (Strapparava, Valitutti, 2004) is a hierarchy of "affective domain labels", with which the synsets representing affective concepts are further annotated. We used the subset of WordNet-Affect provided as emotions lists by the SemEval organizers. To improve it, we manually added to the emotion lists new words that we found impor­tant on the task trial data.</p><p>The synsets of emotions lists were considered as seeds; our system recursively propagated their emotions to their neighbor synsets<footnote anchor="3"/>.</p><p><b>SentiWordNet</b></p><p>SentiWordNet (Esuli, Sebastiani, 2006) describes itself as a lexical resource for opinion mining. SentiWordNet assigns to each synset of WordNet three sentiment scores <footnote anchor="4"/> : positivity, negativity, objectivity, the sum of which always equals 1.0.</p><p>This resource has been created with a mix of linguistics and statistics (using classifiers). The advantage of this approach is to allow the auto­matic generation of emotion values for all the synsets of WordNet. The disadvantage is that, as all the results are not manually validated, some resulting classifications can appear incorrect<footnote anchor="5"/>.</p><p>We recursively propagate the positivity and negativity values throughout neighbor synsets<footnote anchor="6"/>.</p></subsection></section><section number="3" title="UPAR7 Affective Text system"><subsection number="3.1" title='"De-capitalization" of common words'><p>A preliminary problem that we had to solve was related to the Anglo-Saxon habit of putting initial capital letters in all the words of a title.</p><p>The first pass of our system thus detected news titles that were "improperly" capitalized, and "de-capitalizes" their common words.</p><p>For that, we used the SS-Tagger on the title; ac­cording to the part of speech of each word, infor­mation found in WordNet, and some hand-crafted rules<footnote anchor="7"/>, the system chooses or not to keep the initial.</p><p>The impact of this processing step is far from negligible, from the point of view of the Stanford Parser. Indeed, let us have a look at the difference between the parsing of the title, before (figure 1) and after (figure 2) this processing.</p><table caption='Table 2: Some words added for "fear" emotion.'></table><footnote label="3">Following relations such as Hyponym, Derivation, Adjective Similar, Adjective Participle, Derivation and Pertainym.</footnote><footnote label="4">For instance, the synset estimable #1 (deserving of respect or high regard) has: Positivity = 0.75, Negativity = 0.00, Objectivity = 0.25.</footnote><footnote label="5">For example, rape #3 (the crime of forcing a woman to submit to sexual intercourse against her will) is classified with Positivity=0.25 and Negativity=0.0 despite the presence of the word "crime" in its gloss.</footnote><footnote label="6">Using WordNet's relations such as Hyponym (for noun and verb), Antonym and Derivation. For antonyms, positivity and negativity values are exchanged.</footnote><footnote label="7">For instance, a word that cannot be any form of a WordNet lemma is probably a proper noun, and then we keep its initial.</footnote><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Nouns    Verbs   Adjectives Adverbs</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Anger</p></td><td class="cell"><p>37        26             16 0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Disgust</p></td><td class="cell"><p>35 19</p></td><td class="cell"><p>9 0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Fear</p></td><td class="cell"><p>71         26             20 4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Joy</p></td><td class="cell"><p>50        22             14 1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Sadness</p></td><td class="cell"><p>88        37             29 4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Surprise</p></td><td class="cell"><p>16        29             13 2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Table 1: Counting of new words for each emotion.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Nouns</p></td><td class="cell"><p>Verbs</p></td><td class="cell"><p>Adjectives</p></td><td class="cell"><p>Adverbs</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>cancer danger poverty</p></td><td class="cell"><p>demolish injure kidnap</p></td><td class="cell"><p>comatose nuclear violent</p></td><td class="cell"><p>bloody dead worse</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="3" global="424"/><p>Figure 1 : Output of the Stanford Parser with a title that is <b>"improperly" </b>capitalized.</p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">r</doubt><doubt alpha="83.3" length="6" tooSmall="False" monospace="0.0">nsubj-</doubt><p><b>-cccmpï-</b> <b>Bush/HHE insists/VBZ trccps/HHS stay/YB in/IN Iraq/HNF        pcedicts/VBZ ni </b><b>Lit </b><b>e oi/ J J victcry/HH</b></p><doubt alpha="85.7" length="7" tooSmall="False" monospace="0.0">risubj-</doubt><doubt alpha="75.0" length="4" tooSmall="False" monospace="0.0">;rep</doubt><doubt alpha="50.0" length="10" tooSmall="False" monospace="0.0">&gt;|-pDb:i—J</doubt><doubt alpha="66.7" length="9" tooSmall="False" monospace="0.0">-nsubj ï-</doubt><doubt alpha="62.5" length="8" tooSmall="False" monospace="0.0">-&lt;anicd-</doubt><p>Figure 2 : Output of the Stanford Parser with a title that is <b>"properly" </b>capitalized. (Words are tagged with the right part-of-speech, and dependencies are now correct.)</p></subsection><subsection number="3.2" title="Individual words rating"><p>For the moment, we consider the output of the Stanford Parser as an array of PoS-tagged words. We use WordNet's morphology functions to find the possible base form of each word.</p><p>At this stage, an important question arose: was lexical disambiguation possible? We thought not, because with short sentences, few relevant heuris­tics apply. We chose another solution, by consider­ing that the emotion and valence values of a word were the linear combination of that of all its possi­ble meanings, balanced by the frequency of each lemma.</p><p>We detected emotion and valence values for each word, by using our enriched version of</p><p>WordNet-Affect and SentiWordNet.</p><p>In fact, we also detected some extra information:</p><p>• An additional 7th emotion, that looks like "compassion for people needing protection". Our assumption is that certain words express a subjacent need for protection. For exam­ple, there is "student" behind "school", and "child" behind "adoption". So, we built a list of words designating something that needs protection; we also include in this list words such as "troops", "shoppers".</p><p>• We tried to detect acronyms relating to technology; for this, we defined a list of high-tech companies and a very basic regu­lar expression rule saying that a word (not in WordNet) containing numbers, or capitals not in first position, should be something high-tech. (This very basic rule seems to</p><p>work nicely on PS3, iPod, NASA.). We use these high-tech indications to increase the "joy" emotion.</p><p>• We counted lexical elements that we think are good indicators of surprise: negations, modal auxiliaries, question marks.</p><p>At this stage, we begin some post-processing on individual words. Which factors cause anger rather that sadness? We believe that human intention (to harm) causes the former emotion, while natural factors such as disease or climatic catastrophes cause the latter. So, we used a few rules related to the WordNet noun hierarchy, based on the fact that when a noun is a hyponym of a given synset, we boost some emotions:</p><table caption="Table 3: Hypernyms triggering an emotion boost. Then, the emotions found serve to update the" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Does noun inherit from?</p></td><td class="cell"><p>Emotions to boost</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Unhealthiness</p></td><td class="cell"><p>Fear, sadness</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Atmospheric phenome­non</p></td><td class="cell"><p>Fear, sadness</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Aggression, hostility, wrongful conduct</p></td><td class="cell"><p>Anger, fear, sadness, disgust</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Weaponry, weapon</p><p>system</p></td><td class="cell"><p>Anger, fear, sadness</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Unfortunate person</p></td><td class="cell"><p>Sadness, "compassion"</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Human will</p></td><td class="cell"><p>Anger</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 4: Emotions that change valence." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Emotion</p></td><td class="cell"><p>Positivity</p></td><td class="cell"><p>Negativity</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Joy</p></td><td class="cell"><p>++</p></td><td class="cell"><p>--</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Anger,   disgust, sadness, fear, "compassion"</p></td><td class="cell"><p>--</p></td><td class="cell"><p>++</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="4" global="425"/></subsection><subsection number="3.3" title="Global sentence rating"><p>At this stage, our system tries to find the main subject of the news title. Again, we use the output of the Stanford Parser, but this time, we make use of the dependency graph. We consider that the main word is the root of the dependency graph, i.e. the word that is never a dependant word. (For instance, in figure 2, the main word is "predicts".)</p><p>We think that the contribution of this main word is much more important than that of the other words of the title<footnote anchor="8"/>. So, we multiply its individual valence and emotion by 6.</p><p>The last important part of linguistic processing is the detection of contrasts and accentuations between "good" or "bad" things. We search pat­terns like [noun-&gt;subject-&gt;verb] or [verb-^direct object-^noun] in the dependency graph, with verbs that increase or decrease a quantity<footnote anchor="9"/>. Using the valence of the given noun, this gives our system the ability to detect very good news ("boosts (brain) power") or good news where something bad gets less important ("reduces risk", "slows decline", "hurricane weakens".).</p></subsection></section><section number="4" title="Results"><p>Our rule-based system detects the six emotions in news headlines with an average accuracy reach­ing 89.43% (coarse-grained evaluation). However, <u>recall is low.</u><u></u>__</p><p>Fine-grained _<u>Coase-grained^</u>_ _ <u>Pearson  Accuracy   Precision Recall</u></p><p>Valence |    36.96 |      55.00 |      57.54 | 8.78 Table 6: Results of the valence annotation.</p><footnote label="8">In sentences like "study says...", "scientists say...", "police affirm...", the main head word is the verb of the relative.</footnote><footnote label="9">We "rediscovered" valence shifters (words that modify the sentiment expressed by a sentiment-bearing word, see (Po-lanyi and Zaenen, 2006)).</footnote><p>The valence detection accuracy (55% in coarse­grained evaluation) is lower than in emotion anno­tation. We attribute this difference to the fact that it is easier to detect emotions (that are given by individual words) rather than valence, which needs a global understanding of the sentence.</p></section><section number="5" title="Conclusion"><p>Emotion and valence tagging is a complex and interesting task. For our first attempt, we designed and developed a linguistic rule-based system, using</p><p>WordNet, SentiWordNet and WordNet-Affect lexical resources, that delivers high accuracy results. In our future work, we will explore the potential of simultaneously using a statistical approach, in order to improve recall of sentiment annotation.</p><table caption="Table 5: Results of the emotion annotation." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Fine-grained</p></td><td class="cell"><p>Coase-grained</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Pearson</p></td><td class="cell"><p>Accuracy</p></td><td class="cell"><p>Precision</p></td><td class="cell"><p>Recall</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Anger</p></td><td class="cell"><p>32.33</p></td><td class="cell"><p>93.60</p></td><td class="cell"><p>16.67</p></td><td class="cell"><p>1.66</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Disgust</p></td><td class="cell"><p>12.85</p></td><td class="cell"><p>95.30</p></td><td class="cell"><p>0.00</p></td><td class="cell"><p>0.00</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Fear</p></td><td class="cell"><p>44.92</p></td><td class="cell"><p>87.90</p></td><td class="cell"><p>33.33</p></td><td class="cell"><p>2.54</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Joy</p></td><td class="cell"><p>22.49</p></td><td class="cell"><p>82.20</p></td><td class="cell"><p>54.54</p></td><td class="cell"><p>6.66</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Sadness</p></td><td class="cell"><p>40.98</p></td><td class="cell"><p>89.00</p></td><td class="cell"><p>48.97</p></td><td class="cell"><p>22.02</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Surprise</p></td><td class="cell"><p>16.71</p></td><td class="cell"><p>88.60</p></td><td class="cell"><p>12.12</p></td><td class="cell"><p>1.25</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><references><p>Andrea Esuli, Fabrizio Sebastiani. 2006. <i>SentiWordNet: A Publicly Available Lexical Resource for Opinion Mining. </i>Proceedings of LREC 2006, fifth interna­tional conference on Language Resources and Evalu­ation, pp. 417-422.</p><p>Christopher Manning, Dan Klein. 2002. <i>Fast Exact Inference with a Factored Model for Natural Lan­guage Parsing. </i>Advances in Neural Information Processing Systems 15 (NIPS 2002).</p><p>George Miller. 1995. <i>WordNet: A lexical database. </i>Acts of ACM 38, 39-41.</p><p>Livia Polanyi, Annie Zaenen. 2006. Contextual Valence Shifters. In J. G. Shanahan, Y. Qu, and J. Wiebe, edi­tors, <i>Computing Attitude and Affect in Text: Theory andApplication. </i>Springer Verlag.</p><p>Daniel Sleator, Davy Temperley. 1991. <i>Parsing English with a Link Grammar. </i>Acts of Third International Workshop on Parsing Technologies.</p><p>Carlo Strapparava, Alessandro Valitutti. 2004. <i>Word-Net-Affect: an affective extension of WordNet. </i>Pro­ceedings of the 4th International Conference on Lan­guage Resources and Evaluation (LREC 2004), pp.</p><p>1083-1086.</p><p>Yoshimasa Tsuruoka, Jun'ichi Tsujii. 2005. <i>Bidirec­tional Inference with the Easiest-First Strategy for Tagging Sequence Data. </i>Proceedings of HLT/EMNLP 2005, pp. 467-474.</p></references></body></article>