<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="444"/><title>OpAL: Applying Opinion Mining Techniques for the Disambiguation of Sentiment Ambiguous Adjectives in SemEval-2 Task 18</title><pubinfo>Proceedings of the 5th International Workshop on Semantic Evaluation, ACL 2010,pages 444—447, Uppsala, Sweden, 15-16 July 2010. ©2010 Association for Computational Linguistics</pubinfo><author surname="Balahur" givenname="Alexandra"><org  name="University of Alicante" country="Spain" city="Alicante"/></author><author surname="Montoyo" givenname="Andrés"><org  name="University of Alicante" country="Spain" city="Alicante"/></author></firstpageheader><frontmatter><p><b>OpAL: Applying Opinion Mining Techniques for the Disambiguation of Sentiment Ambiguous Adjectives in SemEval-2 Task 18</b></p><p><b>Alexandra Balahur</b></p><p>University of Alicante Department of Software and Computing Systems</p><p>abalahur@dlsi.ua.es</p><p><b>Andrés Montoyo</b></p><p>montoyo@dlsi.ua.es</p></frontmatter><abstract>The task of extracting the opinion expressed in text is challenging due to different reasons. One of them is that the same word (in particu­lar, adjectives) can have different polarities depending on the context. This paper presents the experiments carried out by the OpAL team for the participation in the SemEval 2010 Task 18 - <i>Disambiguation of Sentiment Ambiguous Adjectives. </i>Our approach is based on three dif­ferent strategies: a) the evaluation of the polar­ity of the whole context using an opinion min­ing system; b) the assessment of the polarity of the local context, given by the combinations between the closest nouns and the adjective to be classified; c) rules aiming at refining the lo­cal semantics through the spotting of modifi­ers. The final decision for classification is tak­en according to the output of the majority of these three approaches. The method used yielded good results, the OpAL system run ranking fifth among 16 in micro accuracy and sixth in macro accuracy. </abstract></header><body><section number="1" title="Credits"><p>This research has been supported by Ministerio de Ciencia e Innovation - Spanish Government (grant no. TIN2009-13391-C04-01), and Consel-leria d'Educacion-Generalitat Valenciana (grant no. PROMETEO/2009/119 and ACOMP/2010/ 288).</p></section><section number="2" title="Introduction"><p>Recent years have marked the beginning and ex­pansion of the Social Web, in which people free­ly express and respond to opinion on a whole variety of topics. Moreover, at the time of taking a decision, more and more people search for in­formation and opinions expressed on the Web on their matter of interest and base their final deci­sion on the information found (Pang and Lee, 2008). Nevertheless, the high quantity of data that has to be analysed imposed the development of specialized Natural Language Processing (NLP) systems that automatically extract, classi­fy and summarize the opinions available on the web on different topics. Research in this field, of <i>opinion mining </i>(sentiment analysis), has ad­dressed the problem of extracting and classifying opinions from different perspectives and at dif­ferent levels, depending on various factors. While determining the overall opinion on a mov­ie is sufficient for taking the decision to watch it or not, when buying a product, people are inter­ested in the individual opinions on the different product characteristics. Especially in this con­text, opinion mining systems are confronted with a difficult problem: the fact that the adjectives used to express opinion have different polarities depending on the characteristic they are men­tioned with. For example, "high price" is nega­tive, while "high resolution" is positive. There­fore, specialized methods have to be employed to correctly determine the contextual polarity of such words and thus accurately assign polarity to the opinion.</p><p>This is the aim of the SemEval 2010 Task 18 -Disambiguation of Sentiment Ambiguous Adjec­tives (Wu and Jin, 2010). In the following sec­tions, we first present state-of-the art approaches towards polarity classification of opinions, sub­sequently describing our approach in the SemEv-al task. Finally, we present the results we ob­tained in the evaluation and our plans for future work.</p><page local="2" global="445"/></section><section number="3" title="State of the Art"><p><i>Subjectivity analysis </i>is defined by (Wiebe, 1994) as the "linguistic expression of somebody's opi­nions, sentiments, emotions, evaluations, beliefs and speculations". <i>Sentiment analysis, </i>on the other hand, is defined as the task of extracting, from a text, the opinion expressed on an object (product, person, topic etc.) and classifying it as positive, negative or neutral. The task of senti­ment analysis, considered a step further to sub­jectivity analysis, is more complex than the lat­ter, because it involves an extra step: the classifi­cation of the retrieved opinion words according to their polarity. There are a series of techniques that were used to obtain lexicons of subjective words - e.g. the Opinion Finder lexicon (Wilson et al., 2005) and opinion words with associated polarity. (Hu and Liu, 2004) start with a set of seed adjectives ("good" and "bad") and apply synonymy and antonymy relations in WordNet. A similar approach was used in building Word­Net Affect (Strapparava and Valitutti, 2004), starting from a larger set of seed affective words, classified according to the six basic categories of emotion (joy, sadness, fear, surprise, anger and disgust) and expanding the lexicon using paths in WordNet. Another related method was used in the creation of SentiWordNet (Esuli and Sebas-tiani, 2005), using a set of seed words whose po­larity was known and expanded using gloss simi­larity. The collection of appraisal terms in (Whi-telaw et al., 2005), the terms also have polarity assigned. MicroWNOp (Cerini et al., 2007), another lexicon containing opinion words with their associated polarity, was built on the basis of a set of terms extracted from the General Inquirer lexicon and subsequently adding all the synsets in WordNet where these words appear. Other methods built sentiment lexicons using the local context of words. (Pang et al., 2002) built a lex­icon of sentiment words with associated polarity value, starting with a set of classified seed adjec­tives and using conjunctions ("and") disjunctions ("or", "but") to deduce orientation of new words in a corpus. (Turney, 2002) classifies words ac­cording to their polarity on the basis of the idea that terms with similar orientation tend to co-occur in documents. Thus, the author computes the Pointwise Mutual Information score between seed words and new words on the basis of the number of AltaVista hits returned when querying the seed word and the word to be classified with the "NEAR" operator. In our work in (Balahur and Montoyo, 2008a), we compute the polarity of new words using "polarity anchors" (words whose polarity is known beforehand) and Nor­malized Google Distance (Cilibrasi and Vitanyi, 2006) scores. Another approach that uses the po­larity of the local context for computing word polarity is (Popescu and Etzioni, 2005), who use a weighting function of the words around the context to be classified.</p><doubt alpha="59.1" length="44" tooSmall="False" monospace="0.0">4    The OpAL system at SemEval 2010 Task 18</doubt><p>In the SemEval 2010 Task 18, the participants were given a set of contexts in Chinese, in which 14 dynamic sentiment ambiguous adjectives are selected. They are: ^jbig, /h|small, #|many, <i>;P </i>|few, rftjhigh, {Ë|low, J?|thick, #|thin, ^jdeep, ^shallow, J!|heavy, g|light, |ÊL;^|huge, J!;^ |grave. The task was to automatically classify the polarity of these adjectives, i.e. to detect whether their sense in the context is positive or negative. The contexts were given in two forms: as plain text, in which the adjective to be classified was marked; in the second for, the text was tokenized and the tokens were tagged with part of speech (POS). There was no training set provided.</p><p>Our approach uses a set of opinion mining re­sources and an opinion mining system that is implemented to work for English. This is why, the first step we took in our approach was to translate the given contexts into English using the Google Translator1. In order to perform this task, we first split the initial file into 10 smaller files, using a specialized program - GSplit3 2. The OpAL adjective polarity disambiguation system combines supervised methods with unsu-pervised ones. In order to judge the polarity of the adjectives, it uses three types of judgments. The first one is the general polarity of the con­text, determined by our in-house opinion mining system - based on SVM machine learning on the NTCIR data and the EmotiBlog (Boldrini et al., 2009) annotations and different subjectivity, opi­nion and emotion lexica (Opinion Finder, Mi­croWordNet Opinion, General Inquirer, Word­Net Affect, emotion triggers (Balahur and Mon-toyo, 2008b). The second one is the local polari­ty, given by the highest number of results ob­tained when issuing queries containing the clos­est noun with the adjective to be disambiguated followed by the conjunction "AND" and a prede­fined set of 6 adjectives whose polarity is nonambiguous - 3 positive - "positive", "beautiful", "good" and 3 negative - "negative", "ugly", "bad".<page local="3" global="446"/> An example of such queries is "price high and good". The third component is made up of rules, depending on the presence of specific modifiers in a window of 4 words before the ad­jective. The final verdict is given based on the vote given by the majority of the three compo­nents, explained in detail in the next sections:</p><footnote label="1"> http://translate.google.com/</footnote><footnote label="2"> www.gdgsoft.com/gsplit/</footnote><subsection number="4.1" title="The OpAL opinion mining component"><p>First, we process each context using Minipar<footnote anchor="3"/>. We compute, for each word in a sentence, a se­ries of features, computed from the NTCIR 7 data and the EmotiBlog annotations. These words are used to compute vectors of features for each of the individual contexts:</p><p>• the part of speech (POS) opinionatedness/intensity - if the word is annotated as opinion word, its polarity, i.e. 1 and -1 if the word is positive or negative, re­spectively and 0 if it is not an opinion word, its intensity (1, 2 or 3) and 0 if it is not a subjective word</p><p>syntactic relatedness with other opinion word - if it is directly dependent of an opi­nion word or modifier (0 or 1), plus the po­larity/intensity and emotion of this word (0 for all the components otherwise)</p><p>• role in 2-word, 3-word, 4-word and sen­tence annotations: opinionatedness, intensity and emotion of the other words contained in the annotation, direct dependency relations with them if they exist and 0 otherwise.</p><p>We add to the opinion words annotated in EmotiBlog the list of opinion words found in the Opinion Finder, Opinion Finder, MicroWordNet Opinion, General Inquirer, WordNet Affect, emotion triggers lexical resources. We train the model using the SVM SMO implementation in</p><p>Weka4.</p></subsection><subsection number="4.2" title="Assessing local polarity using Google queries"><p>This approach aimed at determining the polarity of the context immediately surrounding the ad­jective to be classified. To that aim, we con­structed queries using the noun found before the adjective in the context given, and issued six dif­ferent queries on Google, together with six pre­defined adjectives whose polarity is known (3 positive - "positive", "beautiful", "good" and 3 negative - "negative", "ugly", "bad"). The form of the queries was "noun+adjective+AND+pre-defined adjective". The local polarity was consi­dered as the one for which the query issued the highest number of total results (total number of results for the 3 queries corresponding to the pos­itive adjectives or to the negative adjectives, re­spectively).</p><footnote label="3"> http://webdocs.cs.ualberta.ca/~lindek/minipar.htm</footnote><footnote label="4"> http://www.cs.waikato.ac.nz/ml/weka/</footnote></subsection><subsection number="4.3" title="Modifier rules for contextual polarity"><p>This rule accounts for the original, most fre­quently used polarity of the given adjectives (e.g. <i>high </i>is <i>positive, low </i>is <i>negative). </i>For each of them, we define its default polarity. Subsequent­ly, we determine whether in the window of 4 words around the adjective there are any modifi­ers (valence shifters). If this is the case, and they have an opposite value of polarity, the adjective is assigned a polarity value opposite from its de­fault one (e.g. <i>too high </i>is <i>negative). </i>We employ a list of 82 positive and 87 negative valence shif­ters.</p></subsection></section><section number="5" title="Evaluation"><p>Table 1 and Table 2 present the results obtained by the OpAL system in the SemEval 2010 Task 18 competition. The system ranked fifth, with a Micro accuracy of 0.76037 and sixth, with a Ma­cro accuracy of 0.7037.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>System name</p></td><td class="cell"><p>Micro accura­cy</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>98-35 result</p></td><td class="cell"><p>0.942064</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>437-381_HITSZ_CITYU_ Task18_Run1 .key</p></td><td class="cell"><p>0.936236</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>437-380_HITSZ_CITYU_</p></td><td class="cell"><p>0.93315</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Task18_Run2.key</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>53-211_dsaa</p></td><td class="cell"><p>0.880699</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>186-325_OpAL_results.txt</p></td><td class="cell"><p>0.76037</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>291-389_submission4.txt</p></td><td class="cell"><p>0.724717</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>291-388 submission3.txt</p></td><td class="cell"><p>0.715461</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>437-382_HITSZ_CITYU_ Task18 Run3</p></td><td class="cell"><p>0.665752</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Table 1: Results - top 8 runs (micro accuracy)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>System name</p></td><td class="cell"><p>Macro accu­racy</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>437-380_HITSZ_CITYU_</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Task18 Run2.key</p></td><td class="cell"><p>0.957881</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>437-381_HITSZ_CITYU_</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Task18 Run1.key</p></td><td class="cell"><p>0.953238</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>98-35 result</p></td><td class="cell"><p>0.929308</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>53-211 dsaa</p></td><td class="cell"><p>0.861964</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="4" global="447"/><doubt alpha="39.4" length="33" tooSmall="False" monospace="0.0">291-388 submission3.txt__0.755387</doubt><doubt alpha="41.2" length="34" tooSmall="False" monospace="0.0">186-325_OpAL_results.txt__0.703777</doubt><doubt alpha="38.2" length="34" tooSmall="False" monospace="0.0">291 -389_submission4.txt__0.698037</doubt><doubt alpha="62.5" length="104" tooSmall="False" monospace="0.0">460383_New_Task18_Chinese_test_pos_QiuLikun_R.rar 0.695448Table 2: Results - top 8 runs (macro accuracy)</doubt><p>Since the gold standard was not provided, we were not able to perform an exhaustive analysis of the errors. However, from a random inspec­tion of the system results, we could see that a large number of errors was due to the translation - through which modifiers are placed far from the word they determine or the words are not translated with their best equivalent.</p></section><section number="6" title="Conclusions and future work"><p>In this article we presented our approach towards the disambiguation of polarity ambiguous adjec­tives depending on the context in which they ap­pear. The OpAL system's run was based on three subcomponents working in English - one assess­ing the overall polarity of the context using an opinion mining system, the second assessing the local polarity using Google queries formed by expressions containing the noun present in the context before the adjective to be classified and the third one evaluating contextual polarity based on the adjective's default value and the modifiers around it. The final output is based on the vote given by the majority of the three components. The approach had a good performance, the OpAL system run ranking fifth among 16 runs. Future work includes the separate evaluation of the three components and their combination in a unique approach, using machine learning, as well as a thorough assessment of errors that are due to translation.</p></section><references><p>Balahur, A. and Montoyo, A. 2008a. <i>A feature-driven approach to opinion mining and classification. </i>In Proceedings of the NLPKE 2008.</p><p>Balahur, A. and Montoyo, A. 2008b. <i>Applying a cul­ture dependent emotion triggers database for text valence and emotion classification. </i>Procesamiento del Lenguaje Natural, 40(40).</p><p>Boldrini, E., Balahur, A., Martinez-Barco, P., and Montoyo, A. 2009. <i>EmotiBlog: an annotation scheme for emotion detection and analysis in non-traditional textual genres. </i>In Proceedings of the 5th International Conference on Data Mining (DMIN 2009).</p><p>Cerini, S., Compagnoni, V., Demontis, A., Formentel-li, M., and Gandini, G. 2007. <i>Micro-WNOp: A gold standard for the evaluation of automatically com­piled lexical resources for opinion mining.</i></p><p>Cilibrasi, D. and Vitanyi, P. 2006. <i>Automatic Mean­ing Discovery Using Google. </i>IEEE Journal of Transactions on Knowledge and Data Engineering.</p><p>Esuli, A. and Sebastiani, F. 2006. <i>SentiWordNet: a publicly available resource for opinion mining. </i>In Proceedings of the 6th International Conference on Language Resources and Evaluation.</p><p>Hu, M. and Liu, B. 2004. <i>Mining Opinion Features in Customer Reviews. </i>In Proceedings of Nineteenth National Conference on Artificial Intellgience</p><p>AAAI-2004.</p><doubt alpha="65.0" length="137" tooSmall="False" monospace="0.0">Pang, B. and Lee, L. 2008.Opinion mining and sen­timent analysis.Foundations and Trends in Infor­mation Retrieval 2(1-2), pp. 1-135, 2008</doubt><p>Pang, B., Lee, L., and Vaithyanathan, S. 2002. <i>Thumbs up? Sentiment classification using ma­chine learning techniques. </i>In Proceedings of EMNLP-02, the Conference on Empirical Methods in Natural Language Processing.</p><p>Popescu, A. M. and Etzioni, O. 2005. <i>Extracting product features and opinions from reviews. </i>In In Proceedings of HLTEMNLP 2005.</p><doubt alpha="59.6" length="52" tooSmall="False" monospace="0.0">Stone, P., Dumphy, D. C., Smith, M. S., and Ogilvie,</doubt><p>D. M. 1966. <i>The General Inquirer: A Computer Approach to Content Analysis. </i>The MIT Press.</p><p>Strapparava, C. and Valitutti, A. 2004<i>.WordNet-</i> <i>Affect:</i><i> an affective extension of WordNet. </i>In Pro­ceedings of the 4th International Conference on Language Resources and Evaluation (LREC 2004).</p><p>Turney, P. 2002. <i>Thumbs up or thumbs down? Seman­tic orientation applied to unsupervised classifica­tion of reviews. </i>In Proceedings 40th Annual Meet­ing of the Association for Computational Linguis­tics.</p><p>Whitelaw, C., Garg, N., and Argamon, S. 2005. <i>Using appraisal groups for sentiment analysis. </i>In Pro­ceedings of the CIKM 2005.</p><p>Wiebe, J. (1994). <i>Tracking point of view in narrative. </i>Computational Linguistics, 20.</p><p>Wilson, T., Wiebe, J., and Hoffmann, P. 2005. <i>Re­cognizing contextual polarity in phrase-level sen­timent analysis. </i>In Proceedings of HLT-EMNLP 2005.</p><p>Wu, Y., Jin, P. 2010. <i>SemEval-2010 Task 18: Disam-biguating Sentiment Ambiguous Adjectives. </i>In Pro­ceedings of the SemEval 2010 Workshop, ACL 2010.</p></references></body></article>