Twitter is one of the most popular social network websites and has been growing at a very fast pace. The number of Twitter users reached an estimated 75 million by the end of 2009, up from approx­imately 5 million in the previous year. Through the twitter platform, users share either information or opinions about personalities, politicians, prod­ucts, companies, events (Prentice and Huffman, 2008) etc. This has been attracting the attention of different communities interested in analyzing its content.

Sentiment detection of tweets is one of the basic analysis utility functions needed by various appli­cations over twitter data. Many systems and ap­proaches have been implemented to automatically detect sentiment on texts (e.g., news articles, Web reviews and Web blogs) (Pang et al., 2002; Pang and Lee, 2004; Wiebe and Riloff, 2005; Glance et al., 2005; Wilson et al., 2005). Most of these Junlan Feng approaches use the raw word representation (n-grams) as features to build a model for sentiment detection and perform this task over large pieces of texts. However, the main limitation of using these techniques for the Twitter context is mes­sages posted on Twitter, so-called tweets, are very short. The maximum size of a tweet is 140 char­acters.

In this paper, we propose a 2-step sentiment analysis classification method for Twitter, which first classifies messages as subjective and ob­jective, and further distinguishes the subjective tweets as positive or negative. To reduce the la­beling effort in creating these classifiers, instead of using manually annotated data to compose the training data, as regular supervised learning ap­proaches, we leverage sources of noisy labels as our training data. These noisy labels were pro­vided by a few sentiment detection websites over twitter data. To better utilize these sources, we verify the potential value of using and combining them, providing an analysis of the provided labels, examine different strategies of combining these sources in order to obtain the best outcome; and, propose a more robust feature set that captures a more abstract representation of tweets, composed by meta-information associated to words and spe­cific characteristics of how tweets are written. By using it, we aim to handle better: the problem of lack of information on tweets, helping on the generalization process of the classification algo­rithms; and the noisy and biased labels provided by those websites.

The remainder of this paper is organized as fol­lows. In Section 2, we provide some context about messages on Twitter and about the websites used as label sources. We introduce the features used in the sentiment detection and also provide a deep analysis of the labels generated by those sources in Section 3. We examine different strategies of combining these sources and present an extensive experimental evaluation in Section 4. Finally, we discuss previous works related to ours in Section 5 and conclude in Section 6, where we outline direc­tions and future work.

In this section, we give some context about Twitter messages and the sources used for our data-driven approach.

Tweets. The Twitter messages are called tweets. There are some particular features that can be used to compose a tweet (Figure 1 illustrates an ex­ample): "RT" is an acronym for retweet, which means the tweet was forwarded from a previous post; "@twUser" represents that this message is a reply to the user "twUser"; "#obama" is a tag pro­vided by the user for this message, so-called hash-tag; and "http://bit.ly/9K4n9p" is a link to some external source. Tweets are limited to 140 charac­ters. Due to this lack of information in terms of words present in a tweet, we explore some of the tweet features listed above to boost the sentiment detection, as we will show in detail in Section 3.

Data Sources. We collected data from 3 differ­ent websites that provide almost real-time senti­ment detection for tweets: Twendz, Twitter Sen­timent and TweetFeel. To collect data, we issued a query containing a common stopword "of", as we are interested in collecting generic data, and retrieved tweets from these sites for three weeks, archiving the returned tweets along with their sen­timent labels. Table 1 shows more details about these sources. Two of the websites provide 3-class detection: positive, negative and neutral and one of them just 2-class detection. One thing to note is our crawling process obtained a very dif­ferent number of tweets from each website. This might be a result of differences among their sam­pling processes of Twitter stream or some kind of filtering process to output. For instance, a site may only present the tweets it has more confi­dence about their sentiment. In Section 3, we present a deep analysis of the data provided by these sources, showing if they are useful to build a sentiment classification.

Figure 1 : Example of a tweet.

Our goal is to categorize a tweet into one of the three sentiment categories: positive, neutral or negative. Similar to (Pang and Lee, 2004; Wil­son et al., 2005), we implement a 2-step sentiment detection framework. The first step targets on dis­tinguishing subjective tweets from non-subjective tweets (subjectivity detection). The second one further classifies the subjective tweets into posi­tive and negative, namely, the polarity detection. Both classifiers perform prediction using an ab­stract representation of the sentences as features, as we show later in this section.

tweets that are disagreed between the data sources in terms of subjectivity;

2. Same user's messages: we observed that the users with the highest number of messages in our dataset are usually those ones that post some objective messages, for example, ad­vertising some product or posting some job recruiting information. For this reason, we allowed in the training data only one message from the same user. As we show later, this boosts the classification performance, mainly because it removes tweets labeled as subjec­tive by the data sources but are in fact objec­tive;

3. Top opinion words: to clean the objective training set, we remove from this set tweets that contain the top-n opinion words in the subjectivity training set, e.g., words as cool, suck, awesome etc.

As we show in Section 4, this process is in fact able to remove certain noisy in the training data, leading to a better performing subjectivity classi­fier.

To illustrate which of the proposed features are more effective for this task, the top-5 features in terms of information gain, based on our training data, are: positive polarity, link, strong subjec­tive, upper case and verbs. Three of them are meta-information (positive polarity, strong sub­jective and verbs) and the other two are tweet syntax features (link and upper case). Here is a typical example of a objective tweet in which the user pointed an external link and used many upper case words: "Starbucks Expands Pay-By-IPhone Pilot to 1,000 Stores—Starbucks cus­tomers with Apple iPhones or iPod touches can http://oohja.com/x9UbC".

We have performed an extensive performance evaluation of our solution for twitter sentiment detection. Besides analyzing its overall perfor­mance, our goals included: examining different strategies to combine the labels provided by the sources; comparing our approach to previous ones in this area; and evaluating how robust our solu­tion is to the noisy and biased data described in Section 3.

There is a rich literature in the area of sentiment detection (see e.g., (Pang et al., 2002; Pang and Lee, 2004; Wiebe and Riloff, 2005; Go et al., 2009; Glance et al., 2005). Most of these ap­proaches try to perform this task on large texts, as e.g., newspaper articles and movie reviews. An­other common characteristic of some of them is the use of n-grams as features to create their mod­els. For instance, Pang and Lee (Pang and Lee, 2004) explores the fact that sentences close in a text might share the same subjectivity to create a better subjectivity detector and, similar to (Pang et al., 2002), uses unigrams as features for the polar­ity detection. However, these approaches do not obtain a good performance on detecting sentiment on tweets, as we showed in Section 4, mainly be­cause tweets are very short messages. In addition to that, since they use a raw word representation, they are more sensible to bias and noise, and need a much higher number of examples in the train­ing data than our approach to obtain a reasonable performance.

The Web sources used in this paper and some other websites provide sentiment detection for tweets. A great limitation to evaluate them is they do not make available how their classification was built. One exception is TwitterSentiment (Go et al., 2009), for instance, which considers tweets with good emoticons as positive examples and tweets with bad emoticons as negative examples for the training data, and builds a classifier using unigrams and bigrams as features. We showed in Section 4 that our approach works better than theirs for this problem, obtaining lower error rates.

We have presented an effective and robust sen­timent detection approach for Twitter messages, which uses biased and noisy labels as input to build its models. This performance is due to the fact that: (1) our approach creates a more abstract representation of these messages, instead of using a raw word representation of them as some pre­vious approaches; and (2) although noisy and bi­ased, the data sources provide labels of reasonable quality and, since they have different bias, com­bining them also brought some benefits.

The main limitation of our approach is the cases of sentences that contain antagonistic sentiments. As future work, we want to perform a more fine grained analysis of sentences in order to identify its main focus and then based the sentiment clas­sification on it.