Sentiment tagging of short text spans — sentences, headlines, or clauses — poses considerable chal­lenges for automatic systems due to the scarcity of sentiment clues in these units: sometimes, the deci­sion about the text span sentiment has to be based on just a single sentiment clue and the cost of every error is high. This is particularly true for headlines, which are typically very short. Therefore, an ideal system for sentiment tagging of headlines has to use a large set of features with dependable sentiment an­notations and to be able to reliably deduce the senti­ment of the headline from the sentiment of its com­ponents.

The valence labeling subtask ofthe Affective Text task requires ternary — positive vs. negative vs. neutral — classification of headlines. While such categorization at the sentence level remains rela­tively unexplored, the two related sentence-level, binary classification tasks — positive vs. negative and subjective vs. objective — have attracted con­siderable attention in the recent years (Hu and Liu, 2004; Kim and Hovy, 2005; Riloff et al., 2006; Tur-ney and Littman, 2003; Yu and Hatzivassiloglou, 2003). Unsupervised knowledge-based methods are the preferred approach to classification of sentences into positive and negative, mostly due to the lack of adequate amounts of labeled training data (Gamon and Aue, 2005). These approaches rely on presence and scores of sentiment-bearing words that have been acquired from dictionaries (Kim and Hovy, 2005) or corpora (Yu and Hatzivassiloglou, 2003). Their accuracy on news sentences is between 65 and 68%.

Sentence-level subjectivity detection, where train­ing data is easier to obtain than for positive vs. neg­ative classification, has been successfully performed using supervised statistical methods alone (Pang and Lee, 2004) or in combination with a knowledge-based approach (Riloff et al., 2006).

Since the extant literature does not provide clear evidence for the choice between supervised machine learning methods and unsupervised knowledge-based approaches for the task of ternary sentiment classification of sentences or headlines, we devel­oped two systems for the Affective Text task at SemEval-2007. The first system (CLaC) relies on the knowledge-rich approach that takes into consid-

'To our knowledge, the only work that attempted such clas­sification at the sentence level is (Gamon and Aue, 2005) that classified product reviews.

The CLaC system relies on a knowledge-based, domain-independent, unsupervised approach to headline sentiment detection and scoring. The system uses three main knowledge inputs: a list of sentiment-bearing unigrams, a list of valence shifters (Polanyi and Zaenen, 2006), and a set of rules that define the scope and results of com­bination of sentiment-bearing words with valence shifters.

Supervised statistical methods have been very suc­cessful in sentiment tagging of texts and in subjec­tivity detection at sentence level: on movie review texts they reach an accuracy of 85-90% (Aue and accuracy on classifying movie review snippets into subjective and objective using both Nave Bayes and SVM (Pang and Lee, 2004). These methods per­form particularly well when a large volume of la­beled data from the same domain as the test set is available for training (Aue and Gamon, 2005). The lack of sufficient data for training appears to be the main reason for the virtual absence of experiments with statistical classifiers in sentiment tagging at the sentence level.

In order to explore the potential of statistical ap­proaches on sentiment classification of headlines, we implemented a basic Naïve Bayes classifier with smoothing using Lidstone's law of succession (with A=0.1). No feature selection was performed.

The development set for the Affective Text task consisted of only 250 headlines, which is not suf­ficient for training of a statistical classifier. In or­der to increase the size of the training corpus, we augmented it with a balanced set of 900 manually annotated news sentences on a variety of topics ex­tracted from the Canadian NewsStand database and 200 headlines from different domains collected from Google News in January 2007.

The probabilities assigned by the classifier were mapped to [-100, 100] as follows: all negative head­lines received a score of -100, all positive headlines +100, and neutral headlines 0.

Table 1 shows the results of the two CLaC systems for valence labeling subtask of Affective Text task compared to all participating systems average. The best subtask scores are highlighted in bold.

The comparison between the two CLaC systems clearly demonstrates the relative advantages of the two approaches. The knowledge-based unsuper-vised system performed well above average on three main measures: the Pearson correlation between fine-grained sentiment assigned by CLaC system and the human annotation; the accuracy for ternary classification; and the precision of binary (positive vs. negative) classification. These results demon­strate that an accurately annotated list of sentiment-bearing words combined with sophisticated valence shifter handling produces acceptably accurate senti­ment labels even for such difficult data as news head­lines. This system, however, was not able to provide good recall.

On the contrary, supervised machine learning has very good recall, but low accuracy relative to the results of the unsupervised knowledge-based ap­proach. This shortcoming could be in part reduced if more uniformly labeled headlines were available for training. However, we can hardly expect large amounts of such manually annotated data to be handy in real-life situations.

The two CLaC systems that we submitted to the Affective Text task have tested the applicability of two main sentiment tagging approaches to news headlines annotation. The results of the two sys­tems indicate that the knowledge-based unsuper-vised approach that relies on an automatically ac­quired list of sentiment-bearing unigrams and takes into account the combinatorial properties of valence shifters, can produce high quality sentiment annota­tions, but may miss many sentiment-laden headlines. On the other hand, supervised machine learning has good recall even with a relatively small training set, but its precision and accuracy are low. In our future work we will explore the potential of combining the two approaches in a single system in order to im­prove both recall and precision of sentiment annota­tion.