This a report of an experiment proposed as the "Af­fective Text" task of the 4th international Work­shop on Semantic Evaluation (SemEval) to deter­mine whether news headlines are loaded with pre­eminently positive or negative emotion or valence. An example of a test headline can be:

Discovered boys bring shock, joy

Our approach takes as its starting point the obser­vation that lexical resources always are noisy, out of date, and most often suffer simultaneously from being both too specific and too general. For our ex­periments, our only lexical resource consists ofa list of eight positive words and eight negative words, as shown below in Table 1. We use a medium-sized corpus of general newsprint to build a general word space, and use our minimal lexical resource to orient ourselves in it.

A word space is a high-dimensional vector space built from distributional statistics (Schütze, 1993; Sahlgren, 2006), in which each word in the vocab­ulary is represented as a context vector v of occur­rence frequencies: v\ = [fj7 •••, /„] where / is the frequency ofwordi in some context j.

The point of this representation is that seman­tic similarity between words can be computed us­ing vector similarity measures. Thus, the similar­ity in meaning between the words and can be quantified by computing the similarity between their respective context vectors: sim sim .

The semantics of such a word space are deter­mined by the data from which the occurrence in­formation has been collected. Since the data set in the SemEval Affective Text task consists of news headlines, a relevant word space should be pro­duced from topically and stylistically similar texts, such as newswire documents. For this reason, we trained our model on a corpus of English-language newsprint which is available for experimentation for participants in the Cross Language Evaluation Forum (CLEF). The corpus consists of some 100 000 newswire documents from Los Angeles Times for the year 1994. We presume any similarly sized collection of newsprint would produce similar re­sults. We lemmatized the data using tools from Connexor, and removed stop words, leaving some 28 million words with a vocabulary of approxi­mately 300 000 words. Since the data for the af­fective task only consisted of news headlines, we treated each headline in the LA times corpus as a separate document, thus doubling the number of documents in the data.

For harvesting occurrence information, we used documents as contexts and standard tfidf-weighting of frequencies, resulting in a 220 220-dimensional word space. No dimensionality reduction was used.

In order to construct valence vectors, we used a set of manually selected seed words (8 positive and 8 negative words), shown in Table 1. These words were chosen (subjectively) to represent typical ex­pression of positive or negative attitude in news texts. The size of the seed set was determined by a number of initial experiments on the development data, where we varied the size of the seed sets from these 8 words to some 700 words in each set (us­ing the WordNet Affect hierarchy (Strapparava and

Valitutti, 2004)).

As comparison, Turney and Littman (2003) used seed sets consisting of 7 words in their word valence annotation experiments, while Turney (2002) used minimal seed sets consisting of only one positive and one negative word ("excellent" and "poor") in his experiments on review classification. Such min­imal seed sets of antonym pairs are not possible to use in the present experiment because they are often nearest neighbors to each other in the word space. Also, it is difficult to find such clear paradigm words for the newswire domain.

The seed words were used to postulate one posi­tive and one negative point (i.e. vector) in the word space by simply taking the centroid of the seed word points: vs = J2vw<as where S is one of the seed sets, and is a word in this set.

Our hypothesis is that words carrying most of the valence in news headlines in the experimental test set are syntagmatically rather than paradigmatically related to the kind of very general words used in our seed set. As an example, consider test headline 501:

Two Hussein allies are hanged, Iraqi official says.

It seems reasonable to believe that this headline should be annotated with a negative valence, and that the desicive word in this case is "hanged." Ob­viously, "hanged" has no paradigmatic neighbors (e.g. synonyms, antonyms or other 'nyms) among the seed words. However, it is likely that "hanged" will co-occur with (and therefore have a syntagmatic relation to) general negative terms such as "danger­ous" and maybe "war." In fact, in this example head­line, the most negatively associated words are prob­ably "Hussein" and "Iraqi," which often co-occur with general negative terms such as "war" and "dan­gerous" in newswire text.

To produce a word space that contains predomi-nantely syntagmatic relations, we built the distribu­tional relations using entire documents as contexts (i.e. each dimension in the word space corresponds to a document in the data). If we would have used words as contexts instead, we would have ended up with a paradigmatic word space.

The relations between words in headlines were mod­eled using the most simple operation conceivable: we simply add all words' context vectors to a com­pound headline vector and use that as the represen­tation of the headline: where is a test headline, and is a word in this headline.

This is obviously a daring, if not foolhardy, ap­proach to modelling syntactic structure, composi­tional semantics, and all types of intra-sentential se­mantic dependencies. It can fairly be expected to be improved upon through an appropriate finer-grained analysis of word presence, adjacency and syntactic relationships. However, this approach is similar to that taken by most search engines in use today, is a useful first baseline, and as can be seen from our results below, does deliver acceptable results.

To perform the valence annotation, we first lem-matized the headlines and removed stop words and words with frequency above 10 000 in the LA times corpus. For each headline, we then summed — as discussed above — the context vectors of the remaining words, thus producing a 220 220-dimensional vector for each headline. This vector was then compared to each ofthe postulated valence vectors by computing the cosine of the angles be­tween the vectors.

We thus have for each headline two cosines, one between the headline and the positive vector and one between the headline and the negative vector. The valence vector with highest cosine score (and thus the smallest spatial angle) was chosen to annotate the headline. For the negative valence vector we as­signed a negative valence value, and for the positive vector a positive value. In 11 cases, a value of was ascribed, either because all headline words were removed by frequency and stop word filtering, or be­cause none of the remaining words occurred in our newsprint corpus.

Our method thus only delivers a binary valence decision — either positive or negative valence. Granted, we could have assigned a neutral valence to very low cosine scores, but as any threshold for deciding on a neutral score would be completely ar­bitrary, we decided to only give fully positive or negative scores to the test headlines. Also, since our aim was to provide a high-recall result, we did not wish to leave any headline with an equivocal score. We scaled the scores to fit the requirements of the coarse-grained evaluation: for each headline with a non-zero score, we multiplied the value with and boosted each value with 50. By this scaling op­eration we guaranteed a positive or a negative score for each headline (apart from the 11 exceptions, in effect unanalyzed by our algorithm, as mentioned above).

The results from the fine-grained and coarse-grained evaluations are shown in Table 2. They show, much as we anticipated, that the coarse-grained evaluation was appropriate for our purposes.

8.1 Correlation coefficients, normality assumptions, and validity of results

The fine-grained evaluation as given by the organ­isers and as shown in Table 2 was computed using Pearson's product-moment coefficient. Pearson's correlation coefficient is a parametric statistic and assumes normal distribution of the data it is testing for correlation. While we have no idea of neither the other contributions' score distribution, nor that of the given test set, we certainly do know that our data are not normally distributed. We would much prefer to evaluate our results using a non-parametric correlation test, such as Spearman's , and suggest that the all results would be rescored using some non-parametric method instead — this would reduce the risk of inadvertent false positives stemming from divergence from the normal distribution rather than divergence from the test set.

Our approach to valence annotation is similar to the second method described by Turney and Littman (2003). In short, their method uses sin­gular value decomposition to produce a reduced-dimensional word space, in which word valence is computed by subtracting the cosine between the word and a set of negative seed words from the co­sine between the word and a set of positive seed words.

The difference between our approach and theirs is that our approach does not require any computation­ally expensive matrix decomposition, as we do not see any reason to restructure our word space. Turney and Littman (2003) hypothesize that singular value decomposition is beneficial for the results in valency annotation because it infers paradigmatic relations between words in the reduced space. However, as we argued in Section 5, we believe that the headline valency annotation task calls for syntagmatic rather than paradigmatic relations. Furthermore, we fail to see the motivation for using singular value decom­position, since if paradigmatic relations are what is needed, then why not simply use words as dimen­sions of the word space?

Our results show that a resource-poor but data-rich method can deliver sensible results. This is in keep­ing with our overall approach, which aims for as lit­tle pre-computed resources as possible.

At almost every juncture in our processing we made risky and simplistic assumptions — using sim­ple frequencies of word occurrence as a semantic model; using a small seed set of positive and nega­tive terms as a target; postulating one semantic locus each for positive and negative emotion; modelling syntactic and semantic relations between terms by vector addition — and yet we find that the seman­tic structure of distributional statistics yields a signal good enough for distinguishing positive from nega­tive headlines with a non-random accuracy. Despite its simplicity, out method produces very good recall (60.17) in the coarse-grained evaluation (the median recall for all systems is 29.59). This speaks to the power of distributional semantics and gives promise of improvement if some of the choice points during the process are returned to: some decisions can well benefit from being made on principled and informed grounds rather than searching under the street lamp, as it were.