Each of our classifiers uses the same set of context features, taken directly from the data provided by the task organizers. The features we used included:

• Bag-of-words (unigrams)

• Bigrams and trigrams around the target word

• Weighted unigrams surrounding the target word

The weighted unigram features increased the fre­quencies of the ten words before and after the tar­get word by inserting them multiple times into the bag-of-words.

Our system consists of five unique classifiers. Three of the classifiers were selected by our combination system, while the other two were found to be detri­mental to its performance. We describe the con­tributing classifiers first. Table 1 shows the results of each classifier, as well as our classifier combina­tion system.

The classifier combination algorithm that we imple­ment is based on a simple voting system. Each clas­sifier returns a score for each sense: the Naïve Bayes classifier returns a probability, the cosine-based clas­sifiers (including LSA) return a cosine distance, and the decision list classifier returns the weight asso­ciated with the chosen feature (if no feature is se­lected, the frequency of the most frequent sense is used). The scores from each classifier are normal­ized to the range [0,1], multiplied by an empirically determined weight for that classifier, and summed for each sense. The combiner then chooses the sense with the highest score. We used cross validation to determine the weight for each classifier, and it was during that test that we discovered that the best con­stant for the LSA and k-NN classifiers was zero. The most likely explanation for this is that the LSA and k-NN are doing similar, only less accurate, classi­fications as the nearest neighbor classifier, and so have little new knowledge to add to the combiner. We also implemented a simple majority voting sys­tem, where the chosen sense is the sense chosen by the most classifiers, but found it to be less accurate.

To increase the accuracy of our system, we needed to optimize various parameters by running the training data through 10-way cross-validation and averaging the scores from each set. Table 2 shows the results of this cross-validation in determining the 6 value used in the additive smoothing for both the Naïve Bayes classifier and for the decision list classifier.

We also experimented with different feature sets. The results of these experiments are shown in Ta­ble 3.

Table 1: The (micro-averaged) precision of each of our classifiers in cross-validation, plus the actual re­sults from our entry in SemEval-2007.

Table 2: On cross-validated training data, system precision when using different smoothing parame­ters in the Naïve Bayes and decision list classifiers.

We presented a supervised system that used simple n-gram features and a combination of five different classifiers. The methods used are applicable to any lexical sample task, and have been applied to lexical sample tasks in previous Senseval competitions.

The goal of Task 14: Affective Text is to take a list of headlines and meaningfully annotate their emotional content. Each headline was scored along seven axes: six predefined emotions (Anger, Disgust, Fear, Joy, Sadness, and Surprise) on a scale from 0 to 100, and the negative/positive polarity (valence) of the head­line on a scale from —100 to +100.

Our system is trained on a set of pre-annotated head­lines, building up a knowledge-base of individual words and their emotional significance.

We were initially provided with a trial-set of 250 annotated headlines. We ran 5-way cross-validation with a preliminary version of our system, and found that a dataset of that size was too sparse to effec­tively tag new headlines. In order to generate a more meaningful knowledge-base, we created a sim­ple web interface for human annotation of headlines. We used untrained, non-experts to annotate an addi­tional 1,000 headlines for use as a training set. The headlines were taken from a randomized collection of headlines from the Associated Press.

We included a subset of the original test set in the set that we put online so that we could get a rough estimate of the consistency of human annota­tion. We found that consistency varied greatly across the emotions. As can be seen in Table 4, our annota-tors were very consistent with the trial data annota-tors on some emotions, while inconsistent on others.

In ad-hoc, post-annotation interviews, our anno-tators commented that the task was very difficult. What we had initially expected to be a tedious but mindless exercise turned out to be rather involved. They also reported that some emotions were consis­tently harder to annotate than others. The results in Table 4 seem to bear this out as well.

One difficulty reported by our annotators was de­termining whether to label the emotion experienced by the reader or by the subject of the headline. For example, the headline "White House surprised at reaction to attorney firings" clearly states that the White House was surprised, but the reader might not have been.

Another of the major difficulties in properly an­notating headlines is that many headlines can be an­notated in vastly different ways depending on the viewpoint of the annotator. For example, while the headline "Hundreds killed in earthquake" would be universally accepted as negative, the headline "Italy defeats France in World Cup Final," can be seen as positive, negative, or even neutral depending on the viewpoint of the reader. These types of problems made it very difficult for our annotators to provide consistent labels.

Before we can process a headline and determine its emotions and valence, we convert our list of tagged headlines into a useful knowledge base. To this end, we create a word-emotion mapping.

Table 5: A comparison of results on the provided trial data as headlines are added to the training set. The scores are given as Pearson correlations of scores for training sets of size 100, 250, and 1000 headlines.

As can be seen in Table 5, four out ofsix emotions and the valence increase along with training set size.

This leads us to believe that further increases in the size of the training set would continue to improve results. Lack of time prevents a full analysis that can explain the sudden drop of Disgust and Joy.

Table 6 shows our full results from this task. Our system finished third out of five in the valence sub-task and second out of three in the emotion sub-task.

Table 6: Our full results from SemEval-2007, Task 14, as reported by the task organizers. Fine-grained scores are given as Pearson correlations. Coarse­grained scores are given as accuracy (A), preci­sion (P), and recall (R).

We presented a supervised system that used a un­igram model to annotate the emotional content of headlines. We also used synonym expansion on the emotion label words. Our annotators encountered significant difficulty while tagging training data, due to ambiguity in definition of the task.