Both word sense disambiguation and named entity recognition have benefited enormously from shared task evaluations, for example in the Senseval, MUC and CoNLL frameworks. Similar campaigns have not been developed for the resolution of figurative language, such as metaphor, metonymy, idioms and irony. However, resolution of figurative language is an important complement to and extension of word sense disambiguation as it often deals with word senses that are not listed in the lexicon. For exam­ple, the meaning of stopover in the sentence He saw teaching as a stopover on his way to bigger things is a metaphorical sense of the sense "stopping place in a physical journey", with the literal sense listed in WordNet 2.0 but the metaphorical one not being listed. The same holds for the metonymic reading of rattlesnake (for the animal's meat) in Roast rat­tlesnake tastes like chicken Again, the meat reading of rattlesnake is not listed in WordNet whereas the meat reading for chicken is.

As there is no common framework or corpus for figurative language resolution, previous computa­tional works (Fass, 1997; Hobbs et al., 1993; Barn-den et al., 2003, among others) carry out only small-scale evaluations. In recent years, there has been growing interest in metaphor and metonymy resolu­tion that is either corpus-based or evaluated on larger datasets (Martin, 1994; Nissim and Markert, 2003; Mason, 2004; Peirsman, 2006; Birke and Sarkaar, 2006; Krishnakamuran and Zhu, 2007). Still, apart from (Nissim and Markert, 2003; Peirsman, 2006) who evaluate their work on the same dataset, results are hardly comparable as they all operate within dif­ferent frameworks.

This situation motivated us to organise the first shared task for figurative language, concentrating on metonymy. In metonymy one expression is used to refer to the referent of a related one, like the use of an animal name for its meat. Similarly, in Ex. 1, Vietnam, the name of a location, refers to an event (a war) that happened there.

(1) Sex, drugs, and Vietnam have haunted Bill Clinton's campaign.

In Ex. 2 and 3, BMW, the name ofa company, stands for its index on the stock market, or a vehicle manu­factured by BMW, respectively.

(3) His BMW went on to race at Le Mans

The importance of resolving metonymies has been shown for a variety of NLP tasks, such as machine translation (Kamei and Wakao, 1992), ques­tion answering (Stallard, 1993), anaphora resolution geographical information retrieval (Leveling and

Hartrumpf, 2006).

Although metonymic readings are, like all figu­rative readings, potentially open ended and can be innovative, the regularity of usage for word groups helps in establishing a common evaluation frame­work. Many other location names, for instance, can be used in the same fashion as Vietnam in Ex. 1. Thus, given a semantic class (e.g. location), one can specify several regular metonymic patterns (e.g. place-for-event) that instances of the class are likely to undergo. In addition to literal readings, regu­lar metonymic patterns and innovative metonymic readings, there can also be so-called mixed read­ings, similar to zeugma, where both a literal and a metonymic reading are evoked (Nunberg, 1995).

The metonymy task is a lexical sample task for English, consisting of two subtasks, one concentrat­ing on the semantic class location, exemplified by country names, and another one concentrating on or­ganisation, exemplified by company names. Partici­pants had to automatically classify preselected coun­try/company names as having a literal or non-literal meaning, given a four-sentence context. Addition­ally, participants could attempt finer-grained inter­pretations, further specifying readings into prespec-ified metonymic patterns (such as place-for-event) and recognising innovative readings.

We distinguish between literal, metonymic, and mixed readings for locations and organisations. In the case of a metonymic reading, we also specify the actual patterns. The annotation categories were motivated by prior linguistic research by ourselves (Markert and Nissim, 2006), and others (Fass, 1997; Lakoff and Johnson, 1980).

We used the CIA Factbook and the Fortune 500 list as sampling frames for country and company names respectively. All occurrences (including plu­ral forms) of all names in the sampling frames were extracted in context from all texts of the BNC, Ver­sion 1.0. All samples extracted are coded in XML and contain up to four sentences: the sentence in which the country/company name occurs, two be­fore, and one after. If the name occurs at the begin­ning or end of a text the samples may contain less than four sentences.

For both the location and the organisation subtask, two random subsets of the extracted samples were selected as training and test set, respectively. Before metonymy annotation, samples that were not under­stood by the annotators because of insufficient con­text were removed from the datsets. In addition, a sample was also removed if the name extracted was a homonym not in the desired semantic class (for ex­ample Mr. Greenland when annotating locations).For those names that do have the semantic class location or organisation, metonymy anno­tation was performed, using the categories described in Section 2. All training set annotation was carried out independently by both organisers. Annotation was highly reliable with a kappa (Carletta, 1996) of

Table 1 : Reading distribution for locations .88/.89 for locations/organisations. As agreement was established, annotation of the test set was car­ried out by the first organiser. All cases which were not entirely straightforward were then independently checked by the second organiser. Samples whose readings could not be agreed on (after a reconcil­iation phase) were excluded from both training and test set. The reading distributions of training and test sets for both subtasks are shown in Tables 1 and 2.

In addition to a simple text format including only the metonymy annotation, we provided participants with several linguistic annotations of both training and testset. This included the original BNC tokeni-sation and part-of-speech tags as well as manually annotated dependency relations for each annotated name (e.g. BMW subj-of-slip for Ex. 2).

Teams were allowed to participate in the location or organisation task or both. We encouraged super­vised, semi-supervised or unsupervised approaches.

Systems could be tailored to recognise metonymies at three different levels of granularity: coarse, medium, or fine, with an increasing number and specification of target classification categories, and thus difficulty. At the coarse level, only a distinction between literal and non-literal was asked for; medium asked for a distinction between literal, metonymic and mixed readings; fine needed a classification into literal readings, mixed readings, any of the class-dependent and class-independent metonymic patterns (Section 2) or an innovative metonymic reading (category othermet).

Systems were evaluated via accuracy (acc) and coverage (cov), allowing for partial submissions.

For each target category c we also measured:

precisioncrecallc = fscorec

# correct assignments of c # assignments of c # correct assignments of c # dataset instances of c 2precisioncrecallcprecisionc-\-recallc

A baseline, consisting of the assignment of the most frequent category (always literal), was used for each task and granularity level.

We received five submissions (FUH, GYDER, up13, UTD-HLT-CG, XRCE-M). All tackled the location task; three (GYDER, UTD-HLT-CG, XRCE-M) also participated in the organisation task. All systems were full submissions (coverage of 1) and participated at all granularity levels.

There is a wide range of opportunities for future fig­urative language resolution tasks. In the SemEval corpus the reading distribution mirrored the actual distribution in the original corpus (BNC). Although realistic, this led to little training data for several phenomena. A future option, geared entirely to­wards system improvement, would be to use a strat­ified corpus, built with different acquisition strate­gies like active learning or specialised search proce­dures. There are also several options for expand­ing the scope of the task, for example to a wider range of semantic classes, from proper names to common nouns, and from lexical samples to an all-words task. In addition, our task currently covers only metonymies and could be extended to other kinds of figurative language.

We are very grateful to the BNC Consortium for let­ting us use and distribute samples from the British National Corpus, version 1.0.