The Multilevel Semantic Annotation of Catalan and Spanish task is split into the following three sub-tasks:

Noun Sense Disambiguation (NSD): Disambigua­tion of all frequent nouns ("all words" style).

Named Entity Recognition (NER): The annotation of (possibly embedding) named entities with basic entity types.

Semantic Role Labeling (SRL): Including also two subtasks, i.e., the annotation of verbal predicates with semantic roles (SR), and verb tagging with semantic-class labels (SC).

All semantic annotation tasks are performed on exactly the same corpora for each language. We pre­sented all the annotation levels together as a com­plex global task, since we were interested in ap­proaches which address these problems jointly, pos­sibly taking into account cross-dependencies among them. However, we were also accepting systems ap­proaching the annotation in a pipeline style, or addressing any of the particular subtasks in any of the languages.

In Section 2 we describe the methodology fol­lowed to develop the linguistic corpora for the task. Sections 3 and 4 summarize the task setting and the participant systems, respectively. Finally, Section 5 presents a comparative analysis of the results. For any additional information on corpora, resources, formats, tagsets, annotation manuals, etc. we refer the reader to the official website of the task.

The corpora used in this SemEval task are a subset of CESS-ECE, a multilingual Treebank, composed of a Spanish (CESS-ESP) and a Catalan (CESS-CAT) corpus of 500K words each (Marti et al., 2007b). These corpora were enriched with different kinds of semantic information: argument structure, thematic roles, semantic class, named entities, and WordNet synsets for the 150 most frequent nouns. The an­notation process was carried out in a semiautomatic way, with a posterior manual revision of all auto­matic processes.

A sequential approach was adopted for the anno­tation of the corpus, beginning with the basic lev­els of analysis, i.e., POS tagging and chunking (au­tomatically performed) and followed by the more complex levels: syntactic constituents and functions (manually tagged) and semantic annotation (man­ual and semiautomatic processes with manual com­pletion and posterior revision). Furthermore, some experiments concerning inter-annotator agreement were carried out at the syntactic (Civit et al., 2003) and semantic levels (Marquez et al., 2004) in order to evaluate the quality of the results.

Data formats are similar to those of CoNLL-2004/2005 shared tasks on SRL (column style pre­sentation of levels of annotation), in order to be able to share evaluation tools and already developed scripts for format conversion.

In Figure 1 you can find an example of a fully an­notated sentence in the column-based format. There is one line for each token, and a blank line after the last token of each sentence. The columns, separated by blank spaces, represent different annotations of the sentence with a tagging along words. For struc­tured annotations (parse trees, named entities, and arguments), we use the Start-End format. Columns 1-6 correspond to the input information; columns 7 and above contain the information to be predicted. We can group annotations in five main categories:

BASICJNPUTJNFO (columns 1-3). The basic input information, including: (a) WORD (column 1) words of the sentence; (b) TN (column 2) target nouns of the sentence, marked with '*' (those that are to be assigned WordNet synsets); (c) TV (column 3) target verbs of the sentence, marked with '*' (those that are to be annotated with semantic roles).

EXTRAJNPUTJNFO (columns 4-6). The extra input information, including: (a) LEMMA (column 4) lem­mas of the words; (b) POS (column 5) part-of-speech tags; (c) SYNTAX (column 6) Full syntactic tree. NE (column 7). Named Entities. NS (column 8). WordNet sense of target nouns.

SR (columns 9 and above). Information on semantic roles, including: (a) SC (column 9). Semantic class of the verb; (b) PROPS (columns 10 and above). For each target verb, a column representing the argu­ment structure. Core numbered arguments include

the thematic role labels. ArgM's are the adjuncts. Columns are ordered according to the textual order of the predicates.

All these annotations in column format are ex­tracted automatically from the syntactic-semantic trees from the CESS-ECE corpora, which were dis­tributed with the datasets. Participants were also provided with the whole Catalan and Spanish Word­Nets (v1.6), the verbal lexicons used in the role la­beling annotation, the annotation guidelines as well as the annotated corpora.

About a dozen teams expressed their interest in the task. From those, only 5 registered and downloaded datasets, and finally, only two teams met the dead­line and submitted results. ILK 2 (Tilburg Univer­sity) presented a system addressing Semantic Role Labeling, and UPC* (Technical University of Cat­alonia) presented a system addressing all subtasks independently. The ILK2 SRL system is based on memory-based classification of syntactic con­stituents using a rich feature set. UPC* used several machine learning algorithms for addressing the dif­ferent subtasks (AdaBoost, SVM, Perceptron). For SRL, the system implements a re-ranking strategy using global features. The candidates are generated using a state-of-the-art SRL base system.

Although the task targeted at systems addressing all subtasks jointly none of the participants did it.We believe that the high complexity of the whole task together with the short period of time avail­able were the main reasons for this failure. From this point of view, the conclusions are somehow dis­appointing. However, we think that we have con­tributed with a very valuable resource for the future research and, although not complete, the current sys­tems provide also valuable insights about the task and are very good baselines for the systems to come.

In the following subsections we present an analysis of the results obtained by participant systems in the three subtasks. Results on the test set are presented along 2 dimensions: (a) language ('ca'=Catalan; 'es'=Spanish); (b) corpus source ('in'=in-domain corpus; 'out'=out-of-domain corpus). We will use a language.source pair to denote a particular test set. Finally, '*' will denote the addition of the two sub-corpora, either in the language or source dimensions.

Results on the NSD subtask are presented in Table 1. BSL stands for a baseline system consisting of as­signing to each word occurrence the most frequent sense in the training set. For new nouns the first sense in the corresponding WordNet is selected. The upc* team trained a SVM classifier for each word in a pre-selected subset and applied the baseline in the rest of cases. The selected words are frequent words (more than 15 occurrences in the training corpus) showing a not too skewed distribution of senses in the training set (the most predominant sense covers less than 90% of the cases). No other teams pre­sented results for this task.

The left part of the table ("all words") contains results on the complete test sets, while the right part ("selected words") contains the results restricted to the set of words with trained SVM classifiers. This set covers 31.0% of the word occurrences in the training set and 28.2% in the complete test set.

The main observation is that training/test corpora contain few sense variations. Sense distributions are very skewed and, thus, the simple baseline shows a very high accuracy (almost 85%). The UPC* system only improves BSL accuracy by one point. This can be partly explained by the small size of the word-based training corpora. Also, this improvement is diminished because UPC* only treated a subset of words. However, looking at the right-hand side of the table, the improvement over the baseline is still modest (~3 points) when focusing only on the treated words. As a final observation, no significant differences are observed across languages and cor­pora sources.

Results on the NER subtask are presented in Table 2. This time, BSL stands for a baseline system consist­ing of collecting a gazetteer with the strong NEs ap­pearing in the training set and assigning the longest matches of these NEs in the test set. Weak entities are simply ignored by BSL. UPC* presented a system which treats strong and weak NEs in a pipeline of two processors. Classifiers trained with multiclass

AdaBoost are used to predict the strong and weak NEs. See authors' paper for details.

UPC* system largely overcomes the baseline, mainly due to the low recall of the latter. By lan­guages, results on Catalan are significantly better than those on Spanish. We think this is attributable mainly to corpora variations across languages. By corpus source, "in-domain" results are slightly bet­ter, but the difference is small (1.78 points). Overall, the results for the NER task are in the mid seventies, a remarkable result given the small training set and the complexity of predicting embedded NEs.

Detailed results on concrete entity types are pre­sented in Table 3 (sorted by decreasing Fi). As ex­pected, DAT and NUM are the easiest entities to rec­ognize since they can be easily detected by simple patterns and POS tags. On the contrary, entity types requiring more semantic information present fairly lower results. org per and loc are in the sev­enties, while oth is by far the most difficult class, showing a very low recall. This is not surprising since oth agglutinates a wide variety of entity cases which are difficult to characterize as a whole.

Another interesting analysis is to study the differ­ences between strong and weak entities (see Table 4) . Contrary to our first expectations, results on weak entities are much better (up to 11 Fi points higher). Weak NEs are simpler for two reasons: (a) there exist simple patters to characterize them, without the need of fully recognizing their internal strong NEs; (b) there is some redundancy in the corpus when tagging many equivalent weak NEs in embed­ded noun phrases. It is worth noting that the low re­sults for strong NEs come from classification rather than recognition (recognition is almost 100% given the "proper noun" PoS tag), thus the recall for weak entities is not diminished by the errors in strong en­tity classification.

SRL is the most complex and interesting problem in the task. We had two participants ILK2 and UPC*, which participated in both subproblems, i.e., label­ing arguments of verbal predicates with thematic roles (SR), and assigning semantic class labels to target verbs (SC). Detailed results of the two sys­tems are presented in Tables 5 and 6.

The ILK2 system outperforms UPC* in both SR and SC. For SR, both systems use a traditional ar­chitecture of labeling syntactic tree nodes with the­matic roles using supervised classifiers. We would attribute the overall Fi difference (2.68 points) to a better feature engineering by ILK2, rather than to differences in the Machine Learning techniques used. Overall results in the eighties are remarkably high given the training set size and the granularity of the thematic roles (though we have to take into account that systems work with gold parse trees). Again, the results are comparable across languages and slightly better in the "in-domain" test set.