The English CLEF data from years 2000-2005 com­prises corpora from 'Los Angeles Times' (year 1994) and 'Glasgow Herald' (year 1995) amounting to 169,477 documents (579 MB of raw text, 4.8GB in the XML format provided to participants, see Sec­tion 2.3) and 300 topics in English and Spanish (the topics are human translations of each other). The relevance judgments were taken from CLEF. This might have the disadvantage of having been pro­duced by pooling the results of CLEF participants, and might bias the results towards systems not using WSD, specially for monolingual English retrieval. We are considering the realization of a post-hoc analysis of the participants results in order to ana­lyze the effect on the lack of pooling.

Due to the size of the document collection, we de­cided that the limited time available in the competi­tion was too short to disambiguate the whole collec­tion. We thus chose to take a sixth part of the corpus at random, comprising 29,375 documents (874MB in the XML format distributed to participants). Not all topics had relevant documents in this 17% sam­ple, and therefore only 201 topics were effectively used for evaluation. All in all, we reused 21,797 relevance judgements that contained one of the doc­uments in the 17% sample, from which 923 are pos-itive. For the future we would like to use the whole collection.

For expansion and translation we used the publicly available Multilingual Central Repository (MCR) from the MEANING project (Atserias et al., 2004). The MCR follows the EuroWordNet design, and currently includes English, Spanish, Italian, Basque and Catalan wordnets tightly connected through the Interlingual Index (based on WordNet 1.6, but linked to all other WordNet versions).

We only expanded (translated) the senses returned by the WSD systems. That is, given a word like 'car', it will be expanded to 'automobile' or 'railcar' (and translated to 'auto' or 'vagon' respectively) de­pending on the sense in WN 1.6. If the systems re­turns more than one sense, we choose the sense with maximum weight. In case of ties, we expand (trans­late) all. The participants could thus implicitly affect the expansion results, for instance, when no sense could be selected for a target noun, the participants could either return nothing (or NOSENSE, which would be equivalent), or all senses with 0 score. In the first case no expansion would be performed, in the second all senses would be expanded, which is equivalent to full expansion. This fact will be men­tioned again in Section 3.5.

Note that in all cases we never delete any of the words in the original text.

In addition to the expansion strategy used with the participants, we tested other expansion strategies as baselines:

noexp no expansion, original text fullexp expansion (translation in the case of English to Spanish expansion) to all synonyms of all

wsd50 expansion to the best 50% senses as returned by the WSD system. This expansion was tried over the in-house WSD system of the organizer only.

The retrieval engine is an adaptation of the Twenty-One search system (Hiemstra and Kraaij, 1998) that was developed during the 90's by the TNO research institute at Delft (The Netherlands) getting good re­sults on IR and CLIR exercises in TREC (Harman, 2005). It is now further developed by Irion technolo­gies as a cross-lingual retrieval system (Vossen et al., ). For indexing, the TwentyOne system takes Noun Phrases as an input. Noun Phases (NPs) are detected using a chunker and a word form with POS lexicon. Phrases outside the NPs are not indexed, as well as non-content words (determiners, prepositions, etc.) within the phrase.

The Irion TwentyOne system uses a two-stage re­trieval process where relevant documents are first extracted using a vector space matching and sec­ondly phrases are matched with specific queries. Likewise, the system is optimized for high-precision phrase retrieval with short queries (1 up 5 words with a phrasal structure as well). The system can be stripped down to a basic vector space retrieval sys­tem with an tf.idf metrics that returns documents for topics up to a length of30 words. The stripped-down version was used for this task to make the retrieval results compatible with the TREC/CLEF system.

The Irion system was also used for pre­processing. The CLEF corpus and topics were con­verted to the TwentyOne XML format, normalized, and named-entities and phrasal structured detected. Each of the target tokens was identified by an unique identifier.

The participants were provided with the following:

This section present the results obtained by the par­ticipants and baselines in the two IR settings. The

Table 1: Retrieval results given as MAP. IRtops stands for English IR with topic expansion. IR-docs stands for English IR with document expan­sion. CLIR stands for CLIR results for translated documents.

second and third columns of Table 1 present the re­sults when disambiguating the topics and the docu­ments respectively. Non ofthe expansion techniques improves over the baseline (no expansion).

Note that due to the limitation of the search en­gine, long queries were truncated at 50 words, which might explain the very low results of the full expan­sion.

The last column of Table 1 shows the CLIR results when expanding (translating) the disambiguated documents. None of the WSD systems attains the performance of full expansion, which would be the baseline CLIR system, but the WSD ofthe organizer gets close.

In addition to the IR and CLIR results we also pro­vide the WSD performance of the participants on the Senseval 2 and 3 all-words task. The documents from those tasks were included alongside the CLEF documents, in the same formats, so they are treated as any other document. In order to evaluate, we had to map automatically all WSD results to the respec­tive WordNet version (using the mappings in (Daude et al., 2000) which are publicly available).

The results are presented in Table 2, where we can see that the best results are attained by the organizers WSD system.

First of all, we would like to mention that the WSD and expansion strategy, which is very simplistic, de­grades the IR performance. This was rather expected, as the IR experiments had an illustration goal, and are used for comparison with the CLIR experiments. In monolingual IR, expanding the top­ics is much less harmful than expanding the docu­ments. Unfortunately the limitation to 50 words in the queries might have limited the expansion of the topics, which make the results rather unreliable. We plan to fix this for future evaluations.

Regarding CLIR results, even if none of the WSD systems were able to beat the full-expansion base­line, the organizers system was very close, which is quite encouraging due to the very simplistic expan­sion, indexing and retrieval strategies used.

In order to better interpret the results, Table 3 shows the amount of words after the expansion in each case. This data is very important in order to un­derstand the behavior of each of the systems. Note that UNIBA returns 3 synsets at most, and therefore the wsd50 strategy (select the 50% senses with best score) leaves a single synset, which is the same as taking the single best system (wsdbest). Regarding PUTOP, this system returned a single synset, and therefore the wsd50 figures are the same as the ws-dbest figures.

Comparing the amount of words for the two par­ticipant systems, we see that UNIBA has the least words, closely followed by PUTOP. The organizers WSD system gets far more expanded words. The explanation is that when the synsets returned by a WSD system all have 0 weights, the wsdbest expan­sion strategy expands them all. This was not explicit in the rules for participation, and might have affected the results.

A cross analysis of the result tables and the num­ber of words is interesting. For instance, in the IR exercise, when we expand documents, the results in

Table 3: Number of words in the document col­lection after expansion for the WSD system and all baselines. wsdbest stands for the expansion strategy used with participants.

the third column of Table 1 show that the ranking for the non-informed baselines is the following: best for no expansion, second for random WSD, and third for full expansion. These results can be explained because of the amount of expansion: the more ex­pansion the worst results. When more informed WSD is performed, documents with more expansion can get better results, and in fact the WSD system of the organizers is the second best result from all sys­tem and baselines, and has more words than the rest (with exception of wsd50 and full expansion). Still, the no expansion baseline is far from the WSD re­sults.

Regarding the CLIR result, the situation is in­verted, with the best results for the most productive expansions (full expansion, random WSD and no ex­pansion, in this order). For the more informed WSD methods, the best results are again for the organizers WSD system, which is very close to the full expan­sion baseline. Even if wsd50 has more expanded words wsdbest is more effective. Note the very high results attained by random. These high results can be explained by the fact that many senses get the same translation, and thus for many words with few translation, the random translation might be valid. Still the wsdbest, 1st sense and wsd50 results get better results.