For this evaluation campaign we initially deliv­ered a trial corpus for the potential participants. The trial data consisted of an adapted version of the WePS corpus described in (Artiles et al., 2006). The predominant feature of this corpus is a high number of entities in each document set, due to the fact that the ambiguous names were extracted from the most common names in the US Census. This corpus did not completely match task specifications because it did not consider documents with internal ambiguity, nor it did consider non-person entities; but it was, however, a cost-effective way of releasing data to play around with. During the first weeks after releas­ing this trial data to potential participants, some an­notation mistakes were noticed. We preferred, how­ever, to leave the corpus "as is" and concentrate our efforts in producing clean training and test datasets, rather than investing time in improving trial data.

In order to provide different ambiguity scenarios, we selected person names from different sources:

US Census. We reused the Web03 corpus (Mann, 2006), which contains 32 names randomly picked from the US Census, and was well suited for the task.

Wikipedia. Another seven names were sampled from a list of ambiguous person names in the En­glish Wikipedia. These were expected to have a few predominant entities (popular or historical), and therefore a lower ambiguity than the previous set.

ECDL. Finally, ten additional names were ran­domly selected from the Program Committee listing of a Computer Science conference (ECDL 2006). This set offers a scenario of potentially low am­biguity (computer science scholars usually have a stronger Internet presence than other professional fields) with the added value of the a priori knowl­edge of a domain specific type of entity (scholar) present in the data.

All datasets consist of collections of web pages obtained from the 100 top results for a person name query to an Internet search engine . Note that 100 is an upper bound, because in some occasions the URL returned by the search engine no longer exists.

The second and third datasets (developed explic­itly for our task) consist of 17 person names and 1685 associated documents in total (99 documents per name in average). Each web page was down­loaded and stored for off-line processing. We also stored the basic metadata associated to each search result, including the original URL, title, position in the results ranking and the corresponding snippet generated by the search engine.

In the process of generating the corpus, the se­lection of the names plays an important role, poten­tially conditioning the degree of ambiguity that will be found later in the Web search results. The reasons for this variability in the ambiguity of names are di­verse and do not always correlate with the straight­forward census frequency. A much more decisive feature is, for instance, the presence of famous en­tities sharing the ambiguous name with less popular people. As we are considering top search results, these can easily be monopolized by a single entity that is popular in the Internet.

After the annotation of this data (see section 2.1.4.) we found our predictions about the average ambiguity of each dataset not to be completely ac­curate. In Table 1 we see that the ECDL-06 average ambiguity is indeed relatively low (except for the documents for "Thomas Baker" standing as the most ambiguous name in the whole training). Wikipedia names have an average ambiguity of 23,14 entities per name, which is higher than for the ECDL set. The WEB03 Corpus has the lowest ambiguity (5,9 entities per name), for two reasons: first, randomly picked names belong predominantly to the long tail of unfrequent person names which, per se, have low ambiguity. Being rare names implies that in average there are fewer documents returned by the search en­gine (47,20 per name), which also reduces the pos­sibilities to find ambiguity.

For the test data we followed the same process described for the training. In the name selection we tried to maintain a similar distribution of ambigu­ity degrees and scenario. For that reason we ran­domly extracted 10 person names from the English Wikipedia and another 10 names from participants in the ACL-06 conference. In the case of the US cen­sus names, we decided to focus on relatively com­mon names, to avoid the problems explained above.

Unfortunately, after the annotation was finished (once the submission deadline had expired), we found a major increase in the ambiguity degrees (Ta­ble 2) of all data sets. While we expected a raise in the case of the US census names, the other two cases just show that there is a high (and unpredictable) variability, which would require much larger data sets to have reliable population samples.

This has made the task particularly challenging for participants, because naive learning strategies (such as empirical adjustment of distance thresholds to optimize standard clustering algorithms) might be misleaded by the training set.

The annotation of the data was performed sepa­rately in each set ofdocuments related to an ambigu­ous name. Given this set of approximately 100 doc­uments that mention the ambiguous name, the an­notation consisted in the manual clustering of each document according to the actual entity that is re­ferred on it.

When non person entities were found (for in­stance, organization or places named after a person) the annotation was performed without any special rule. Generally, the annotator browses documents following the original ranking in the search results; after reading a document he will decide whether the mentions of the ambiguous name refer to a new en­tity or to a entity previously identified. We asked the annotators to concentrate first on mentions that strictly contained the search string, and then to pay attention to the co-referent variations of the name. For instance "John Edward Fox" or "Edward Fox Smith" would be valid mentions. "Edward J. Fox", however, breaks the original search string, and we do not get into name variation detection, so it will be considered valid only if it is co-referent to a valid mention.

In order to perform the clustering, the annotator was asked to pay attention to objective facts (bi­ographical dates, related names, occupations, etc.) and to be conservative when making decisions. The final result is a complete clustering of the docu­ments, where each cluster contains the documents that refer to a particular entity. Following the pre­vious example, in documents for the name "Edward Fox" the annotator found 16 different entities with that name. Note that there is no a priori knowledge about the number of entities that will be discovered in a document set. This makes the task specially difficult when there are many different entities and a high volume of scattered biographical information to take into account.

In cases where the document does not offer enough information to decide whether it belongs to a cluster or is a new entity, it is discarded from the evaluation process (not from the dataset). Another common reason for discarding documents was the absence of the person name in the document, usually due to a mismatch between the search engine cache and the downloaded URL.

We found that, in many cases, different entities were mentioned using the ambiguous name within a single document. This was the case when a doc­ument mentions relatives with names that contain the ambiguous string (for instance "Edward Fox" and "Edward Fox Jr."). Another common case of intra-document ambiguity is that of pages contain­ing database search results, such as book lists from Amazon, actors from IMDB, etc. A similar case is that of pages that explicitly analyze the ambiguity of a person name (Wikipedia "disambiguation" pages). The way this situation was handled, in terms of the annotation, was to assign each document to as many clusters as entities were referred to on it with the ambiguous name.