We use the term entity to mean a specific person or object. A mention is a reference to an entity such as a word or phrase in a document. Taken to­gether, all mentions that refer to the same real-world object model that entity (Mitchell et al. 2004). Entity disambiguation inherently involves resolving many-to-many relationships. Multiple distinct strings may refer to the same entity. Si­multaneously, multiple identical mentions refer to distinct entities (Bagga and Baldwin, 1998).

Fair Isaac's entity disambiguation software is based largely on language-independent algorithms that resolve mentions in the context of the entire corpus. The system utilizes multiple types of con­text as evidence for determining whether two men­tions correspond to the same entity and it auto­matically learns the weight of evidence of each context item via corpus statistics.

The goal of the Web People Search task (Artiles et al. 2007) is to assign Web pages to groups, where each group contains all (and only those) pages that refer to one unique entity. A page is assigned to multiple groups if it mentions multiple entities, for example "John F. Kennedy" and the "John F. Kennedy Library". The pages were se­lected via a set of keyword queries, and the disam­biguation is evaluated only on those query entities. This differs from Fair Isaac's system in a few key ways: our system deals with mentions rather than documents, our system does not require a filter on mentions, and our system is generally used for large collections of documents containing very many names rather than small sets of highly am­biguous documents dealing with one specific name. Nevertheless, it was possible to run the Fair Isaac entity disambiguation system on the Web People Search task data with almost no modifica­tions and achieve accurate results.

The remaining sections of this paper describe Fair Isaac's automatic entity disambiguation meth­odology and report on the performance of the sys­tem on the WePS data.

In unstructured text, each document provides a natural context for entity disambiguation. After cleaning up extraneous markup we carry out within-document co-reference resolution, aggregat­ing information about each entity mentioned in each document. We then use these entity attributes as features in determining which documents deal with the same entity.

We have tested our entity disambiguation system on several semi-structured and unstructured text data sets. Here, we report the performance on the training data provided for the Web People Search task. This corpus consists of raw Web pages with substantial variation in capitalization, punctuation, grammar, and spelling - characteristics that make NER challenging. A few other issues also nega­tively impact our performance, including extrane­ous text, long lists of entities, and the issue of find­ing the correct document to parse.

The NER process identified a ratio of approxi­mately 220 mentions per document across 3,359 documents. Within-document entity disambigua­tion reduced this to approximately 113 entities per document, which we refer to as document-level entities. Of these, 3,383 Persons (including those Organizations and Locations which were relabeled as Persons) contained a query name. Cross-document entity disambiguation reduced this to 976 distinct persons with 721 distinct standardized names. Thus, 2,407 merge operations were per­formed in this step. On average, there are 48 men­tions per query name. Our system found an aver­age of 14 unique entities per query name. In the gold standard, the average is 9 unique entities per query name.

Looking at the names that matched in the out­put, it is clear that NER is very important to the process. Post submission of our initial run, we used proper tokenization of punctuation and an additional NER system, which corrected many mistakes in the grouping of names. Also, many of the names that were incorrectly merged would not have been compared if not for the introduction of the additional key that compares all mentions that match a query name.

For the WePS evaluation submission, we con­verted our results to document-level entities by mapping each mention to the document that it was part of and removing duplicates. If we did not find a mention in a document, we labeled the document as a singleton entity.

We also used a number of standard metrics for our internal evaluation. Most of these operate on document-level entities rather than on documents. To convert the ground truth provided for the task to a form usable for these metrics, we assume that each entity contains all mentions in the corresponding document group. These metrics test the cross-document disambiguation rather than the NER and within-document disambiguation. These metrics should not be used to compare between different versions of NER and within-document disambiguation, since the ground truth used in the evaluation is generated by these processes.

In Table 1, we compare a run with the additional newswire data and the comparison key (our WePS submission), leaving out the additional newswire data and the additional comparison key, and leav­ing out only the additional comparison key.

In Table 2, we compare runs based on the im­proved NER (available only after the WePS sub­mission deadline). The first uses the same parame­ters as our submission, the second uses an in­creased threshold, and the third utilizes the word vector-based clustering (document topics)._ WithExtraKey 0.670 0.545~ 0.906 0.818 NoAddedData 0.743 0.752 0.584 0.841 NoExtraKey |q.770 |q.767 |q.624 |q.861 Table 1. Results of pairwise comparisons and clus-terwise harmonic mean of purity and inverse purity on various disambiguation runs. Each metric is averaged across the individual results for every query name.

Acc. Prec. Recall Harm. Purity WithExtraKey 0.690 0.618~0.552 0.815 1.25 Thresh 0.720 0.733 0.500 0.812 Topic Info |0.719 |o.645 |o.545 |o.818 Table 2. Results based on improved named entity recognition. These should not be directly com­pared against those in Table 1, since the different NER yields different ground truth for these evalua­tion metrics.

Most of our metrics are based on pairwise com­parisons - all document-level entities are compared against all other document-level entities that match the same query name, noting whether the pair was coreferent in the results and in the ground truth. With such comparison, we obtain measures includ­ing precision, recall, and accuracy. In this training data, depending on which NER is used, 35,000­50,000 pairwise comparisons are possible.

We also define a clusterwise measure of the harmonic mean between purity and inverse purity with respect to mentions. This is different from the metric provided by WePS, purity and inverse purity at the document level. Since some documents contain multiple entities, the latter metric does not perform correctly. Mentions, on the other hand, are always unique in our disambiguation. How­ever, because the ground truth was specified at the document level, documents containing multiple entities that match a query yield ambiguous men­tions. These decrease all purity-related scores equally and do not vary between runs.

The addition of the newswire data improved re­sults. Inclusion of an extra comparison based on query name matches allowed for comparison of entities with names that do not match the format of person names, and only slightly reduced overall performance. The new NER run can only be com­pared on the last three runs. to the system per­forms better with topic context than without it.

In comparison, in the 2005 Knowledge Discov­ery and Dissemination (KD-D) Challenge Task ER-1a (the main entity disambiguation task), we achieved an accuracy of 94.5%. The margin of error in the evaluation was estimated at 3% due to errors in the "ground truth". This was a pure dis­ambiguation task with no NER or name standardi­zation required. The evaluation set contained 100 names, 9027 documents, and 583,152 pair-wise assertions.

Although the primary purposes of Fair Isaac's en­tity disambiguation system differ from the goal of the Web People Search task, we found that with little modification it was possible to fairly accu­rately cluster Web pages with a given query name according to the real-world entities mentioned on the page. Most of the errors that we encountered are related to information extraction from unstruc­tured data as opposed to the cross-document entity disambiguation itself.

Acknowledgment

This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR0011-06-C-0023.

Any opinions, findings and conclusions or rec­ommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Defense Advanced Research Projects

Agency (DARPA).