Being able to automatically distinguish between John Doe, the musician, and John Doe, the actor, on the Web is a task of significant importance with ap­plications in IR and other information management tasks. Mann and Yarowsky (2004) used bigograph-ical data annotated with named entitities and per­form fusion of extracted information across multiple documents. Bekkerman and McCallum (2005) stud­ied the problem in a social network setting exploit­ing link topology to disambiguate namesakes. Al-Kamha and Embley (2004) used a combination of attributes (like zipcodes, state, etc.), links, and page similarity to derive the name clusters while Wan et. al. (2005) used lexical features and named entities.

Our framework focuses on the K-means clustering model using both bag of words as features and vari­ous augumented feature sets. We experimented with several similarity functions and chose Pearson's cor­relation coefficient as the distance measure for clus­tering. The weights for the features were set to the term frequency of their respective words in the doc-ument.

The main parameter for K-means clustering is choosing the number of clusters, K. We optimized K over the training data varying K from 10%, 20%,- • ,100% of the number of documents as well as varying absolute K values from 10, 20, • • • to 100 documents. The evaluation score ofF-measure can be highly sensitive to this parameter K, as shown in Table 3. The value of K that gives the best F-measure on training set using vanilla bag of words (BOW) model is K = 10%, however we see in Ta­ble 3 that this value of K actually performs much worse on the test data as compared to other K val­ues.

4 Training/Test discrepancy and re-evaluation using cross validation on test data

Table 4 compares cluster statistics between the train­ing and test data. This data was derived from Artiles et. al (2007). The large difference between aver­age number of clusters in training and test sets in­dicates that the parameter K, optimized on training set cannot be transferred to test set as these two sets belong to a very different distribution. This can be emprically seen in Table 3 where applying the best K on training results in a significant performance drop on test set given this divergence when param­eters are optimized for F0 5 (although performance does transfer well when parameters are optimized on F0 2). This was observed in our primary evaluation system which was optimized for F0 5 and resulted in a low official score of F0 5 = .53 and F0 2 = .65.

Thus an interesting question is to measure per­formance when parameters are chosen on data shar­ing the distributional character of the test data rather than the highly divergent training set. To do this, we used a standard 2-fold cross validation to estimate clustering parameters from a held-out, alternate-half portion of the test data, which more fairly repre­sents the character of the other half of the test data than does the very different training data. We di­vide the test set into two equal halves (taking first fifteen names alphabetically in one set and the rest in another). We optimize K on the first half, test on the other half and vice versa. We report the two K-values and their corresponding F-measures in Ta­ble 5 and we also report the average in order to com­pare it with the results on the test set obtained using K optimized on training. Further, we also report what would be oracle best K, that is, if we optimize K on the entire test data . We can see in Table 5 that how optimizing K on a devlopment set with same distribution as test set can give us F-measure in the range of 77%, a significant increase as com­pared to the F-measure obtained by optimizing K on given training data. Further, Table 5, also indicates results by a custom clustering method, that takes the best K-means clustering using vanilla bag of words model, retains the largest cluster and splits all the other clusters into singleton clusters. This method gives an improved 2-fold F-measure score over the simple bag of words model, implying that most of the namesakes in test data have one (or few) domi­nant cluster and a lot of singleton clusters. Table 6 shows a full enumeration of model variance under this cross validated test evaluation. POS and Rich­Feats yield small gains, and a best F05 performance of.776.

Table 5: Comparision oftraining and test results us­ing Vanilla Bag-of-words model. The values indi­cated with * represent the average value.

We presented a K-means clustering approach for the task of person name disambiguation using several augmented feature sets including HTML meta fea­tures, part-of-speech-filtered features, and inclusion of additional web snippets extracted from Google to facilitate term bridging. The latter showed sig­nificant empirical gains on the training data. Best performance on test data, when parameters are op­timized for F02 on training (Table 3), yielded a top performing F0 2 of .855 on test data (and F0 5=.773 on test data). We also explored the striking discrep­ancy between training and test data characteristics and showed how optimizing the clustering param­eters on given training data does not transfer well to the divergent test data. To control for similar training and test distributional characteristics, we re-evaluated our test results estimating clustering pa­rameters from alternate held-out portions of the test set. Our models achieved cross validated F0.5 of.77-.78 on test data for all feature combinations, further showing the broad strong performance ofthese tech­niques.