Keyphrases are phrases that represent the impor­tant topics of a document. There are two types of keyphrases associated with scholarly publications: author-assigned ones and reader-assigned ones. In the Keyphrase Extraction Task (Kim et al., 2010), each system receives two set of scientific papers from the ACM digital library; a training set and a testing set. The author-assigned keyphrases and reader-assigned keyphrases are given for each pa­per in the training set. The objective is to produce the keyphrases for each article in the testing set.

This paper is organized as followed. First, We describe our keyphrase extraction system, SEER­LAB. We then discuss its performance in SemEval 2010. Lastly, we analyze the effectiveness of each feature used by SEERLAB, and provide a sum­mary of our findings.

SEERLAB consists of three main components: a section parser, a candidate keyphrase extractor, and a keyphrase ranker. To generate keyphrases for a paper, the section parser first segments the document into pre-defined generic section types. Secondly, the candidate keyphrase extractor gen­erates a list of candidate phrases based on the doc­ument content. Then, the keyphrase ranker ranks each candidate according to the likelihood that it is a keyphrase. The top candidates are selected as keyphrases of the paper.

In this section, we discuss the performance and the implementation detail of our system in the Keyphrase Extraction Task. Each model in the ex­periment is trained on the training data, containing 144 documents, and is evaluated on a separate data set of 100 documents. The performance of each model is measured using Precision (P), Recall (R) and F-measure (F) for the top 5, 10 and 15 can­didates. A keyphrase is considered correct if and only if it exactly matches one of the answer keys. No partial credit is given.

Three baseline systems were provided by the or­ganizer: TFIDF, NB and ME. All baselines use the simple unigrams, bigrams and trigrams as candi­dates and TFIDF as features. TFIDF is an unsu­pervised method that ranks each candidate based on TFIDF scores. NB and ME are supervised Naive Bayes and Maximum Entropy respectively.

We use the randomForest package in R for our

M ean DecreaseAccu r MeanDecreaseGin

Figure 1 : Variable importance for each feature keyphrase ranker (Liaw and Wiener, 2002). All RF models are built with the following parame­ters: the number of trees = 200 and the number of features considered at each split = 3. The average training and testing time are around 15s and 5s.

Table 1. compares the performance of three different SEERLAB models against the baselines. RFo is the basic model, where the training data is imbalanced. For RF\:\, the negative examples are down-sampled to make the data balanced. For RFi-j, the negative examples are down-sampled to where its ratio with the positive examples is 7 to 1. All three models significantly outperform the baselines. The RFi-j model has the high­est performance, while the RF\:\ model performs slightly worse than the basic model -RFo. This shows that while the sampling strategy helps, over­doing it can hurt the performance. The optimal sampling ratio (RFi-j) is chosen according to a 10-fold cross-validation on the training data. For the top 15 candidates, RFi-j's F-score (C) ranks sixth among the 19 participants with a 24.34% F-score approximately 1% lower than the third place team. We also observed that SEERLAB performs quite well for the top 5 candidates with 39% Preci­sion (C). Its F-scores at the top 5, 19.84% (C) and 18.19% (R), place SEERLAB third and second re­spectively among other participants.

Figure 1. shows two variable importance in­dicators for each feature: mean decrease accu­racy (MDA) and mean decrease Gini (MDG). Both indicators measure each feature's contribu­tion in identifying whether a candidate phrase is a keyphrase. The MDA of a feature is computed by randomly permuting the value of that feature in the training data and then measuring the decrease

in prediction accuracy. If the permuted feature is 184

Table 1 : Performance (%) comparison for the Keyphrase Extraction Task. R (Reader) indicates that the reader-assigned keyword is used as the gold-standard and C (Combined) means that both author-assigned and reader-assigned keyword sets are used.

a very good predictor, then the prediction accu­racy should decrease substantially from the orig­inal model. The MDG of a feature implies that average Gini decreases for the nodes in the forest that use that feature as the splitting criteria.

TFIDF and DF are good indicators of perfor­mance according to both MDA and MDG. Both are very effective when used as splitting criteria, and the prediction accuracy is very sensitive to them. Surprisingly, the length of the phrase (AO also has high importance. Also, TFutie and ACRO have high MDA but low MDG. They have high MDA because if a candidate phrase is an acronym or appears in the title, it is highly likely that it is a keyphrase. However, most keyphrases are not acronyms and do not appear in titles. Thus, on average as splitting criteria, they do not de­crease Gini index by much, resulting in a low MDG. Also, TFrdated_wark and TFheaders have lower MDA and MDG than TF of other sections (TFintro, TFabs, and TFconclusion). This might suggest that the occurrences in the "Related Work" section or section headers are not strong indica­tors of being a keyphrase as the occurrences in the sections "Introduction," "Abstract" and "Conclu­sion."

We have described our SEERLAB system that participated in the Keyphrase Extraction Task. SEERLAB combines unsupervised corpus-based approach with Random Forests to identify keyphrases. The experimental results show that our system performs well in the Keyphrase Ex­traction Task, especially on the top 5 key phrase candidates. We also show that the down-sampling strategy can be used to enhance our performance.