Key terms (or keyphrases or keywords) are meta­data providing general information about the con­tent of a document. Their selection by authors or readers is, to a large extent, subjective which makes automatic extraction difficult. This is, how­ever, a valuable exercise, because such key terms constitute good topic descriptions of documents which can be used in particular for information retrieval, automatic document clustering and clas­sification. Used as subject headings, better key­words can lead to higher retrieval rates of an arti­cle in a digital library.

We view automatic key term extraction as a sub-task of the general problem of extraction of tech­nical terms which is crucial in technical and scien­tific documents (Ahmad and Collingham, 1996).

Among the extracted terms for a given scientific document in a given collection, which key terms best characterize this document?

This article describes the system realized for the Semeval 2010 task 5, based on GROBID's (GeneRation Of Bibliographie Data) module ded­icated to key term extraction. GROBID is a tool for analyzing technical and scientific documents, focusing on automatic bibliographical data extrac­tion (header, citations, etc.) (Lopez, 2009) and structure recognition (section titles, figures, etc).

As the space for the system description is very limited, this presentation focuses on key aspects. We present first an overview of our approach, then our selection of features (section 3), the different tested machine learning models (section 4) and the final post-ranking (section 5). We briefly describe our unsuccessful experiments (section 6) and we conclude by discussing future works.

Principle As most of the successful works for keyphrase extraction, our approach relies on Ma­chine Learning (ML). The following steps are ap­plied to each document to be processed:

1. Analysis of the structure of the article.

2. Selection of candidate terms.

3. Calculation of features.

4. Application of a ML model for evaluating each candidate term independently.

5. Final re-ranking for capturing relationships between the term candidates.

For creating the ML model, steps 1-3 are applied to the articles of the training set. We view steps 1 and 5 as our main novel contributions. The struc­ture analysis permits the usage of reliable features in relation to the logical composition of the arti­cle to be processed. The final re-ranking exploits general relationships between the set of candidates which cannot be captured by the ML models.

Candidate term selection In the following, word should be understood as similar to token in the sense of MAF. Step 2 has been implemented in a standard manner, as follows:

2. Remove all candidate n-grams starting or ending with a stop word,

3. Filter from these candidates terms having mathematical symbols,

4. Normalize each candidate by lowercasing and by stemming using the Porter stemmer.

Training data The task's collection consists of articles from the ACM (Association for Computa­tional Machinery) in four narrow domains (C.2.4 Distributed Systems, H.3.3 Information Search and Retrieval, 1.2.6 Learning and J.4 Social and Behavioral Sciences). As training data, we used this task's training resources (144 articles from ACM) and the National University of Singapore (NUS) corpus (156 ACM articles from all com­puting domains). Adding the additional NUS training data improved our final results (+7.4% for the F-score at top 15, i.e. from 25.6 to 27.5).

We experimented different ML models: Decision tree (C4.5), Multi-Layer perceptron (MLP) and Support Vector Machine (SVM). In addition, we combined these models with boosting and bagging techniques. We used WEKA (Witten and Frank, 2005) for all our experiments, except for SVM where LIBSVM (Chang and Lin, 2001) was used. We failed to obtain reasonable results with SVM. Our hypothesis is that SVM is sensitive to the very large number of negative examples compared to the positive ones and additional techniques should be used for balancing the training data. Results for decision tree and MLP were similar but the lat­ter is approx. 57 times more time-consuming for training. Bagged decision tree appeared to per­form constantly better than boosting (+8,4% for F-score). The selected model for the final run was, therefore, bagged decision tree, similarly as, for instance, in (Medelyan, 2009).

Post-ranking uses the selected candidates as a whole for improving the results, while in the pre­vious step, each candidate was selected indepen­dently from the other. If we have a ranked list of term Ti_jv, each having a score s(Ti), the new score s' for the term Tj is obtained as follow:

where a is a constant in [0 — 1] for control­ling the re-ranking factor, a has been set ex­perimentally to 0.8. P{Tj\Ti) is the probability that the keyword Tj is chosen by the author when the keyword Tj has been selected. For obtain­ing these probabilities, we use statistics for the HAL research archive. HAL contains approx. 139,000 full texts articles described by a rich set of metadata, often including author's keywords. We use the keywords appearing in English and in the Computer Science domain (a subset of 29,000 ar­ticles), corresponding to a total of 16,412 different keywords. No smoothing was used. The usage of open publication repository as a research resource is in its infancy and very promising.

Our system was ranked first of the competition among 19 participants. Table 1 presents our offi­cial results (Precision, Recall, F-score) for com­bined keywords and reader keywords, together with the scores of the systems ranked second (WINGNUS and KX FBK).

The previously described features were selected because they all had a positive impact on the ex­traction accuracy based on our experiments on the task's collection. The following intuitively perti­nent ideas appeared, however, to deteriorate or to be neutral for the results.

Noun phrase filtering We applied a filtering of noun phrases based on a POS tagging and extrac­tion of all possible NP based on typical patterns. This filtering lowered both the recall and the pre­cision (—7.6% for F-score at top 15).

Term variants We tried to apply a post-ranking by conflating term variants using FASTR, result­ing in a disappointing —11.5% for the F-score.

Global keywordness We evaluated the key­wordness using also the overall HAL keyword fre­quencies rather than only the training corpus. It had no impact on the results.

Language Model deviation We experimented the usage of HMM deviation using LingPipe as alternative informativeness measure, resulting in —3.7% for the F-score at top 15.

Wikipedia term Relatedness Using Wikipedia Miner, we tried to apply as post-ranking a boosting of related terms, but saw no impact on the results.

We think that automatic key term extraction can be highly valuable for assisting self-archiving of research papers by authors in scholarly reposito­ries such as arXiv or HAL. We plan to experiment keyword suggestions in HAL based on the present system. Many archived research papers are cur­rently not associated with any keyword.

We also plan to adapt our module to a large col­lection of approx. 2.6 million patent documents in the context of CLEF IP 2010. This will be the op­portunity to evaluate the relevance of the extracted key terms for large scale topic-based IR.