KP-Miner: Participation in SemEval-2
Samhaa R. El-Beltagy
Cairo University Giza, Egypt.
samhaa@computer.org
Ahmed Rafea
The American University in Cairo New Cairo, Egypt.
rafea@aucegypt.edu
KP-Miner (El-Beltagy, 2006) (El-Beltagy, 2009) is a system for the extraction of keyphrases from English and Arabic documents. When developing the system, the goal was to build a general purpose keyphrase extraction system that can be easily configured by users based on their understanding of the documents from which keyphras-es are to be extracted and without the need for any training documents or the use of any sophisticated natural language processing or linguistic tools. As such, the keyphrase extraction process in KP-Miner is an un-supervised one. When building a general purpose keyphrase extraction system, this was an important objective, as training data is not always readily available for any type of data. The goal of entering the KP-Miner system into the SemEval-2 competition, was to see how well it will perform on a specific task, without making any changes in its default parameters.
Keyphrase extraction in the KP-Miner system is a three step process: candidate keyphrase selection, candidate keyphrase weight calculation and finally keyphrase refinement. Each of these steps, is explained in the following sub-sections. More details about the employed algorithm, and justification for using certain values for selected parameters, can be found in (El-Beltagy, 2009).
In KP-Miner, a set of rules is employed in order to elicit candidate keyphrases. As a phrase will never be separated by punctuation marks within some given text and will rarely have stop words within it, the first condition a sequence of words has to display in order to be considered a candidate keyphrase, is that it is not be separated by punctuation marks or stop words. A total of 187 common stopwords (the, then, in, above, etc) are used in the candidate keyphrase extraction step. After applying this first condition on any given document, too many candidates will be generated; some of which will make no sense to a human reader. To filter these out, two further conditions are applied. The first condition states that a phrase has to have appeared at least n times in the document from which keyphrases are to be extracted, in order to be considered a candidate keyphrase. This is called the least allowable seen frequency(lasf) factor and in the English version of the system, this is set to 3. However, if a document is short, n is decremented depending on the length of the document.
The second condition is related to the position where a candidate keyphrase first appears within an input document. Through observation as well as experimentation, it was found that in long documents, phrases occurring for the first time after a given threshold, are very rarely keyphras-es. So a cutoff constant CutOff is defined in terms of a number of words after which if a phrase appears for the first time, it is filtered out and ignored. The initial prototype of the KP-Miner system (El-Beltagy, 2006), set this cutoff value to a constant (850). Further experimentation carried out in (El-Beltagy, 2009) revealed that an optimum value for this constant is 400. In the implementation of the KP-Miner system, the phrase extraction step described above is carried out in two phases.
Single key features obtained from documents by models such as TF-IDF (Salton and Buckley, 1988) have already been shown to be representative of documents from which they've been extracted as demonstrated by their wide and successful use in clustering and classification tasks. However, when applied to the task of keyphrase extraction, these same models performed very poorly (Turney, 1999). By looking at almost any document, it can be observed that the occurrences of phrases is much less frequent than the occurrence of single terms within the same document. So it can be concluded that one of the reasons that TF-IDF performs poorly on its own when applied to the task of keyphrase extraction, is that it does not take this fact into consideration which results in a bias towards single words as they occur in larger numbers. So, a boosting factor is needed for compound terms in order to balance this bias towards single terms. In this work for each input document d from which keyphras-es are to be extracted, a boosting factor Bd is calculated as follows:
and if Bd > a then Bd = a
Here INdI is the number of all candidate terms in document d, IPdI is the number of candidate terms whose length exceeds one in document d and « and a are weight adjustment constants. The values used by the implemented system are 3 for a and 2.3 for « .
To calculate the weights of document terms, the TF-IDF model in conjunction with the introduced boosting factor, is used. However, another thing to consider when applying TF-IDF for a general application rather than a corpus specific one, is that keyphrase combinations do not occur as frequently within a document set as do single terms. In other words, while it is possible to collect frequency information for use by a general single keyword extractor from a moderately large set of random documents, the same is not true for keyphrase information. There are two possible approaches to address this observation. In the first, a very large corpus of a varied nature can be used to collect keyphrase related frequency information. In the second, which is adopted in this work, any encountered phrase is considered to have appeared only once in the corpus. This means that for compound phrases, frequency within a document as well as the boosting factor are really what determine its weight as the idf value for all compound phrases will be a constant c determined by the size of the corpus used to build frequency information for single terms. If the position rules described in (El-Beltagy, 2009) are also employed, then the position factor is also used in the calculation for the term weights. In summary, the following equation is used to calculate the weight of candidate keyphrases whether single or compound:
wij = tfi* idf* Bi* Pf Where:
wij = weight of term tj in Document Ditfij = frequency of term tj in Document Diidf = log2 N/n where N is the number of documents in the collection and n is number of documents where term tj occurs at least once. If the term is compound, n is set to 1.
Bi = the boosting factor associated with document Di
Pf= the term position associated factor. If position rules are not used this is set to 1.
The KP-Miner system, allows the user to specify a number n of keyphrases s/he wants back and uses the sorted list to return the top n keyphrases requested by the user. The default number of n is five. As stated in step one, when generating candidate keyphrases, the longest possible sequence of words that are uninterrupted by possible phrase terminators, are sought and stored and so are sub-phrases contained within that sequence provided that they appear somewhere in the text on their own.
One of the new tracks introduced to SemEval this year is a track dedicated entirely to keyphrase extraction from scientific articles. The task was proposed with the aim of providing participants with "the chance to compete and benchmark" this technology (SemEval2, 2010).
In this competition, participants were provided with 40 trial documents, 144 training documents, and 100 test documents. For the trial and training data, three sets of answers were provided: author-assigned keyphrases, reader-assigned keyp-hrases, and finally a set that is simply a combination between the 2 previous sets. Unlike author-assigned keyphrases, which may or may not occur in the content, all reader-assigned keyphrases were said to have been extracted from the papers. The participants were then asked to produce the top 15 keyphrases for each article in the test document set and to submit the stemmed version of these to the organizers.
Evaluation was carried out in the traditional way in which keyphrase sets extracted by each of the participants were matched against answer sets (i.e. author-assigned keyphrases and reader-assigned keyphrases) to calculate precision, recall and F-score. Participants were then ranked by F-score when extracting all 15 keyphrases.
Since the KP-miner system is an unsupervised keyphrase extraction system, no use was made of the trial and training data. The system was simply run of the set of test documents, and the output was sent to the organizers. 2 different runs were submitted: one produced used the initial prototype of the system, (El-Beltagy, 2006), while the second was produced using the more mature version of the system (El-Beltagy, 2009). Both systems were run without making any changes to their default parameters. The idea was to see how well the KP-Miner would fair among other keyphrase extraction systems without any additional configuration. The more mature version of the system performed better when its results were compared to the author-reader combined keyphrase set and consequently was the one whose final results were taken into consideration in the competition. The system ranked at 2 , with a tie between it and another system when extracting 15 keyphrases from the combined keypharse set. The results are shown in table 1.
Precision | Recall | F-Score | |||
HUMB | 27.2% | 27.8% | 27.5% | ||
WINGNUS | 24.9% | 25.5% | 25.2% | ||
KP-Miner | 24.9% | 25.5% | 25.2% | ||
SZTERGAK | 24.8% | 25.4% | 25.1% | ||
ICL | 24.6% | 25.2% | 24.9% | ||
SEERLAB | 24.1% | 24.6% | 24.3% | ||
Table 1: Performance of all participating systems over combined keywords when extracting 15 keyphrases
When evaluating the system on reader assigned keyphrases only (again when extracting 15 keyphrases), the KP-Miner system ranked at 6 with a tie between it and another system. The system's precision, recall, and f-score were:
19.3%, 24.1% , 21.5% respectively.
To test whether the phrase refinement step described in section 2.3 would improve the results or not, this option was turned on, and the results were evaluated using the script and the golden dataset provided by the competition organizers. The results are shown in tables 2 and 3.
__Precision Recall__F-Score
Had these results been submitted, the system would have still ranked at number 2 (but more comfortably so) when comparing its results to the combined author-reader set of keywords, but it would jumped to third place for the reader assigned keyphrases. This improvement confirms what the authors hypothesized in (El-Beltagy, 2009) which is that the usage of the final refinement step does lead to better quality keyphrases.
Despite the fact that the KP-Miner was designed as a general purpose keyphrase extraction system, and despite the simplicity of the system and the fact that it requires no training to function, it seems to have performed relatively well when carrying out the task of keyphrase extraction from scientific documents. The fact that it was outperformed, seems to indicate that for optimal performance for this specific task, further tweaking of the system's parameters should be carried out. In future work, the authors will investigate the usage of machine learning techniques for configuring the system for specific tasks. A further improvement to the system can entail allowing certain stopwords to appear within the produced keyphrases. It is worth noting that the organizers stated that 55 of the reader assigned keyphrases and 6 of the author assigned keyphrases (making a total of 61 keyphrases in the combined dataset), contained the "of" stop-word. However, none of these would have been detected by the KP-Miner system as currently "of" is considered as a phrase terminator.
KX_FBK | 23.6% | 24.2% | 23.9% | ||
DERIUNLP | 22.0% | 22.5% | 22.3% | ||
Maui | 20.3% | 20.8% | 20.6% | ||
DFKI | 20.3% | 20.7% | 20.5% | ||
BUAP | 19.0% | 19.4% | 19.2% | ||
SJTULTLAB | 18.4% | 18.8% | 18.6% | ||
UNICE | 18.3% | 18.8% | 18.5% | ||
UNPMC | 18.1% | 18.6% | 18.3% | ||
JU_CSE | 17.8% | 18.2% | 18.0% | ||
LIKEY | 16.3% | 16.7% | 16.5% | ||
UvT | 14.6% | 14.9% | 14.8% | ||
NIRAJIIITH | 14.1% | 14.5% | 14.3% | ||
POLYU | 13.9% | 14.2% | 14.0% | ||
UKP | 5.3% | 5.4% | 5.3% | ||
Precision | Recall | F-Score | |||
Top 5 | 29.6% | 12.3% | 17.4% | ||
Top 10 | 23.3% | 20.5% | 24.3% | ||
Top 15 | 25.3% | 26.1% | 25.8% | ||
M Porter. 1980. An Algorithm for Suffix Stripping, Program, 14, 130-137.
G. Salton and C. Buckley. 1988. Term-weighting Approaches in Automatic Text Retrieval, Information Processing and Management, 24:513-523.
Peter D. Turney. 1999. Learning to Extract Keyphras-es from Text, National Research Council, Institute for Information Technology, ERB-1057.
Miner: A Keyphrase Extraction System for English and Arabic Documents Information Systems, 34(1):132-144.
System for Effective Keyphrase Extraction. Proceeding of the 3rd IEEE International Conference on Innovations in Information Technology (IIT '06), Dubai, UAE.
SemEval. 2010. http://semeval2.fbk.eu/semeval2.php