<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="20"/><title>Classifying articles in English and German Wikipedia</title><author surname="Ringland" givenname="Nicky"><org  name="University of Sydney" country="Australia" city="Sydney"/></author><author surname="Nothman" givenname="Joel"><org  name="University of Sydney" country="Australia" city="Sydney"/></author><author surname="Murphy" givenname="Tara"><org  name="University of Sydney" country="Australia" city="Sydney"/></author><author surname="Curran" givenname="James R"><org  name="University of Sydney" country="Australia" city="Sydney"/></author></firstpageheader><frontmatter><p><b>Classifying articles in English and German Wikipedia</b></p><p><b>Nicky Ringland </b>and <b>Joel Nothman </b>and <b>Tara Murphy </b>and <b>James R. Curran</b></p><p>School of Information Technologies University of Sydney NSW 2006, Australia {nicky,joel,tm,james}@it.usyd.edu.au</p></frontmatter><abstract>Named Entity (ne) information is criti­cal for Information Extraction (ie) tasks. However, the cost of manually annotating sufficient data for training purposes, espe­cially for multiple languages, is prohibitive, meaning automated methods for develop­ing resources are crucial. We investigate the automatic generation of ne annotated data in German from Wikipedia. By incor­porating structural features of Wikipedia, we can develop a German corpus which accurately classifies Wikipedia articles into ne categories to within 1% F-score of the state-of-the-art process in English. </abstract></header><body><section number="1" title="Introduction"><p>Machine Learning methods in Natural Language Processing (nlp) often require large annotated training corpora. Wikipedia can be used to au­tomatically generate robust annotated corpora for tasks like Named Entity Recognition (ner), com­petitive with manual annotation (Nothman et al., 2009). The CoNLL-2002 shared task defined ner as the task of identifying and classifying the names of people (per), organisations (org), places (loc) and other entities (misc) within text (Tjong Kim Sang, 2002). There has been extensive research into recognising nes in news­paper text and domain-specific corpora, however most of this has been in English. The cost of pro­ducing sufficient ner annotated data required for training makes manual annotation unfeasible, and the generation of this data is even more impor­tant for languages other than English, where gold­standard corpora are harder to obtain.</p><p>German ner is especially challenging since various features used successfully in English ner, including proper noun capitalisation, do not apply to German language data, making nes harder to detect and classify (Nothman et al., 2008). Furthermore, German has partially free word order which affects the reliability of con­textual evidence, such as previous and next word features, for ne detection.</p><p>Nothman et al. (2008) devised a novel method of automatically generating English ne train­ing data by utilising Wikipedia's internal struc­ture. The approach involves classifying all arti­cles in Wikipedia into classes using a features-based bootstrapping algorithm, and then creating a corpus of sentences containing links to articles identified and classified based on the link's target.</p><p>We extend the features used in Nothman et al. (2008) for use with German Wikipedia by cre­ating new heuristics for classification. We en­deavour to make these as language-independent as possible, and evaluate on English and German.</p><p>Our experiments show that we can accurately classify German Wikipedia articles at an F -score of 88%, and 91% for entity classes only, achieving results very close to the state-of-the-art method for English data by Nothman et al. (2008) who reported 89% on all and 92% on entities only. Nothman et al.'s (2009) ner training corpus cre­ated from these entity classifications outperforms the best cross-corpus results with gold standard training data by up to 12% F-score using CoNLL-2003-style evaluation. Thus, we show that it is possible to create free, high-coverage ne anno­tated German-language corpora from Wikipedia.</p><page local="2" global="21"/></section><section number="2" title="Background"><p>The area of ner has developed considerably from the Message Understanding Conferences (muc) of the 1990s where the task first emerged. met, the Multilingual Entity Task associated with muc introduced ner in languages other than English (Merchant et al., 1996) which had previously made up the majority of research in the area. The conll evaluations of 2002 and 2003 shifted the focus to Machine Language, and further multilingual ner research incorporated language-independent ner. CoNLL-2002 evalu­ating on Spanish and Dutch (Tjong Kim Sang, 2002) and CoNLL-2003 on English and German (Tjong Kim Sang and De Meulder, 2003).</p><p>The results of the CoNLL-2002 shared task showed that whilst choosing an appropriate ma­chine learning technique affected performance, feature choice was also vital. All of the top 8 systems at CoNLL-2003 used lexical features, pos tags, affix information, previously predicted ne tags and orthographic features.</p><p>The best-performing CoNLL-2003 system achieved an F-score of 88.8% on English and 72.4% on German (Florian et al., 2003). It com­bined Maximum Entropy Models and robust risk minimisation with the use of external knowledge in the form of a small gazetteer of names. This was collected by manually browsing web pages for about two hours and was composed of 4500 first and last names, 4800 locations in Germany and 190 countries. Gazetteers are very costly to create and maintain, and so considerable research has gone into their automatic generation from online sources including Wikipedia (Toral and Munoz, 2006).</p><p>The CoNLL-2003 results for German were con­siderably lower than for English, up to 25% difference in F-score (Tjong Kim Sang and De Meulder, 2003). The top performing systems all achieved F-scores on English more than 15 higher than on German.</p><subsection number="2.1" title="German NER"><p>German is a very challenging language for ner, because various features used in English do not apply. There is no distinction in the capitalisa­tion of common and proper nouns so the number of word forms which must be considered as po­tential nes is much larger than for languages such as English. German's partially free word order also means that surface cues, such as per enti­ties often preceding verbs of communication, are much weaker.</p><p>A final consideration is gender. The name Mark is likely to be on any list of German person names, but also makes up part of Germany's old currency, the Deutsche Mark, also known as D-Mark or just Mark (gender: female; <i>die), </i>and also has the meaning 'marrow' (gender: neuter; <i>das). </i>Whilst gender can sometimes disambiguate word senses, in more complicated sentence construction, gen­der distinctions reflected on articles and adjec­tives can change or be lost when a noun is used in different cases.</p></subsection><subsection number="2.2" title="Cross-language Wikipedia"><p>The cross-lingual link structure of Wikipedia rep­resents a valuable resource which can be ex­ploited for inter-language nlp applications. Sorg and Cimiano (2008) developed a method to au­tomatically induce new inter-language links by classifying pairs of articles of two different lan­gauges as connected by a inter-language link. They use a classifier utilising various text and graph-based features including edit distance be­tween the title of articles and link patterns. They find that since the fraction of bidirectional links (cases where the English article e.g. Dog is linked to the German article Hund which is linked to the original English article) is around 95% for Ger­man and English, they can be used in a bootstrap­ping manner to find new inter-language links. The consistency and accuracy of the links was also found to vary, with roughly 50% of German lan­guage articles being linked to their English equiv­alents, and only 14% from English to German.</p><p>Richman and Schone (2008) proposed a sys­tem in which English Wikipedia article classi­fications are used to produce ne-annotated cor­pora in other languages, achieving an F-score of up to 84.7% on French language data, evaluated against human-annotated corpora with the muc evaluation metric. So far there has been very little research into directly classifying articles in non-English Wikipedias.</p><page local="3" global="22"/></subsection><subsection number="2.3" title="Learning NER"><p>Machine learning approaches to ner are flexible due to their statistical data-driven approach, but training data is key to their performance (Noth­man et al., 2009). The size and topical coverage of Wikipedia makes its text appropriate for train­ing general nlp systems.</p><p>The method of Nothman et al. (2008) for trans­forming Wikipedia into an ne annotated corpus relies on the fact that links between Wikipedia ar­ticles often correspond to nes. By using structural features to classify an article, links to it can be la­belled with an ne class.</p><p>The process of deriving a corpus of NE anno­tated sentences from Wikipedia consists of two main sub-tasks: (1) selecting sentences to include in the corpus; and (2) classifying articles linked in those sentences into NE classes. By relying on redundancy, articles that are difficult to classify with confidence may simply be discarded.</p><p>This method of processing Wikipedia enables the creation of free, much larger ne-annotated corpora than have previously been available, with wider domain applicability and up-to-date, copy­right free text. We focus on the first phase of this process: accurate classification of articles.</p><p>nlp tasks in languages other than English are disadvantaged by the lack of available data for training and testing. Developing more automated methods of language-resource generation which is independent of existing data sets is an impor­tant and challenging goal. We work towards gen­erating high-coverage training corpora which can be used for a range of German nlp.</p><doubt alpha="66.7" length="6" tooSmall="False" monospace="0.0">3 Data</doubt><p>To learn a classification of German Wikipedia ar­ticles, we labelled a corpus of English Wikipedia articles. Wikipedia's inter-language links allow us to then develop classifiers for all articles in En­glish and German (or other language) Wikipedias. We use xml dumps of Wikipedia from March 2009 for both languages.</p></subsection><subsection number="3.1" title="Article selection"><p>Both Nothman et al. (2008) and Dakka and Cucerzan (2008) have labelled collections of Wikipedia articles with gold standard classifica-</p><p>Table 1: Most frequently viewed Wikipedia articles from August 2008, retrieved from http://stats.grok.se tions. Both of these consist of randomly se­lected articles, Dakka and Cucerzan's consisting of a random set of 800 pages, expanded by list co-occurrence. Nothman et al.'s data set ini­tially consisted of 1100 randomly sampled ar­ticles from among all Wikipedia articles. This biased the sample towards entity types that are frequent in Wikipedia, such as authors and al­bums, but poorly represented countries, for ex­ample, which are important but are only a small proportion of Wikipedia's articles. A high num­ber of the selected articles were stubs or other pages which were comparatively underdeveloped in structure and text. As a result, the data set was augmented with a further 200 articles, randomly sampled from among articles with at least 700 in­coming links <i>(in-links).</i></p><p>We took a more complex approach to choos­ing articles for inclusion in our data set, to ensure greater utility for multilingual Wikipedia tasks. We selected ^2300 articles from:</p><p>• the top 1000 most frequently viewed, based on August 2008 statistics (see Table 1), and</p><p>• the most linked-to articles (see Table 2),</p><p>with the constraint that they appear in at least the top 10 largest language Wikipedias (Table 3).</p><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Rank</p></td><td class="cell"><p>Article</p></td><td class="cell"><p>pageviews</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p>2008 Summer Olympics</p></td><td class="cell"><p>4437251</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2</p></td><td class="cell"><p>Wiki</p></td><td class="cell"><p>4030068</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3</p></td><td class="cell"><p>Sarah palin</p></td><td class="cell"><p>4004853</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>4</p></td><td class="cell"><p>Michael phelps</p></td><td class="cell"><p>3476803</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>5</p></td><td class="cell"><p>YouTube</p></td><td class="cell"><p>2685316</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>6</p></td><td class="cell"><p>Bernie Mac</p></td><td class="cell"><p>2013775</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>7</p></td><td class="cell"><p>Olympic Games</p></td><td class="cell"><p>2003 678</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>8</p></td><td class="cell"><p>Joe Biden</p></td><td class="cell"><p>1 966877</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>9</p></td><td class="cell"><p>Georgia (country)</p></td><td class="cell"><p>1 757967</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>10</p></td><td class="cell"><p>The Dark Knight (film)</p></td><td class="cell"><p>1 427 277</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 2: Most linked-to articles of English Wikipedia." class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Rank</p></td><td class="cell"><p>Title</p></td><td class="cell"><p>Inlinks</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p>United States</p></td><td class="cell"><p>543 995</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2</p></td><td class="cell"><p>Australia</p></td><td class="cell"><p>344969</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3</p></td><td class="cell"><p>Wikipedia</p></td><td class="cell"><p>272073</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>4</p></td><td class="cell"><p>Association Football</p></td><td class="cell"><p>241 514</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>5</p></td><td class="cell"><p>France</p></td><td class="cell"><p>227 464</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="4" global="23"/><p>Table 4: Average size (in paragraphs, sentences and categories) of Nothman et al.'s ^1300 labelled articles from 2008 and our ~2300 articles from March 2009.</p><p>We experimented with selecting the articles with the most inter-language links, but results were not meaningful; languages such as Volapuk may have fewer than 30 speakers, but more than 100,000 articles, most of which are stubs created and edited automatically. Reducing the languages of interest to 10 allowed us to focus on select­ing more meaningful articles, using the criteria above. Although we deemed these criteria appro­priate, they skew the corpus to events relevant in August 2008; the Summer 2008 Olympics, up­coming American presidential Election and con­flict between Russia and Georgia were prominent in the data.</p><p>In Table 4, we show that we succeed in select­ing articles which are more substantial than the random sample of Nothman et al. (2008). Our method largely avoided the "long tail" of more obscure articles, such as old songs, sports players or archaeological finds, whose representation in a random sample is disproportionate to their utility.</p></subsection><subsection number="3.2" title="Annotation"><p>Our corpus was created by manually classifying approximately 2300 English articles which had German equivalents, selected as described in sec­tion 3.1 using a custom annotation tool described</p><p>Table 5: Breakdown of manual classifications: Peo­ple, Locations, Organisations, Miscellaneous, Com­mon and Disambiguation.</p><p>in Tardiff et al. (2009). It allowed for an arbi­trary number of annotators, and for multiple an­notations to be compared.</p><p>Annotation was carried out using a hierarchical fine-grained tag-set based upon guidelines from the bbn Technologies' 150 answer types (Brun­stein, 2002). Categories were able to be added into the hierarchy and either 'grown' or 'shrunk' to better fit the data as the annotators saw it. The ability to add categories is especially important when annotating Wikipedia because many cate­gories such as types of works of art or products are not adequately covered in bbn.</p><p>The corpus was annotated using fine-grained categories, adding more information for use in future work, and enabling easier annotation, as they allow an annotator to classify a topic into a well-defined sub-category, which can then be uni­formly mapped to a coarse-grained category. For example, all hotels can be classified as hotel, which then can be mapped to either org or loc as decided after annotation.</p><p>The annotation process allowed for a high level of feedback to annotators, with statistics includ­ing inter-annotator agreement and a list of articles not uniformly classified available during annota­tion. This allowed annotators to quickly and eas­ily identify digressions from one another.</p><p>All articles were double-annotated. After tag­ging the first 78 articles, we discussed conflicts and refined the annotation scheme. The two an­notators then both classified a further 1100 arti­cles each, achieving inter-annotator agreement of 97.5% on fine-grained tags, and 99.5% on coarse­grained tags. A further discussion and annota­tion round of the remaining ^1100 followed, and the final inter-annotator agreement was 99.7% on fine-grained tags and 99.9% on coarse-grained tags, creating a highly accurate corpus which we plan to release upon publication. Coarse-grained class distribution is given in Table 5.</p><table caption="Table 3: Top ten Wikipedia languages by number of articles (nearest 50 000) as at September 2009." class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Wikipedia</p></td><td class="cell"><p>Articles</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>English</p></td><td class="cell"><p>3 500000</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>German</p></td><td class="cell"><p>950000</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>French</p></td><td class="cell"><p>850000</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>polish</p></td><td class="cell"><p>650000</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Japanese</p></td><td class="cell"><p>650000</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Italian</p></td><td class="cell"><p>600000</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Dutch</p></td><td class="cell"><p>550000</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Spanish</p></td><td class="cell"><p>500000</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>portuguese</p></td><td class="cell"><p>500000</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Russian</p></td><td class="cell"><p>450000</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>per</p></td><td class="cell"><p>loc</p></td><td class="cell"><p>org</p></td><td class="cell"><p>misc</p></td><td class="cell"><p>non</p></td><td class="cell"><p>dis</p></td><td class="cell"><p>Total</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>271</p></td><td class="cell"><p>648</p></td><td class="cell"><p>229</p></td><td class="cell"><p>392</p></td><td class="cell"><p>650</p></td><td class="cell"><p>79</p></td><td class="cell"><p>2269</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>12%</p></td><td class="cell"><p>29%</p></td><td class="cell"><p>10%</p></td><td class="cell"><p>17%</p></td><td class="cell"><p>29%</p></td><td class="cell"><p>3%</p></td><td class="cell"><p>100%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Dataset</p></td><td class="cell"><p># articles</p></td><td class="cell"><p>Paras</p></td><td class="cell"><p>Sents</p></td><td class="cell"><p>Cats</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>English 0805</p></td><td class="cell"><p>1296</p></td><td class="cell"><p>3.0</p></td><td class="cell"><p>36.4</p></td><td class="cell"><p>4.6</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>English 0903</p></td><td class="cell"><p>2269</p></td><td class="cell"><p>8.8</p></td><td class="cell"><p>122.4</p></td><td class="cell"><p>6.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>German 0903</p></td><td class="cell"><p>2269</p></td><td class="cell"><p>4.8</p></td><td class="cell"><p>84.1</p></td><td class="cell"><p>3.3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="5" global="24"/><p><b>Hand-labelled seed data</b></p><figure caption="Figure 1: A bootstrapping approach to article classification"></figure></subsection></section><section number="4" title="Classification"><p>Classification of Wikipedia's articles into seman­tic groupings is useful for applications such as named entity recognition (Kazama and Torisawa, 2007; Nothman et al., 2008) and ontology con­struction (Suchanek et al., 2007). The Wikipedia category hierarchy is a folksonomy and not di­rectly suitable for nlp tasks. Instead, rule-based (Toral and Mufioz, 2006; Richman and Schone, 2008), semi-supervised (Nothman et al., 2008; Mika et al., 2008) and supervised (Dakka and Cucerzan, 2008) article classifiers have derived coarse-grained entity groupings or taxonomies.</p><p>Features used in classification are varied. Suchanek et al. (2007) used Wikipedia categories to map articles to WordNet, but noted that con­ceptual categories in English usually have plural head nouns (e.g. Coastal <b>cities </b>in Australia) which describe the nature of member articles, as opposed to thematic categories like James Bond. Richman and Schone (2008) scanned the hierar­chy of categories for known phrases to classify articles into named entity categories.</p><p>Since an article's topic is usually defined in its first sentence, Toral and Mufioz (2006) try to match words from the opening sentence to a re­lated class through the WordNet taxonomy. The specific use of the predicative head noun follow­ing a copula (is, were, etc.) in the first sentence was suggested by Kazama and Torisawa (2007) as a single feature by which articles may be grouped.</p><p>Other approaches utilise the co-occurrence of entities in lists (Watanabe et al., 2007; Bhole et al., 2007; Dakka and Cucerzan, 2008); presence of entities in particular fields of <i>infobox </i>templates which summarise the properties and relations of article topics (Mika et al., 2008); and bag-of-words svm classification (Dakka and Cucerzan, 2008; Bhole et al., 2007).</p><p>Although using different data sets, both Noth-man et al. (2008) and Dakka and Cucerzan (2008) have reported F-scores of approximately 90% for classification into conll style entity categories.</p><subsection number="4.1" title="Classifying Wikipedia articles"><p>Nothman et al. (2008)'s bootstrapping classifier works as follows (see Figure 1): By initially as­sociating features of each training instance with its gold-standard class label, an initial classifica­tion of all articles in Wikipedia is produced. Fea­tures that are consistently associated with a partic­ular predicted class are then mapped to that class, including those not present in the hand-labelled data. These classification and mapping stages are then repeated, increasing feature coverage until the classifications are generally stable. Such an approach allows for high recall over sparse multi­valued features like the Wikipedia category mem­bership of each article. We extend their approach to German Wikipedia.</p></subsection><subsection number="4.2" title="Increasing non-entity recall"><p>The following rules help to determine whether an article describes a named entity, or a non-entity topic (non).</p><p><b>Capitalisation </b>In English, all named entities are proper nouns, which are conventionally cap­italised. This can be utilised by observing the capitalisation of all incoming links, with ba­sic features allowing for determiners and non-conventional orthographies such as gzip or iPod.</p><p>In German, since all nouns are capitalised, this distinction is lost. Furthermore, adjectival forms<page local="6" global="25"/></p><p><u>University of Sydney</u></p><doubt alpha="86.8" length="38" tooSmall="True" monospace="0.0">rrorn Wikipedia. the free encyclopedia</doubt><p><b>The University of Sydney </b>(informally <b>Sydney Uni </b>or <b>USyd) </b>is the <u>oldes:|.." ye sity|</u>in <u>Australia</u>. It was established in Australia's "<u>Grhup of Eigrif</u> are highly ranged in terms of\heir research performance students and [ making it the <b>ï</b></p><p><b>The University of Sydney</b></p><p>Lowercase links       Definition NP     Plural NP categories</p><p>Figure 2: A portion of the Wikipedia article on the University of Sydney with some useful features marked.</p><p>including nes of countries (eg. "Australian") are not capitalised in German ("australisch"), which means even a basic heuristic to check whether a link is a noun is not feasible.</p><p><b>List identification </b>If the English article title be­gins List of or German, Liste, we mark it as non.</p><p><b>Disambiguation identification </b>Wikipedia's disambiguation articles list candidate referents for a particular title. The German page Mark lists amongst others, the name, substance (marrow), river, saints and various British tanks from WW1 of the same name. Most disambiguation pages are children of a category of Disambiguation, many have the word Disambiguation or Begriffskiarung in the title, and further information is available in the form of disambiguation templates.</p></subsection><subsection number="4.3" title="Bootstrapped features"><p>For general classification, we extracted features from articles, which may each be mapped to an entity class. These mappings are produced by the bootstrapping process.</p><p><b>Category nouns </b>Head nouns from Wikipedia category titles for both English and German, ex­tracted using C&amp;C tools (Curran and Clark, 2003) in English and the Tree-Tagger in German (Schmid, 1995) to pos-tag and chunk the sentences. In En­glish, the category feature only applied to plural head nouns (and bigrams thereof) following Suchanek et al.'s (2007) suggestion that these best represent ontol­ogy. Differences in both language and the structure of the German Wikipedia project invalidate this ap­proach in German: conceputal categories are not plu­ral, and forms that are bigrams in English are generally compound nouns. Hence we experimented with asv toolbox (Chris Biemann and Holz, 2008) to extract a head morpheme. This allows Premierminister (Prime Minister) and Wissenschaftsminister (Science Minister) to both be interpreted as Minis­ter, and Kernbrennstoffaufbereitungsan­lage (nuclear fuel treatment facility) to become an­lage (facility).</p><p><b>Definition nouns </b>We term a <i>definition noun </i>to be the first noun following a copula in the first sentences, such as university in Figure 2. Definition nouns are extracted using pos-tagging and chunking as per cat­egory nouns, from articles which had been split into sentences and tokenised according to the method de­scribed in Nothman et al. (2008).</p><p>For each article, the number of category nouns mapped to each class is counted, and the most fre­quent class is used to label the article. If this is in­conclusive, or the highest class leads by only one cat­egory, the definition noun is used to decide the class. Where there are no mapped category nouns or defini­tion nouns available, or no winning class can be de­cided, the article class is marked as unknown (unk).</p><p>An article classification is considered confident for use in bootstrapping if it is not labelled unk, and if none of the features disagree (i.e. all category and def­inition features available map to the same class).</p><page local="7" global="26"/><p>Table 7: Results of bootstrapping iterations on the held-out test set of German and English, compared to English 0805, as reported in Nothman (2008).</p></subsection></section><section number="5" title="Results"><p>We present results for our German Wikipedia classi­fier, exploring the effect of bootstrapping and feature variants in comparison to Nothman et al.'s (2008) En­glish Wikipedia classifier.</p><p>We were able to achieve 88% F-score for German classification on a held-out test-set of 15% of the data (Table 6). These results are comparable to those pre­sented by Nothman on English, but slightly lower than those using our larger annotated training corpus on a 2009 dump of English Wikipedia.</p><p><b>Data Validation </b>The inter-language links between the German and English Wikipedias were checked and found to be reliable, with only two errors in links from English to German pages from the test set used for ex­perimentation, which is consistent with the findings of Sorg and Cimiano (2008). Both of these were ar­ticles pointing to disambiguation pages: Nikon (de­scribing the company) and Isosceles (describing the type of triangle). In the held-out training set, similar, though few, mis-links were found: the English arti­cle on French playwright Molière linking to Molière (1978), a French film depicting his life, and the Ger­man article Ryan Vikedal, a former member of the band Nickelback, links to the English article of the same name, which itself redirects to Nickelback. It should be noted that all of these examples were cor­rectly classified by our process. When these errors were corrected, F-score improved by under 0.1%, showing that even with occasional noise, inter-wiki language links can be used to produced good-quality data. The results we present use a uncorrected test set.</p><p><b>Bootstrapping </b>Bootstrapping was found to be less effective than in Nothman (2008) (see Table 7), where it was more needed to increase recall given less manually-labelled seed data. With the larger seed, bootstrapping proved more important on the German data than English, with recall increasing 5% compared to 2%, still falling short of the 11% increase found by Nothman. In our experiments, we found that the re­sults were unchanging after the second feedback stage.</p><p><b>Feature Analysis </b>In Table 6, we examine the effects of removing some classification features, and compare against the same process on English Wikipedia. In English, the capitalisation feature im­proves recall slightly, as opposed to the substantial increase found in Nothman's work; we might expect German, in which capitalisation is not used, to be dis-advantaged by a similar amount.</p><p>Category nouns are seen to be by far the most im­portant feature, especially in German. Our experi­ments to extract the morphology-based head from each category noun were an attempt to increase recall. We observed a slightly higher recall in the seed classifi­cation, but the bootstrapping process - also designed to improve recall - was more effective with the finer granularity of whole category noun features. This ulti­mately led to slightly reduced recall, leading us to use whole category nouns in our remaining experiments.</p><p>Definition nouns gave mixed results. In German they improved recall but had little effect on precision, while in English they improved precision and recall.</p><p><b>Cross-validation </b>The results of ten-fold cross-validation are shown in Table 8, with a class break­down. Our system left 8% of German and 6% of En­glish Wikipedia articles unclassified (unk). Nothman (2008) reports that 10% of articles were left unclassi­fied. Our present work was able to classify a greater proportion due to our selection of more, higher-quality seed articles.</p><p>The German system performs very well on loc and on misc, which is known to be difficult to classify, achieving almost equivalent scores to English. The system also achieves a high F-score on per. All of the false negatives when classifying people were on articles describing fictional characters such as Luzifer, Godzilla and Hermaphroditos. The error analysis of org also shows that we fail to correctly classify ar­ticles which the annotators also were unsure of, such as eBay and amazon.com, and Jedi. misc often ap­peared incorrectly classified as org, showing the of­ten blurred distinction between a product and the or­ganisation which produces it (eg: Jeep and Airbus A380). The bbn guidelines also proved difficult for the classifier to adhere to, with 'attractions' such as the Nürburgring being classified as loc not misc.</p><p>Table 9 compares the precision, recall and F-score of English and German overall and on entity classes only. We also report the standard deviation of per­formance over the ten folds of cross-validation. The larger gap between all-class and entity class results in German reflects the low non recall (76% as opposed to 90% in English), likely due to no available capitali­sation feature.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Iter</p></td><td class="cell"><p>German 0903</p><p><i>P     R F</i></p></td><td class="cell"><p>English 0903</p><p><i>P     R F</i></p></td><td class="cell"><p>English 0805</p><p><i>P    R F</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>0</p></td><td class="cell"><p>93</p></td><td class="cell"><p>79</p></td><td class="cell"><p>85</p></td><td class="cell"><p>95</p></td><td class="cell"><p>88</p></td><td class="cell"><p>92</p></td><td class="cell"><p>93</p></td><td class="cell"><p>73 82</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p>93</p></td><td class="cell"><p>83</p></td><td class="cell"><p>88</p></td><td class="cell"><p>97</p></td><td class="cell"><p>91</p></td><td class="cell"><p>94</p></td><td class="cell"><p>93</p></td><td class="cell"><p>79 85</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2</p></td><td class="cell"><p>93</p></td><td class="cell"><p>84</p></td><td class="cell"><p>88</p></td><td class="cell"><p>95</p></td><td class="cell"><p>90</p></td><td class="cell"><p>93</p></td><td class="cell"><p>93</p></td><td class="cell"><p>80 86</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3</p></td><td class="cell"><p>93</p></td><td class="cell"><p>84</p></td><td class="cell"><p>88</p></td><td class="cell"><p>95</p></td><td class="cell"><p>90</p></td><td class="cell"><p>93</p></td><td class="cell"><p>95</p></td><td class="cell"><p>84 89</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="8" global="27"/><p>Table 6: Subtractive feature analysis on the held-out test set, comparing German Wikipedia with English (0903) performance, and the results reported by Nothman (2008) (English 0805).</p><p>Table 9: Classification performance (average and standard deviation) over ten-fold cross-validation.</p><p>Table 8: Class distribution of manually classified arti­cles and average results of ten-fold cross-validation.</p></section><section number="6" title="Conclusion"><p>Our work develops a semi-supervised classifier assign­ing German Wikipedia articles to named entity tags, in comparison to English Wikipedia. In doing so, we labelled a large corpus (2269 articles) of English Wikipedia pages, and validated the use of Wikipedia's inter-language links to transfer those training classifi­cations to the smaller German encyclopedia.</p><p>In distinction from previous annotations of Wikipedia data, we produced a corpus with fine­grained classes, extending on BBN's 150 answer types (Brunstein, 2002), and consisting of only articles which satisfy popularity criteria.</p><p>The classifier we have produced for German Wikipedia achieves very high precision (97%) and recall (87%) on entity classes. Due to differences between English and German language, orthography and Wikipedia editorial style, we had to modify the semantic and structural features previously used to classify English Wikipedia articles (Nothman et al., 2008). Our use of bootstrapping to spread this seman­tic knowledge to features unseen in training greatly improves performance in German, in which capitalisation features cannot be easily applied to distinguish nes from non-entities, and in which there are fewer features available for classification, due to a smaller, less-developed Wikipedia.</p><p>We intend to improve the classifier by exploring fur­ther features, as well as the integrity of article resolu­tion and inter-language links.</p><p>The results we have presented in German are only 3% F-score lower than on English articles and 1% F-score lower when only evaluating on nes. The CoNLL-2003 shared task presented a 12% minimum reduction in performance for German ner when com­pared to English (Tjong Kim Sang and De Meulder, 2003). This substantial difference is due either to the difficulty of the ner task in German, or to the paucity of training data available in the CoNLL-2003 shared task, where the German training data marked only half as many nes as the English corpus. By transforming the links in Wikipedia into entity annotations, we in­tend to generate large ne-annotated corpora, and to evaluate their use for learning German ner. Our high-accuracy classifier therefore reduces the need for ex­pensive manual annotation in languages other than En­glish where resources tend to be scarce.</p></section><section title="Acknowledgments"><p>We would like to thank members of the Language Technology Research Group and the anonymous re­viewers for their helpful feedback. This work was par­tially supported by the Capital Markets Cooperative Research Centre Limited (cmcrc) and by Australian Research Council Discovery Project DP0665973. Joel Nothman was supported by a University of Sydney Vice-Chancellor's Research Scholarship and a cmcrc PhD Scholarship.</p><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Classification features</p></td><td class="cell"><p>German 0903</p></td><td class="cell"><p>English 0903</p></td><td class="cell"><p>English 0805</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>P</p></td><td class="cell"><p>R</p></td><td class="cell"><p><i>F</i></p></td><td class="cell"><p>P</p></td><td class="cell"><p><i>R</i></p></td><td class="cell"><p><i>F</i></p></td><td class="cell"><p>P</p></td><td class="cell"><p><i>R</i></p></td><td class="cell"><p><i>F</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>All features</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>95</p></td><td class="cell"><p>90</p></td><td class="cell"><p>93</p></td><td class="cell"><p>95</p></td><td class="cell"><p>84</p></td><td class="cell"><p>89</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>- Capitalisation</p></td><td class="cell"><p>93</p></td><td class="cell"><p>84</p></td><td class="cell"><p>88</p></td><td class="cell"><p>96</p></td><td class="cell"><p>89</p></td><td class="cell"><p>92</p></td><td class="cell"><p>92</p></td><td class="cell"><p>80</p></td><td class="cell"><p>85</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>+ Category morphology</p></td><td class="cell"><p>93</p></td><td class="cell"><p>83</p></td><td class="cell"><p>88</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>— Definition nouns</p></td><td class="cell"><p>93</p></td><td class="cell"><p>81</p></td><td class="cell"><p>87</p></td><td class="cell"><p>93</p></td><td class="cell"><p>88</p></td><td class="cell"><p>91</p></td><td class="cell"><p>95</p></td><td class="cell"><p>80</p></td><td class="cell"><p>87</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>- Category nouns</p></td><td class="cell"><p>48</p></td><td class="cell"><p>7</p></td><td class="cell"><p>12</p></td><td class="cell"><p>76</p></td><td class="cell"><p>28</p></td><td class="cell"><p>41</p></td><td class="cell"><p>48</p></td><td class="cell"><p>13</p></td><td class="cell"><p>21</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Wikipedia</p></td><td class="cell"><p>P</p></td><td class="cell"><p><i>R</i></p></td><td class="cell"><p><i>F</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>English 0903</p></td><td class="cell"><p>All</p></td><td class="cell"><p>94</p></td><td class="cell"><p>±2</p></td><td class="cell"><p>89</p></td><td class="cell"><p>±1</p></td><td class="cell"><p>91</p></td><td class="cell"><p>±1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Entities</p></td><td class="cell"><p>98</p></td><td class="cell"><p>±1</p></td><td class="cell"><p>89</p></td><td class="cell"><p>±2</p></td><td class="cell"><p>93</p></td><td class="cell"><p>±1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>German 0903</p></td><td class="cell"><p>All</p></td><td class="cell"><p>91</p></td><td class="cell"><p>±3</p></td><td class="cell"><p>84</p></td><td class="cell"><p>±3</p></td><td class="cell"><p>88</p></td><td class="cell"><p>±2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Entities</p></td><td class="cell"><p>97</p></td><td class="cell"><p>±2</p></td><td class="cell"><p>87</p></td><td class="cell"><p>±4</p></td><td class="cell"><p>92</p></td><td class="cell"><p>±3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Class</p></td><td class="cell"><p>%</p></td><td class="cell"><p>German 0903</p><p>P     R F</p></td><td class="cell"><p>English 0903</p><p>P     R F</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>non</p></td><td class="cell"><p>29</p></td><td class="cell"><p>85</p></td><td class="cell"><p>76</p></td><td class="cell"><p>80</p></td><td class="cell"><p>86</p></td><td class="cell"><p>90 88</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>dab</p></td><td class="cell"><p>3</p></td><td class="cell"><p>92</p></td><td class="cell"><p>99</p></td><td class="cell"><p>96</p></td><td class="cell"><p>100</p></td><td class="cell"><p>90 95</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>loc</p></td><td class="cell"><p>29</p></td><td class="cell"><p>98</p></td><td class="cell"><p>95</p></td><td class="cell"><p>96</p></td><td class="cell"><p>99</p></td><td class="cell"><p>97 98</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>misc</p></td><td class="cell"><p>17</p></td><td class="cell"><p>89</p></td><td class="cell"><p>71</p></td><td class="cell"><p>78</p></td><td class="cell"><p>97</p></td><td class="cell"><p>67 79</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>org</p></td><td class="cell"><p>10</p></td><td class="cell"><p>93</p></td><td class="cell"><p>85</p></td><td class="cell"><p>89</p></td><td class="cell"><p>97</p></td><td class="cell"><p>91 94</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>per</p></td><td class="cell"><p>12</p></td><td class="cell"><p>92</p></td><td class="cell"><p>94</p></td><td class="cell"><p>93</p></td><td class="cell"><p>96</p></td><td class="cell"><p>98 97</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="9" global="28"/></section><references><p>Abhijit Bhole, Blaz Fortuna, Marko Grobelnik, and Dunja Mladenic. 2007. Extracting named entities and relating them over time based on Wikipedia. <i>Informatica, </i>31:463—468. Ada Brunstein. 2002. Annotation guidelines for an­swer types. LDC2005T33, Linguistic Data Consor­tium, Philadelphia.</p><p>Gerhard Heyer Chris Biemann, Uwe Quasthoff and Florian Holz. 2008. Asv toolbox: a modular collection of language exploration tools. In <i>Pro­ceedings of the Sixth International Language Re­sources and Evaluation (LREC'08), </i>pages 1760­1767, Marrakech, Morocco, May. European Lan­guage Resources Association (ELRA). James R. Curran and Stephen Clark. 2003. Language independent NER using a maximum entropy tag­ger. In <i>Proceedings ofthe seventh conference on Natural Language Learning, </i>pages 164-167, Mor-ristown, NJ, USA.</p><p>Wisam Dakka and Silviu Cucerzan. 2008. Augment­ing Wikipedia with named entity tags. <i>Proceedings ofthe 3rd International Joint Conference on Natu­ral Language Processing, </i>pages 545-552.</p><p>Radu Florian, Abe Ittycheriah, Hongyang Jing, and Tong Zhang. 2003. Named entity recognition through classifier combination. In <i>Proceedings of the 7th Conference on Natural Language Learning, </i>pages 168-171. Jun'ichi Kazama and Kentaro Torisawa. 2007. Exploiting Wikipedia as external knowledge for named entity recognition. In <i>Proceedings ofthe 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, </i>pages 698-707.</p><p>Roberta Merchant, Mary Ellen Okurowski, and Nancy Chinchor. 1996. The multilingual entity task (MET) overview. In <i>Proceedings of the Tipster Text Program Phase II, </i>pages 445-447, Vienna, Vir­ginia, May.</p><p>Peter Mika, Massimiliano Ciaramita, Hugo Zaragoza, and Jordi Atserias. 2008. Learning to tag and tag­ging to learn: A case study on Wikipedia. <i>IEEE Intelligent Systems, </i>23(5):26-33. Joel Nothman, James R. Curran, and Tara Murphy.</p><p>2008. Transforming wikipedia into named entity training data. In <i>Proceedings of the Australian Lan­guage Technology Workshop, </i>pages 124-132.</p><p>Joel Nothman, Tara Murphy, and James R. Curran.</p><p>2009. Analysing Wikipedia and gold-standard cor­pora for NER training. <i>In Proceedings of the 12th Conference ofthe European Chapter ofthe ACL (EACL 2009), </i>pages 612-620, March.</p><p>Joel Nothman. 2008. <i>Learning Named Entity Recog-nitionfrom Wikipedia. </i>School of Information Tech­nologies, University of Sydney, Honours Thesis.</p><p>Alexander E. Richman and Patrick Schone. 2008. Mining wiki resources for multilingual named en­tity recognition. <i>Proceedings of the 46th Annual Meeting ofthe Association for Computational Lin­guistics: Human Language Technologies, </i>pages 1­9.</p><p>Helmut Schmid. 1995. Improvements in Part-of-Speech Tagging with an Application to German. <i>Proceedings of the ACL-SIGDAT Workshop, </i>March.</p><p>Philipp Sorg and Philipp Cimiano. 2008. Enrich­ing the crosslingual link structure of wikipedia classification-based approach. <i>Proceedings of the AAAI2008 Workshop on Wikipedia and Artifical In­telligence.</i></p><p>Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: a core of semantic knowl­edge — unifying WordNet and Wikipedia. In <i>Pro­ceedings of the 16th international conference on</i></p><doubt alpha="64.4" length="45" tooSmall="False" monospace="0.0">World Wide Web,pages 697-706, Banff, Alberta,</doubt><p>Canada.</p><p>Sam Tardiff, James R. Curran, and Tara Murphy. 2009. Improved text categorisation for Wikipedia named entities. In <i>Proceedings ofthe Australian Language Technology Workshop.</i></p><p>Erik F. Tjong Kim Sang and Fien De Meulder.</p><p>2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recogni­tion. In <i>Proceedings ofthe 7th Conference on Nat­ural Language Learning, </i>pages 142-147. Erik F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In <i>Proceedings ofthe 6th Conference on Natural Language Learning, </i>pages 1-4.</p><p>Antonio Toral and Rafael Mufioz. 2006. A proposal to automatically build and maintain gazetteers for named entity recognition using Wikipedia. In <i>Pro-ceedingsofthe Workshop on NEW TEXT, 11th Con­ference ofthe European Chapter ofthe Association for Computational Linguistics.</i></p><p>Yotaro Watanabe, Masayuki Asahara, and Yuji Mat-sumoto. 2007. A graph-based approach to named entity categorization in Wikipedia using conditional random fields. In <i>Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Lan­guage Processing and Computational Natural Lan­guage Learning, </i>pages 649-657.</p></references></body></article>