<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="767"/><title>An Empirical Comparison of Unknown Word Prediction Methods</title><pubinfo>Proceedings of the 5th International Joint Conference on Natural Language Processing,pages 767-775, Chiang Mai, Thailand, November 8 - 13, 2011. ©2011 AFNLP</pubinfo><author surname="Cholakov" givenname="Kostadin"><org  name="Lund University" country="Sweden"/></author><author surname="van Noord" givenname="Gertjan"><org  name="University of Groningen" country="The Netherlands" city="Groningen"/></author><author surname="Kordoni" givenname="Valia"><org  name="Saarland University" country="Germany" city="Saarbrucken"/></author><author surname="Zhang" givenname="Yi"><org  name="Saarland University" country="Germany" city="Saarbrucken"/></author></firstpageheader><frontmatter><p><b>An Empirical Comparison of Unknown Word Prediction Methods</b></p><p><b>Kostadin Cholakov^, Gertjan van Noord*, Valia Kordoni*, Yi Zhang*</b></p><p><b>I </b>University of Groningen, The Netherlands <b><i>X </i></b>Saarland University and DFKI GmbH, Germany {k.cholakov,g.j.m.van.noord}@rug.nl {kordoni,yzhang}@coli.uni -sb.de</p></frontmatter><abstract>We compare two types of methods which deal with unknown words in the context of computational grammars. Methods of the first type are based on the idea of <i>su-pertagging </i>and use a tagger to predict lex­ical descriptions for unknown tokens in a given input. The second type of methods perform <i>lexical acquisition </i>(LA) which, in the context of this paper, refers to the au­tomatic acquisition of new lexical entries for the lexicon of a given grammar. The methods are compared based on the effect their application has on the parsing cov­erage and accuracy of the GG grammar of German (Crysmann, 2003). In particu­lar, we adapt the LA method of Cholakov and van Noord (2010) which was origi­nally developed for the Dutch Alpino sys­tem to be used with the GG. Its impact on coverage and accuracy on a test corpus of German newspaper texts is compared to the results reported previously on the same corpus for methods which employed a tag­ger. Furthermore, in a smaller experiment, we show that the linguistic knowledge this LA method provides can also be used for sentence realisation. </abstract></header><body><section number="1" title="Introduction"><p>Computational grammars of natural language which rely on hand-crafted lexicons containing elaborate linguistic descriptions usually have low lexical coverage. This is due to the fact that it is impossible to list every word in a language in the lexicon. If a word is unknown to the grammar, or its description in the lexicon is wrong or incom­plete, the grammar might fail to produce a (cor­rect) analysis.</p><p>In this paper, we compare two types of ap­proaches for dealing with unknown words. The first type is based on the concept of supertagging while the second one performs LA. Generally, su­pertagging refers to the process of applying a se­quential tagger to assign lexical descriptions asso­ciated with each word in an input string, relative to a given grammar. It was introduced as a means to reduce parsing ambiguity of LTAG grammars (Bangalore and Joshi, 1999), and has since been applied within CCG (Clark, 2002; Clark and Cur-ran, 2004) and HPSG (Dridan et al., 2008; Zhang et al, 2010) grammars. Supertagging has also been employed for dealing with unknown words. However, in such methods, the tagger is used to as­sign lexical descriptions <b>only </b>to the unknown to­kens in a given sentence. It is important to note here that henceforth, we will use the term <i>sup-pertagging </i>in this narrow sense of tagging un­known words only. Supertagging methods often work <i>online. </i>The unknown words are assigned lexical entries when they are encountered in the input during parsing. Therefore, the focus is pri­marily on improving the parsing coverage and ac­curacy of the grammar for a particular input. The performance of such methods is usually evaluated in terms of <i>token accuracy, </i>that is the proportion of correctly tagged unknown lexical tokens in the input.</p><p>LA, on the other hand, aims at learning auto­matically new lexical entries and thus, at extend­ing the lexicon of a given grammar. The strategy is to improve the parsing coverage and accuracy by improving the quality and the coverage of the lex­icon of the grammar. While suppertagging meth­ods are concerned with a particular form of the un­known word within the particular context this form occurs in the input, LA techniques look at various forms and contexts of the unknown word and use more detailed linguistic information to learn a new lexical entry for it. This makes LA methods more complex and time-consuming. That is why, they are often applied <i>offline. </i>LA methods are usually evaluated in terms of <i>type precision </i>and <i>type re­call.</i><page local="2" global="768"/><i> </i>For a given LA method, type precision indi­cates the proportion of correctly predicted lexical entries and type recall indicates how many of the correct lexical entries for a given word are actually found.</p><p>Both supertagging and LA have been success­fully used. For instance, Blunsom and Bald­win (2006) employ a conditional random field-based tagger to predict lexical entries for large-scale HPSG grammars of English (ERG; (Copes-take and Flickinger, 2000)) and Japanese (JACY; (Siegel and Bender, 2002)). Zhang and Kordoni (2006) and Dridan et al. (2008) have developed a maximum entropy-based (ME) tagger and inves­tigate the effect its application has on the pars­ing performance of the ERG and the GG. Other methods which apply the same tagger to the GG include Nicholson et al. (2008) and Cholakov et of the DELPH-IN collaboration<footnote anchor="1"/> and as such, they share the same grammar design and parsing ar­chitecture which facilitates the application of the same tagger. Baldwin (2005) presents a LA ap­proach where various secondary resources (POS taggers, chunkers, etc.) are used to create an ab­straction of words unknown to the ERG and then binary classifiers are employed to learn lexical en­tries for those words. However, learning is done based on incomplete information obtained by the various resources used. Further, no evaluation of the effect the method has on the parsing perfor­mance of the ERG is provided. Cholakov and van Noord (2010) describe a LA method where a ME-based classifier is used to acquire lexical en­tries for all forms in the paradigms of words, un­known to the large-scale Dutch Alpino grammar (van Noord, 2006). The paper also shows that the application of LA improves the parsing accuracy of Alpino on open-domain texts.</p><doubt alpha="60.0" length="45" tooSmall="False" monospace="0.0">al. (2008). The ERG, the GG and JACY are part</doubt><p>For our purpose of comparing supertagging and LA, we investigate how methods implementing these strategies affect the parsing coverage and accuracy when applied with the same grammar, namely the GG. Our choice of German and the GG as a platform for comparison is motivated by the fact that German has richer morphology and a more free word order than both English and Dutch which makes unknown word handling more chal­lenging. We adapt the LA method of Cholakov and van Noord (2010) - henceforth C&amp;vN- and applied with the GG. Then, we compare the pars­ing coverage and accuracy the GG achieves on a German newspaper corpus to those reported for supertagging methods applied previously with the GG on the same corpus. For all techniques, the experiments show that their application leads to an increase in coverage compared to the baseline, that is the standard GG setup. However, the supertag-ging methods achieved this at the price of having lower accuracy than the baseline. The application of the adapted C&amp;vN method, on the other hand, increases parsing accuracy compared to the base­line. This difference in quality might not always be crucial since less accurate parses produced by the grammar can still be used successfully in many NLP applications. In such cases, the less com­plex supertagging methods might be the preferred choice. However, through a small sentence reali­sation experiment, we give an example of an ap­plication where high-quality LA is a prerequisite.</p><footnote label="1">http://wiki.delph-in.net/</footnote><p>Other kinds of LA techniques have also been proposed. Cussens and Pulman (2000) used a symbolic approach employing <i>inductive logic pro­gramming, </i>while Erbach (1990), Barg and Walther (1998) and Fouvry (2003) followed a unification-based approach. However, it is doubtful if those methods are scalable since they have not been ap­plied to large-scale grammars and no meaningful evaluation has been provided.</p><p>The remainder of the paper is organised as fol­lows. Section 2 describes the resources we em­ploy. Section 3 gives an overview of the supertag-ging methods previously applied with the GG. Section 4 describes the adaptation of the C&amp;vN method to the GG. Section 5 gives details on the training procedure for the ME-based classifier used in the C&amp;vN technique. Section 6 evaluates the parsing coverage and accuracy on the German newspaper corpus when this technique is applied with the GG and compares the results to the results reported previously for the suppertagging methods for this corpus. Section 7 explores the possibility of using newly acquired lexical entries in a small sentence realisation task. Section 8 concludes the paper.</p></section><section number="2" title="Resources"><p>The GG (Crysmann, 2003) is an HPSG grammar based on typed feature structures. The GG types are strictly defined within a type hierarchy. The grammar contains constructional and lexical rules, as well as a lexicon where words are assigned lex­ical types.<page local="3" global="769"/> Currently, it consists of 5K types, 115 rules and the lexicon contains approximately 55K entries. There are 411 distinct lexical types which words can be mapped onto. Below is an exam­ple of a GG lexical entry, namely the entry for the word <i>Aufgabe </i>(a task):</p><doubt alpha="64.2" length="137" tooSmall="False" monospace="0.0">aufgabe-n := count-noun-le &amp; [ MORPH.LIST.FIRST.STEM &lt;  "Aufgabe" &gt;, SYNSEM.LKEYS   [  --SUBJOPT -, KEYAGR c-n-f, KEYREL"_aufgabe_n_rel",</doubt><p>KEYSORT abstract,</p><p>MCLASS nclass-9 ]   ] .</p><p>The lexical type 'count-noun-le' shows that the word is a countable noun<footnote anchor="2"/>. The STEM type fea­ture specifies the stem of the word. The stem for nouns is the singular nominative noun form, for adjectives- the base noninflected form and for verbs- the root form. Adverbs in German have a single form which is used as the value of the STEM feature in adverb entries. Some nouns (e.g., <i>Baukosten </i>(building costs)) do not have all forms typical for German nouns. In such cases, the word itself is set as the value of the STEM feature. The KEYAGR feature indicates case, number and gen­der. In the example above, case and number are left underspecified while the gender is set to femi­nine. The value of SUBJOPT shows that this noun is always used with an article and MCLASS indi­cates its morphological paradigm. The KEYREL and KEYSORT features define the semantics of the word.</p><p>Further, we employ the PET system (Callmeier, 2000) to parse with the GG. PET is a system for ef­ficient processing of unification-based grammars. The system comprises a sophisticated preproces­sor, a bottom-up chart parser and a grammar com­piler. For our purposes, we should note that the parser has a built-in unknown word guesser which, based on some simple heuristics, assigns generic types to the unknown words. Most of the features in these types are left underspecified.</p><p>We compare the results of the various meth­ods based on how they affect the parsing cover­age and accuracy of the GG on the Frankfurter Rundschau (FR) newspaper corpus. The corpus contains 614K sentences and the articles in it deal with various domains.</p><footnote label="2">le stands for lexeme.</footnote></section><section number="3" title="Suppertagging Methods"><p>Here, we give an overview of supertagging meth­ods applied previously with the GG. Typically, the unknown words are assigned <i>open-class </i>lex­ical types. In the case of the GG, open-class lex­ical types are those assigned to nouns, adjectives, verbs and adverbs. It is assumed that closed-class words are already handled by the grammar.</p><p>Dridan et al. (2008) adopts the method proposed in Zhang and Kordoni (2006) for English and the ERG and applies it to the GG. A ME-based tagger is trained on features extracted from the context of the unknown word- four characters from the be­ginning and the end of the word, and two words of context (where available) either side of the target word together with their POS tags<footnote anchor="3"/>. The applica­tion of the tagger led to an improved parsing cov­erage on a test corpus containing 700 short Ger­man questions from the CLEF competition. No evaluation in terms of accuracy is provided for German. For the same experiment with the ERG, the accuracy achieved with the tagger was below the baseline accuracy (the standard ERG setup).</p><p>Nicholson et al. (2008) employs the same ME-based tagger but the paper examines its perfor­mance more closely. The tagger is evaluated by performing a 10-fold cross-validation on the tree-bank associated with the GG. Words which ap­peared only in the test fold were considered un­known. Since the sentences in the treebank are an­notated, the method is able to use the lexical types of the context words as features instead of their POS tags. The achieved token accuracy is 58%. Then, the tagger is used to predict lexical types for unknown words in a random sample of 1000 sentences extracted from the FR corpus. However, since no annotation is available for the test corpus, the tagger is run with the same features as in Dri-dan et al. (2008) but without the POS tags of the context words.</p><p>The results are given in Table 1. Coverage in­creased by 8% for the test corpus. An estimation of the parsing accuracy is also given. Parsing ac­curacy is measured as the proportion of sentences for which a correct parse is produced, among the set of parses. 87 sentences were randomly selected and manually examined. The baseline accuracy was 85%. When suppertagging is applied accu­racy drops to 84%. This is consistent with the results reported in Dridan et al. (2008) for the ERG.<page local="4" global="770"/></p><footnote label="3">The corpora used had been POS tagged with the Tree-Tagger (Schmid, 1994).</footnote><p>Cholakov et al. (2008) develops further the idea of using lexical types as features in the tagger. The paper presents the same system setup as Nicholson et al. (2008) but with one crucial difference. The tagset used in the experiments consists ofmore de­tailed descriptions of the GG lexical types. In this new tagset, the values of some of the morphosyn-tactic features from the lexical entries are attached to the type definition, thus making the informa­tion they carry accessible to the tagger. For exam­ple, according to the guidelines given in Cholakov et al. (2008), the lexical type of <i>Aufgabe </i>would become <i>count-noun-le</i>_-_/" after attaching the val­ues of the SUBJOPT and KEYAGR features to the type definition<footnote anchor="4"/>. Having this additional informa­tion, the token accuracy of the tagger increases to 73.54%. The effect the method has on the parsing coverage and accuracy is shown in Table 1. The experiment is done on a random sample of 10K sentences from the FR with the corpus POS tagged and the tags of the context words used as features in the tagger instead of lexical types. Parsing ac­curacy is measured as the proportion of 100 ran­domly selected sentences with at a correct parse produced. Again, accuracy decreases in compari­son to the baseline (the standard GG setup).</p><p>It is important to note that the lexical entries cre­ated by the aforementioned methods for unknown words are sort of generic due to the inability of those methods to access the values ofthe type fea­tures defined in the entries. In Nicholson et al. (2008) only the type of the word can be specified and the word is set as the value of the STEM fea­ture. Every other feature in the entry is either un-derspecified or assigned some default value. The technique of Cholakov et al. (2008) can define more features in the created lexical entry due to the detailed tagset used. However, accuracy still decreased.</p></section><section number="4" title="Lexical Acquisition with The GG"><p>In this section, we describe the adaptation of the C&amp;vN LA method to the GG. This adaptation raises the important question about portability of LA. Because of their complexity and dependence on a particular grammar, LA methods have been applied within a single parsing system, in a single framework, and mostly for a single language. So, it is unclear to what extent the various tech­niques can be used for a different language or pars­ing architecture. However, the C&amp;vN method is claimed explicitly to be portable to other systems and languages provided some conditions are met. These include: a finite set of labels which un­known words are mapped onto, a syntactic parser, and a morphological component which generates the paradigm(s) of a given unknown word.</p><footnote label="4">The values for case and number for nouns are almost al­ways left underspecified; that is why only the value of the gender is attached to the type definition.</footnote><p>Similar to Cholakov et al. (2008), we created a set of labels which consists of more refined defi­nitions of the GG lexical types. Accordingly, we attach the values of some of the type features de­fined in the relevant lexical entries to the type def­initions of those entries. However, for reasons of data sparseness, we choose to exclude some of the features which Cholakov et al. (2008) considered relevant. Experiments with different sets of fea­tures have shown that the best results are achieved when only features designating morphosyntactic agreement are considered. For all noun types and predicative adjectives this is the KEYAGR fea­ture and for verb types allowing for prepositional complements- the COMPAGR and the OCOM-</p><p>PAGR features which indicate the case of the the (oblique) complement. For instance, the lexical type of <i>Aufgabe </i>will become <i>count-noun-le</i>_/ af­ter the gender value of the KEYAGR feature is at­tached.</p><p>C&amp;vN propose a method for the generation of the paradigm(s) of a given unknown Dutch word (Cholakov and van Noord, 2009). The type of word form predicted by the paradigm for that word is used as a feature in the ME-based classifier em­ployed in C&amp;vN for learning unknown words. For example, the paradigm of <i>Aufgabe </i>will indi­cate that the word is a singular feminine noun and this information will be passed on to the classi­fier. Further, unlike the Dutch Alpino grammar, the GG does not have a full form lexicon. All word forms are derived by applying various mor­phological rules defined in the GG to the word stem. Therefore, we need to add only the cor­rect stem of the unknown word the lexicon. Then, all forms of the word will automatically be recog­nised by the grammar. That is why, in the case of the GG, we use the generated paradigms to per­form the mapping between unknown words and their stems. For instance, if <i>Aufgaben </i>(tasks) is an unknown word, we will add <i>Aufgabe, </i>the stem<page local="5" global="771"/></p><p>Table 1: Coverage and accuracy results for the GG and the FR corpus. <i>Lexical </i>refers to Nicholson et al. (2008) where only affixes and context words are used as features. The last row refers to Cholakov et al. (2008) where the POS tags of the context words are also employed.</p><p>indicated by its paradigm, to the lexicon.</p><p>Due to the GG design, it is not straightforward to use the morphological rules of the grammar for paradigm generation. Following the C&amp;vN tech­nique developed for generating the paradigms of Dutch words, we created a German finite state morphology. The morphology does not have ac­cess to any linguistic information and thus, it gen­erates all possible paradigms allowed by the word orthography. Then, the number of search hits Ya­hoo! returns for each form in a given paradigm is combined with some simple heuristics to disam­biguate the output of the morphology and to de­termine the correct paradigm(s). We also apply heuristics to guess the gender for words with gen­erated noun paradigms and to determine if a word which is assigned a verb paradigm starts with a separable particle.</p><p>One could argue that there is a simpler ap­proach for mapping the various forms of the un­known word to its stem. For instance, the Tree-Tagger provides both POS and stem information with high accuracy. However, the generation of the paradigms allows us to consider contexts in which other forms of a given unknown word occur and thus, to have access to much more and linguis­tically diverse data. Further, using the paradigms, we are able to determine which of the morpho­logical classes defined within the GG a given un­known word belongs to and specify the value of the MCLASS type feature in the lexical entry gen­erated for this word. We are also able to deter­mine the value of the VCOMPFORM type feature which indicates the separable particle in verb en­tries. However, those are not crucial for the clas­sification process and they are likely to cause data sparseness. That is why, those type features are as­signed their values after the lexical type of a given word has been predicted. The semantic features in the lexical entries for newly acquired words, as well as the other features whose values are not learnt via LA are assigned default values or left underspecified.</p><p>We adopted the ME-based classifier<footnote anchor="5"/> and the features used for unknown word prediction as de­scribed in C&amp;vN. The probability of a lexical type <i>t, </i>given an unknown word and its context <i>c </i>is:</p><doubt alpha="27.3" length="11" tooSmall="False" monospace="0.0">(1)p(t\c) =</doubt><doubt alpha="51.2" length="43" tooSmall="False" monospace="0.0">exP(J2i&lt;3&gt;ifi(t,c))J2t'eTexP(T,i@i/i(*',c))</doubt><p>where <b><i>fi</i></b><b><i>(t, </i></b><b><i>c) </i></b>may encode arbitrary features from the context and &lt; 61, 62,... &gt; can be evaluated by maximising the pseudo-likelihood on a training corpus (Malouf, 2002).</p><p>Table 2 shows the features for <i>Aufgabe. </i>Since the stem of the unknown word is added to the lex­icon, we also experimented with prefix and suf­fix features extracted from the stem. We assumed that those could allow for a better generalization of morphological properties but they proved to be less informative for LA.</p><p>Features <b>iii) </b>hyphen_no <b>iv) </b>noun_feminine #predicted by the paradigm <b>v) </b>count-noun-le_f, mass-noun-le_f <b>vi) </b>noun(f)</p><doubt alpha="61.1" length="18" tooSmall="False" monospace="0.0">i)A, Au, Auf, Aufg</doubt><doubt alpha="63.2" length="19" tooSmall="False" monospace="0.0">ii)e, be, abe, gäbe</doubt><table caption="Table 2: Features forAufgabe(a task)"></table><p>Rows <b>(v) </b>and <b>(vi) </b>show syntactic features obtained from what C&amp;vN refer to as 'parsing with uni­versal types'. In C&amp;vN, each unknown word is assigned all target lexical types, i.e. it is treated as being maximally ambiguous. However, to re­duce ambiguity and make the parsing computa­tionally feasible, we use the generated paradigms as a filtering mechanism and assign a given word only those types which belong to the POS of the paradigm(s) generated for this word. That is why, <i>Aufgabe </i>is assigned all noun lexical types. Sen­tences containing morphological forms of <i>Aufgabe</i> are then parsed with PET in best-only mode.<page local="6" global="772"/> For each sentence only the best parse selected by the disambiguation model of the parser is preserved. Then, the lexical type that has been assigned to the form of <i>Aufgabe </i>occurring in this parse is stored.</p><footnote label="5">The classifier is developed using the TADM tool; http : //tadm.sourceforge.net/</footnote><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Method</p></td><td class="cell"><p>Coverage (°/</p></td><td class="cell"><p><b>0</b></p></td><td class="cell"><p>Accuracy (%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Baseline (standard GG)</p></td><td class="cell"><p>GG + tagger</p></td><td class="cell"><p>Baseline (standard GG)</p></td><td class="cell"><p>GG + tagger</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Lexical</p></td><td class="cell"><p>12</p></td><td class="cell"><p>20</p></td><td class="cell"><p>85</p></td><td class="cell"><p>84</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Lexical+POS</p></td><td class="cell"><p>8.9</p></td><td class="cell"><p>21.1</p></td><td class="cell"><p>85</p></td><td class="cell"><p>83</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>We employ the most frequently used type(s), based on an empirical threshold (80% for the GG), as features in the classifier (row <b>v). </b>Further, as illustrated in row <b>(vi), </b>each piece of information attached to the type definitions (the part after the underscore) is also taken as a separate feature. By considering these syntactic features, we manage to involve the grammar directly in the prediction pro­cess, unlike the methods discussed in the previous section where only lexical features and POS tags were used. Here, the grammar can decide which lexical type(s) are best suited for a given unknown word. This is an effective way to include the syn­tactic constraints imposed on the unknown word into the prediction process.</p></section><section number="5" title="Training of The Classifier"><p>In order to train the classifier, we need annotated data. That is why, we temporarily remove some words from the GG lexicon and use them for train­ing and tuning the various parameters of the clas­sifier. The lexical entries of the removed words are used as gold standard.</p><p>We remove only words belonging to open-class lexical types. Each such type has at least 10 lexi­cal entries in the lexicon mapped onto it and it is assigned to at least 15 distinct words occurring in large corpora parsed with PET and the GG. The parsed corpus we use consists of roughly 2.5M sentences randomly selected from the German part of the Wacky project (Kilgarriff and Grefenstette, 2003). Following these criteria, we have selected 39 open-class types out of the 411 lexical types defined in the GG. The expansion of the type def­initions of the 39 types with the values of the rel­evant type features resulted in the creation of 68 <i>expanded </i>types. Table 3 gives more details about the type distribution.</p><p>2400 words are removed from the lexicon of the GG. Of these, 2000 are used for training, and 400 words are used as a test set to give an idea about the performance of the classifier. We as­sume that less frequent words are typically un­known and, in order to simulate their behaviour, all 2400 words have between 40 and 100 occur­rences in the parsed corpus. Experiments with a minimum lower than 40 occurrences have shown that this is a reasonable threshold to filter out ty­pos, tokenization errors, etc. The distribution of the parts-of-speech for the 2400 words and the evaluation of the paradigm generation component are shown in Table 4 (some words have more than a single part-of-speech). Accuracy indicates how many of the generated paradigms are correct.</p><p>In the paradigms generated for verbs there were three mistakes. However, the generated verb stems were all correct. Similarly, the stems for all nouns were correct, including the stems of 98 nouns which contained a mistake in their paradigm. In 91 cases the singular genitive form was incorrect, in another 12 cases the predicted gender was wrong. The mapping of the words to their correct stems is correct in all cases.</p><p>We allow for multiple types per word to be predicted but we discard the types accounting to­gether for less than 5% of probability mass. Ad­ditionally, there are four baseline methods. The <b><i>naive </i></b>one assigns the most frequent expanded type in the lexicon, <i>count-noun-le _f, </i>to each unknown word. In the <b><i>naive POS </i></b>baseline each word is given the most frequent expanded type for the POS of each paradigm generated for it. The <b><i>GG </i></b>baseline predicts for the unknown word the target type(s) used as features for this word in the clas­sifier (e.g., for <i>Aufgabe, </i>these are the types from row (v) in Table 2). For the <b><i>TnT</i></b>baseline, we used the treebank available for the GG to train the TnT POS tagger (Brants, 2000) with the tagset consist­ing of all lexical types in the GG. The sentences extracted for each unknown word are then tagged in best-only mode. Similarly to the GG baseline, the unknown word is given the type(s) most frequently assigned to it (80% threshold).<page local="7" global="773"/> The aver­age type precision and type recall for the 400 test words are given in Table 5. Table 6 shows the F-measure per POS for the LA model.</p><table caption="Table 3: Distribution of the target lexical types" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Original types</p></td><td class="cell"><p>Expanded types</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Total</p></td><td class="cell"><p>39</p></td><td class="cell"><p>68</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>-nouns</p></td><td class="cell"><p>5</p></td><td class="cell"><p>15</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>-verbs</p></td><td class="cell"><p>28</p></td><td class="cell"><p>45</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>-adjectives</p></td><td class="cell"><p>4</p></td><td class="cell"><p>6</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>-adverbs</p></td><td class="cell"><p>2</p></td><td class="cell"><p>2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 4: Paradigm generation results" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>overall</p></td><td class="cell"><p>nouns</p></td><td class="cell"><p>adj</p></td><td class="cell"><p>verbs</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>total</p></td><td class="cell"><p>2954</p></td><td class="cell"><p>1196</p></td><td class="cell"><p>651</p></td><td class="cell"><p>694</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>accuracy(%)</p></td><td class="cell"><p>96.45</p></td><td class="cell"><p>91.09</p></td><td class="cell"><p>100</p></td><td class="cell"><p>99.54</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>The LA model improves upon the baselines. The F-measure reaches 80% when 300 words are used for training and the learning curve flattens out at 1600 training words. The method performs very similar to the results reported for Dutch and Alpino, which demonstrates that it can be success­fully applied outside the environment it was pri­marily developed for.</p><p>Predicting lexical entries for verbs is the hard­est task for the LA model. The classifier has a strong bias towards assigning transitive and in­transitive verb types. It either fails to predict in­frequent frames or it wrongly predicts a transitive type for intransitive verbs and vice versa. Another difficulty for the model is the distinction which the GG makes between ergative and non-ergative verbs. The main issue with adverbs is that many of them can be used as adjectives as well. As a con­sequence, the classifier has a strong bias towards predicting an adverb type for words for which an adjective type has also been predicted. No pattern in the errors for nouns and adjectives can be iden­tified.</p></section><section number="6" title="Parsing Coverage and Accuracy"><p>Having adapted the C&amp;VN method to be used with the GG, we can now compare it with the su-pertagging methods by investigating how its ap­plication affects parsing coverage and accuracy on the FR corpus. We were not able to obtain the sentence sets used in Nicholson et al. (2008) and Cholakov et al. (2008), so we created a test set of 450 sentences randomly extracted from the FR corpus. All sentences contained at least one un­known word. The average sentence length is 17.16 tokens, with a ratio of 1.09 unknown words per sentence. For this experiment, we parse the 450 sentences with PET, under three conditions. In the first case, the standard configuration of the GG is used where the unknown word guesser assigns generic types to the unknown words. Dridan et al. (2008) report that passing a POS tagged input to the guesser enhances its performance and im­proves parsing coverage. That is why, in the sec­ond case, the 450 sentences were tagged with TnT outputting all POS tags with non-zero probabili­ties for each word. Then, the tagged sentences are parsed with PET. In the third case, we add to the GG lexicon lexical entries acquired offline by the adapted C&amp;vN method.</p><p>The results are given in Table 7. The mod­els employing POS tags and LA have practically the same coverage performance. The <i>GG+POS </i>one produced at least one parse for 113 sentences and the LA model produced analyses for those 113 sentences plus 4 more, thus giving a total of 117 parsed sentences. The standard GG model was able to cover 57 sentences. The parsed sentences for all models are manually examined and accu­racy is again measured as the percentage of those parsed sentences which had a correct parse pro­duced among the set of parses. There were 99 such sentences for the model which employs LA and 89 and 47 for the <i>GG+POS </i>and the <i>GG-standard </i>model, respectively. The drop in the accuracy for the <i>GG+POS </i>model compared to the standard GG one is consistent with the accuracy results reported note, though, that both the LA and <i>GG+POS </i>mod­els also produced a correct parse for the 47 sen­tences for which the standard model delivered a correct analysis.</p><doubt alpha="63.0" length="46" tooSmall="False" monospace="0.0">in Dridan et al. (2008) for the ERG. We should</doubt><p>The better accuracy result achieved by the LA model has to do mainly with the fact that the built-in guesser assigns noun types to the vast major­ity of the unknown words. The underspecified features in those entries create a lot of ambiguity and make it harder for the parser disambigua­tion model to select the correct analysis.<page local="8" global="774"/> The LA method, on the other hand, supplies the parser with more detailed and linguistically accurate lexical entries which facilitates ambiguity resolution.</p><table caption="Table 5: Results on the 400 test words" class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Model</p></td><td class="cell"><p>Prec(%)</p></td><td class="cell"><p>Rec(%)</p></td><td class="cell"><p>F-meas(%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Naive</p></td><td class="cell"><p>21.75</p></td><td class="cell"><p>21.07</p></td><td class="cell"><p>21.41</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Naive POS</p></td><td class="cell"><p>58.96</p></td><td class="cell"><p>47.65</p></td><td class="cell"><p>52.7</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>GG</p></td><td class="cell"><p>55.03</p></td><td class="cell"><p>57.12</p></td><td class="cell"><p>56.06</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>TnT</p></td><td class="cell"><p>61.21</p></td><td class="cell"><p>49.17</p></td><td class="cell"><p>54.53</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>LA with the GG</p></td><td class="cell"><p>82.04</p></td><td class="cell"><p>86.5</p></td><td class="cell"><p>84.21</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 6: F-measures per POS for the LA model" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>POS</p></td><td class="cell"><p>nouns</p></td><td class="cell"><p>adj</p></td><td class="cell"><p>verbs</p></td><td class="cell"><p>adv</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>F-meas (%)</p></td><td class="cell"><p>92.4</p></td><td class="cell"><p>90.93</p></td><td class="cell"><p>67.25</p></td><td class="cell"><p>75.82</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 7: Coverage and accuracy results for FR.LB Accstands for labelled brackets accuracy." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Model</p></td><td class="cell"><p>Cov (%)</p></td><td class="cell"><p>Acc (%)</p></td><td class="cell"><p>LB Acc (%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>GG-standard</p></td><td class="cell"><p>12.67</p></td><td class="cell"><p>82.46</p></td><td class="cell"><p>92.87</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>GG+POS</p></td><td class="cell"><p>25.11</p></td><td class="cell"><p>78.76</p></td><td class="cell"><p>92.51</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>GG + LA</p></td><td class="cell"><p>26</p></td><td class="cell"><p>84.62</p></td><td class="cell"><p>94.71</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>The results differ from the ones reported on the FR corpus for the suppertagging methods (Table 1) where the application of the tagger increased coverage but the price was lower accuracy. We at­tribute this to the bigger amount offeatures used in the classifier in the C&amp;vN method which enabled the learning of more detailed lexical entries.</p><p>However, the estimation used up till now for accuracy is a rather crude one. Another way of looking at accuracy is to measure how close the top ranked parse for a given sentence is to the cor­rect parse produced for this sentence. We selected the 47 sentences for which all three models have produced the correct parse and used them as our gold standard, to be able to report accuracy num­bers, for the best parse.<footnote anchor="6"/> Accuracy is measured in terms of labelled brackets. The results given in Ta­ble 7 show that not only the LA model produces a correct parse for more sentences but also the top ranked analysis is of better quality.</p></section><section number="7" title="LA for Text Generation"><p>As a further evaluation, we also investigate how the lexical entries acquired with LA affect sen­tence realisation. While in parsing the ambiguity in such less constrained lexical entries dissolves quickly in its context, there is a potential risk of overgeneration in the reverse process.</p><p>We selected 14 words assigned verb types by the tagger in the experiment with the FR corpus. Then, 64 sentences, each of which contains one of those 14 words, are extracted from corpora. We construct manually another sentence set where these words are replaced by verbs from the GG lexicon with a similar lexical type. This com­parison set indicates what the performance of the GG would be with fully constrained, but otherwise similar lexical entries.</p><p>Realisation with the GG is performed within the LKB grammar engineering platform which pro­vides a chart-based generator with various optimi­sations for packed parse forest (Carroll and Oepen, 2005). We parsed the two sentence sets and then used the best analysis to generate a realisation for each sentence. There were 3.28 realisations per sentence for the test set versus 3.16 for the com­parison one. As for accuracy, a realisation is con­sidered correct if it is an exact match of the origi­nal sentence (excluding punctuation). Despite the higher number for realisations per sentence for the test set, the quality of the realisations is the same for both sets- for 60 sentences a correct realisation is produced. Thus, the entries acquired with LA can be employed for both parsing and realisation.</p><footnote label="6">That is why, the reported accuracy numbers are rather high.</footnote><p>We should note that realisation with the generic lexical entries acquired by the unknown word guesser of the parser is computationally not fea­sible because of the many underspecified features in those entries. Nicholson et al. (2008) and Cholakov et al. (2008) did not perform any experi­ments on realisation but we assume that the under-specification in the entries those methods produce would also make sentence realisation practically impossible.</p></section><section number="8" title="Conclusion"><p>Two types of methods for dealing with unknown words were compared. The application of meth­ods, where a tagger was used to predict lexical de­scriptions for words unknown to the GG grammar of German, led to an increase in parsing coverage on a German newspaper corpus. However, accu­racy was below the baseline, that is the accuracy of the standard GG setup. The other type of meth­ods employ LA techniques to extend the lexicon of a grammar with new lexical entries for unknown words. We adapted one such method, namely the one of Cholakov and van Noord (2010), which has been primarily developed for the Dutch Alpino grammar, to be used with the GG. The applica­tion of the method led to an increase in both pars­ing coverage and accuracy on the same newspa­per corpus. We attributed the better performance of the C&amp;vN technique to the fact that it had ac­cess to more features during prediction, including such which came directly from the grammar, and it was able to assign more linguistically accurate entries to the unknown words. Last, in a smaller experiment, we showed that those entries can be successfully used for sentence realisation.</p></section><references><p>Tim Baldwin. 2005. Bootstrapping deep lexical re­sources: Resources for courses. In <i>Proceedings of</i> <i>the ACL-SIGLEX 2005 Workshop on Deep Lexical Acquisition, </i>Ann Arbor, USA.<page local="9" global="775"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">114</doubt><p>Srinivas Bangalore and Aravind Joshi. 1999. Sup-pertagging: An approach to almost parsing. In <i>Com­putational Linguistics, </i>volume 25(2), pages 237-65.</p><p>Petra Barg and Markus Walther. 1998. Processing un­known words in HPSG. In <i>Proceedings of the 36th Conference of the ACL, </i>Montreal, Quebec, Canada.</p><p>Phil Blunsom and Timothy Baldwin. 2006. Multi­lingual deep lexical acquisition for HPSGs via su-pertagging. In <i>Proceedings ofEMNLP 2006, </i>Sid­ney, Australia.</p><p>Thorsten Brants. 2000. TnT- a statistical part-of-speech tagger. In <i>Proceedings ofthe Sixth Con­ference on Applied Natural Language Processing</i> <i>ANLP-2000, </i>Seattle, WA.</p><p>Ulrich Callmeier. 2000. PET- a platform for ex­perimentation with efficient HPSG processing tech­niques. In <i>Journal ofNatural Language Engineer­ing, </i>volume 6, pages 99-107. Cambridge University Press.</p><p>John Carroll and Stephan Oepen. 2005. High ef­ficiency realization for a wide-coverage unification grammar. In <i>Proceedings ofthe 2nd International Joint Conference on Natural Language Processing (IJCNLP 2005), </i>pages 165-176, Jeju Island, Korea.</p><p>Kostadin Cholakov and Gertjan van Noord. 2009. Combining finite state and corpus-based techniques for unknown word prediction. In <i>Proceedings ofthe 7th Recent Advances in Natural Language Process­ing (RANLP) conference, </i>Borovets, Bulgaria.</p><p>Kostadin Cholakov and Gertjan van Noord. 2010. Acquisition of unknown word paradigms for large-scale grammars. In <i>Proceedings ofthe 23rd Inter­national Conference on Computational Linguistics (COLING-2010), </i>Beijing, China.</p><p>Kostadin Cholakov, Valia Kordoni, and Yi Zhang. 2008. Towards domain-independent deep linguistic processing: Ensuring portability and re-usability of lexicalised grammars. In <i>Proceedings ofCOLING 2008 Workshop on Grammar Engineering Across Frameworks (GEAF08), </i>Manchester, UK.</p><p>Stephen Clark and James Curran. 2004. The im­portance of supertagging for wide-coverage CCG parsing. In <i>Proceedings ofthe 20th International Conference on Computational Linguistics (COL-ING'2004), </i>Geneva, Switzerland.</p><p>Stephen Clark. 2002. Supertagging for combinatory categorical grammar. In <i>Proceedings ofthe 6th In­ternational Workshop on Tree Adjoining Grammar and Related Frameworks, </i>Venice, Italy.</p><p>Ann Copestake and Dan Flickinger. 2000. An open-source grammar development environment and broad-coverage English grammar using HPSG. In <i>Proceedings ofLREC 2000, </i>Athens, Greece.</p><p>Berthold Crysmann. 2003. On the efficient implemen­tation of German verb placement in HPSG. In <i>Pro­ceedings ofRANLP 2003, </i>Borovets, Bulgaria.</p><p>James Cussens and Stephen Pulman. 2000. Incorpo­rating linguistic constraints into inductive logic pro­gramming. In <i>Proceedings ofthe Fourth Conference on Computational Natural Language Learning.</i></p><p>Rebecca Dridan, Valia Kordoni, and Jeremy Nichol­son. 2008. Enhancing perfomrmance of lexi-calised grammars. In <i>Proceedings of ACL-08: HLT, </i>Columbus, Ohio.</p><p>Gregor Erbach. 1990. Syntactic processing of un­known words. IWBS report 131. Technical report, IBM, Stuttgart.</p><p>Frederik Fouvry. 2003. Lexicon acquisition with a large-coverage unification-based grammar. In <i>Com­panion to the 10th Conference ofEACL, </i>pages 87­90, Budapest, Hungary.</p><p>Adam Kilgarriff and G Grefenstette. 2003. Introduc­tion to the special issue on the web as a corpus. In <i>Computational Linguistics, </i>volume 29, pages 333­347.</p><p>Robert Malouf. 2002. A comparison of algorithms for maximum entropy parameter estimation. In <i>Pro­ceedings ofthe 6th conference on Natural Language Learning (CoNLL-2002)</i>, pages 49-55, Taipei, Tai­wan.</p><p>Jeremy Nicholson, Valia Kordoni, Yi Zhang, Timothy Baldwin, and Rebecca Dridan. 2008. Evaluating and extending the coverage of HPSG grammars: A case study for German. In <i>Proceedings ofLREC-2008, </i>Marakesh, Morocco.</p><p>Helmut Schmid. 1994. Probabilistic part-of-speech tagging using decision trees. In <i>Proceedings ofthe International Conference on New Methods in Lan­guage Processing, </i>Manchester, UK.</p><p>Melanie Siegel and Emily Bender. 2002. Efficient deep processing of Japanese. In <i>Proceedings ofthe 3rd Workshopon Asian Language Resources andIn-ternational Standardization, </i>Taipei, Taiwan.</p><p>Gertjan van Noord. 2006. At last parsing is now oper­ational. In <i>Proceedings ofTALN, </i>Leuven, Belgium.</p><p>Yi Zhang and Valia Kordoni. 2006. Automated deep lexical acquisition for robust open text processing.</p><p>In <i>Proceedings ofthe Fifth International Confer­ence on Language Resourses and Evaluation (LREC 2006), </i>Genoa, Italy.</p><p>Yaozhong Zhang, Takuya Matsuzaki, and Jun'ichi Tsu-jii. 2010. A simple approach for HPSG supertag-ging using dependency information. In <i>Proceedings of11th Annual Conference ofthe North American Chapter ofthe Association for Computational Lin­guistics (NAACL-HLT'10)</i>, Los Angeles, CA.</p></references></body></article>