<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="363"/><title>Integration of Data from a Syntactic Lexicon into Generative and Discriminative Probabilistic Parsers</title><pubinfo>Proceedings of Recent Advances in Natural Language Processing,pages 363-370, Hissar, Bulgaria, 12-14 September 2011.</pubinfo><author surname="Sigogne" givenname="Anthony"><org  name="Université Paris-Est"/></author><author surname="Constant" givenname="Matthieu"><org  name="LIGM    Université Paris-Est"/></author><author surname="Laporte" givenname="Éric"><org  name="LIGM    Université Paris-Est"/></author></firstpageheader><frontmatter><p><b>Integration of Data from a Syntactic Lexicon into Generative and Discriminative Probabilistic Parsers</b></p><p><b>Anthony Sigogne Matthieu Constant Eric Laporte</b></p><p>Université Paris-Est, LIGM    Université Paris-Est, LIGM    Université Paris-Est, LIGM</p><p>sigogne@univ-mlv.fr mconstan@univ-mlv.fr laporte@univ-mlv.fr</p></frontmatter><abstract>This article evaluates the integration of data extracted from a syntactic lexi­con, namely the Lexicon-Grammar, into several probabilistic parsers for French. We show that by modifying the Part-of-Speech tags of verbs and verbal nouns of a treebank, we obtain accurate perfor­mances with a parser based on Probabilis­tic Context-Free Grammars (Petrov et al., 2006) and a discriminative parser based on a reranking algorithm (Charniak and John­son, 2005). </abstract></header><body><section number="1" title="Introduction"><p>Syntactic lexicons are rich language resources that may contain useful data for parsers like sub-categorisation frames, as they provide, for each lexical entry, information about its syntactic be­haviors. Most of the time, these lexicons only deal with verbs. Few, like the Lexicon-Grammar (Gross, 1994), deal with other categories like nouns, adjectives or adverbs. Many works on symbolic parsing studied the use of a syntactic lexicon, in particular linguistic formalisms like Lexical-Functional Grammars [LFG] (Kaplan and Maxwell, 1994; Riezler et al., 2002; Sagot, 2006) or Tree Adjoining Grammars [TAG] (Joshi, 1987; Sagot and Tolone, 2009; de La Clergerie, 2010). For probabilistic parsing, we can cite LFG (Cahill, 2004; O'Donovan et al., 2005; Schlüter and Gen­abith, 2008), Head-Driven Phrase Structure Gram­mar [HPSG] (Carroll and Fang, 2004) and Proba­bilistic Context-Free Grammars [PCFG] (Briscoe and Carroll, 1997; Deoskar, 2008). The latter has incorporated valence features to PCFGs and lex­icons and observes slight improvements on per­formances. However, lexical resources that con­tain valence features were obtained automatically from a corpus. Furthermore, valence features are mainly used on verbs. In this paper, we will show how we can exploit information contained in the Lexicon-Grammar in order to improve probabilis­tic parsers. We will in particular focus on verbs and verbal nouns<footnote anchor="1"/>.</p><p>In section 2, we describe the probabilistic parsers used in our experiments. Section 3 briefly intro­duces the Lexicon-Grammar. We detail informa­tion contained in this lexicon that can be used for parsing. Then, in section 4, we present methods to integrate this information into parsers and, in sec­tion 5, we describe our experiments and discuss the obtained results.</p></section><section number="2" title="Statistical parsers"><p>In our experiments, we used two types of parsers: a generative parser that generates the <i>n-best </i>parses (n most probable parses) for a sentence accord­ing to a PCFG; a reranker that reranks the <i>n-best </i>parses generated from the PCFG parser according to a discriminative probabilistic model.</p><subsection number="2.1" title="Non-lexicalized PCFG parser"><p>The PCFG parser, used into our experiments, is the Berkeley Parser (called BKY thereafter) (Petrov et al., 2006)<footnote anchor="2"/>. This parser is based on a non-lexicalized PCFG model. The main problem of non-lexicalized context-free grammars is that pre-terminal symbols encode too general information which weakly discriminates syntactic ambiguities. BKY tries to handle the problem by generating a grammar containing complex pre-terminals. It fol­lows the principle of latent annotations introduced by (Matsuzaki et al., 2005). It consists in creating iteratively several grammars, which have a tagset increasingly complex. For each iteration, a sym­bol of the grammar is splitted in several symbols according to the different syntactic behaviors of the symbol that occur in a treebank.<page local="2" global="364"/> Parameters of the latent grammar are estimated with an algo­rithm based on Expectation-Maximisation (EM). Within the framework of French, (Seddah et al., 2009) have shown that BKY produces <i>state-of-the-art </i>performances. They have also shown that several parsers, based on the lexicalized paradigm (phrasal nodes are annotated with their headword), achieved lower scores than BKY.</p><footnote label="1">Verbal nouns are nouns playing the role of a predicate in the sentence.</footnote><footnote label="2">The Berkeley parser is freely available at http://code.google.com/p/berkeleyparser/downloads/list</footnote></subsection><subsection number="2.2" title="Reranking parser"><p>We have also experimented the integration of a reranker as a post-process of BKY output. For a given sentence <i>s, </i>a reranker selects the best parse <i>y </i>among the set of candidates Y (s) according to a scoring function Ve :</p><doubt alpha="60.9" length="23" tooSmall="False" monospace="0.0">y*=argmaxyey{s)Ve(y)(1)</doubt><p>The set of candidates <i>Y</i><i> </i><i>(s)</i><i> </i>is the <i>n-best </i>parses output of the baseline parser (BKY in our case), Y (s) = <i>{y1,y2,</i><i> </i>•••,yn}. The <i>n-best </i>parses cor­respond to the n most probable parses according to the probability model of the parser. The scor­ing function Ve is defined by the dot product of a weight vector 6 and a feature vector f :</p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">m</doubt><doubt alpha="28.6" length="28" tooSmall="False" monospace="0.0">Ve(y) = 6.f (y) =]T6j(y) (2)</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">j=i</doubt><p>where the feature vector f (y) is a vector of m functions f = (f1; f2,fm), and each feature function <i>fj</i><i> </i>maps a parse y to a real number <i>fj</i><i> </i><i>(y)</i>. The first feature f1(y) is the probability of the parse given by the n-best parser (cf. (Charniak and Johnson, 2005)). All remaining features are integer values, and each of them is the number of times that the feature occurs in parse y. Fea­tures belong to feature schemas which are abstract schemas from which specific features are instan­tiated. Feature schemas that we used during our experiments are specified in the table 1. For ex­ample, a feature <i>f10(y),</i><i> </i>which is an instance of the feature schema <i>Rule, </i>counts the number of times that a nominal phrase in y is the head of a rule which has a determinant and a noun as children. The weight vector 6 can be estimated by a machine learning algorithm from a treebank corpus which contains the gold parse for each sentence. In our case, we will use the Maximum Entropy estimator, as in (Charniak and Johnson, 2005).</p><p>Table 1: Features used in this work. Those with a A are from (Collins, 2000), and others are from (Charniak and Johnson, 2005)</p></subsection></section><section number="3" title="Lexicon-Grammar"><p>The Lexicon-Grammar [LG] is the richest source of syntactic and lexical information for French<footnote anchor="3"/>that focuses not only on verbs but also on verbal nouns, adjectives, adverbs and frozen (or fixed) sentences. Its development started in the 70's by Maurice Gross and his team (Gross, 1994). It is a syntactic lexicon represented in the form of tables. Each table encodes lexical items of a particular category sharing several syntactic properties (e.g. subcategorization information). A lexical item is a lemmatized form that can be present in one or more tables depending on its meaning and its syn­tactic properties. Each table row corresponds to a lexical item and a column corresponds to a prop­erty (e.g. syntactic constructions, argument dis­tribution, and so on). A cell encodes whether a lexical item accepts a given property. Figure 1 shows a sample of verb table 12. In this table, we can see that the verb <i>chérir (to cherish) </i>ac­cepts a human subject (pointed out by a + in the property <i>NO =: Nhum) </i>but this verb cannot be in­transitive (pointed out by a — in the property <i>N0 V </i>). Recently, these tables have been made consistent and explicit (Tolone, 2011) in order to be exploitable for NLP.<page local="3" global="365"/> They also have been trans­formed in a XML-structured format (Constant and Tolone, 2008)<footnote anchor="4"/>. Each lexical entry is associated with its table identifier, its possible arguments and its syntactic constructions. For the verbs, we manually constructed a hierar­chy of the tables on several levels<footnote anchor="5"/>. Each level contains classes which group LG tables which may not share all their defining properties but have a relatively similar syntactic behavior. Figure 2 shows a sample of the hierarchy. The tables <i>4, 6 </i>and <i>12 </i>are grouped into a class called <i>QTD2 </i>(transitive sentence with two arguments and sen­tential complements). Then, this class is grouped with other classes at the superior level of the hi­erarchy to form a class called <i>TD2 </i>(transitive sen­tence with two arguments). The characteristics of each level are given in the table 2<footnote anchor="6"/> (level 0 repre­sents the set of tables of the LG). We can state that there are 5,923 distinct verbal forms for 13,862 resulting entries in tables of verbs. The column <i>#classes </i>specifies the number of distinct classes. The columns <i>AVG_1 </i>and <i>AVG_2 </i>respectively indi­cate the average number of entries per class and the average number of classes per distinct verbal form.</p><footnote label="3">We can also cite lexicons like LVF (Dubois and Dubois-Charlier, 1997), Dicovalence (Eynde and Piet, 2003) and Lefff (Sagot, 2010).</footnote><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Feature schemas</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Rule</p></td><td class="cell"><p>Edges</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Word</p></td><td class="cell"><p>WordEdges</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Heavy</p></td><td class="cell"><p>Heads</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>HeadTree</p></td><td class="cell"><p>WProj</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>BigramsA</p></td><td class="cell"><p>NgramTree</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>TrigramsA</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Figure 1: Sample of verb table12" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p><b>&lt;ENT&gt;V</b></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p><b>—</b></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b><i>S</i></b></p></td><td class="cell"><p></p></td><td class="cell"><p>Vim:</p></td><td class="cell"><p></p></td><td class="cell"><p><b>z</b></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>O</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>----</i></p></td><td class="cell"><p><i>-</i></p></td><td class="cell"><p><b>© z</b></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p><b><i>y.</i></b></p></td><td class="cell"><p></p></td><td class="cell"><p><b><i>V.</i></b></p></td><td class="cell"><p><b><i>V.</i></b></p></td><td class="cell"><p><i>i</i></p></td><td class="cell"><p>i</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p><b>■</b><b> </b><b>E-</b></p></td><td class="cell"><p></p></td><td class="cell"><p>chéri</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p><b>&lt;E&gt;</b></p></td><td class="cell"><p></p></td><td class="cell"><p>: * 1111 <b>t </b><b>i </b><b>jikIt-j</b></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>&lt;E&gt;</p></td><td class="cell"><p></p></td><td class="cell"><p>critiquer</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>:</p></td><td class="cell"><p></p></td><td class="cell"><p>debitier</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><figure caption="Figure 2: Sample of the hierarchy of verb tables"></figure><p>The hierarchy of tables have the advantage of re­ducing the number of classes associated with each verb of the tables. We will see that this ambiguity reduction is crucial in our experiments.</p><footnote label="4">These resources are freely available at http://infolingu.univ-mlv.fr&gt;Language_Resources &gt; Lexi-conJGrammar &gt; Download</footnote><footnote label="5">The hierarchy of verb tables is available at : http://igm.univ-mlv.fr/~sigogne/arbre-tables.xlsx</footnote><footnote label="6">We can also state that 3,121 verb forms (3,195 entries) are unambiguous. This means that all their entries occur in a single table.</footnote></section><section number="4" title="Exploitation of the Lexicon-Grammar data"><p>Many experiments about parsing, within the framework of French (Crabbe and Candito, 2008; Seddah et al., 2009), have shown that refining the tagset of the training corpus improves performances of the parser. We will follow their works by integrating information from the Lexicon-Grammar to part-of-speech tags. In this article, we will only focus on tables of verbs and verbal nouns.</p><p>Table identifiers of the lexical entries are im­portant hints about their syntactic behaviors. For example, the table <i>31R </i>indicates that all verbs belonging to this table are intransitive. The first experiment, called <b>AnnotTable, </b>consists in augmenting the part-of-speech tag with the table identifier(s) associated with the noun or the verb. For example, the verb <i>chérir (to cherish) </i>belongs to the table <i>12. </i>Therefore, the induced tag is <i>#tag-12, </i>where <i>#tag </i>is the POS tag associated with the verb. For an ambiguous verb like <i>sanc­tionner </i><i>(to</i><i> punish), </i>belonging to two tables <i>6 </i>and 12, the induced tag is <i>#tagJ6J2.</i></p><p>Then, in the case of verbs, we have done variants of the previous experiment by taking the hierarchy of verb tables into account. This hierarchy provides a tagset with a size which varies according to the level in the hierarchy. Identifiers added to tags depend on the verb and the specific level in the hierarchy. For example, the verb <i>sanctionner, </i>belonging to tables <i>6 </i>and 12, has a tag <i>#tag-QTD2 </i>at level 1. In the case of ambiguous verbs, for a given level in the hierarchy, suffixes contain all classes the verb belongs to. This experiment will be called <b>AnnotVerbs </b>thereafter. In the case of verbal nouns, as such a hierarchy of tables does not exist, we experimented two other methods. The first one, called <b>AnnotIN, </b>consists in adding a suffix <i>IN </i>to the tag of a noun if this noun occurs in the syntactic lexicon, and therefore if it is a verbal noun. The second method, called <b>AnnotNouns, </b>consists in creating a hierarchy of noun tables from the table of classes of verbal nouns. This hierarchy is made accordingly to the maximum number of arguments that a noun of a table can have according to defining properties specified for this table.<page local="4" global="366"/> As a consequence, the hierarchy has a single level. For example, nouns of the table <i>N-aa </i>can have at most 2 arguments contrary to those of table <i>N.an04 </i>which can have only one. The characteristics of each level are specified in table 3<footnote anchor="7"/> (level 0 represents the set of tables of the Lexicon-Grammar). We can state that there are 8,531 distinct nominal forms for 12,351 resulting entries in tables of nouns.</p><table caption="Table 2: Characteristics of the hierarchy of verb tables" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Level</p></td><td class="cell"><p>#classes</p></td><td class="cell"><p>AVG_1</p></td><td class="cell"><p>AVGJ2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>0</p></td><td class="cell"><p>67</p></td><td class="cell"><p>207</p></td><td class="cell"><p>2.15</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p>13</p></td><td class="cell"><p>1,066</p></td><td class="cell"><p>1.82</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2</p></td><td class="cell"><p>10</p></td><td class="cell"><p>1,386</p></td><td class="cell"><p>1.75</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3</p></td><td class="cell"><p>4</p></td><td class="cell"><p>3,465</p></td><td class="cell"><p>1.44</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><section number="5" title="Experimental setup"><p>For our experiments, we used the richest tree-bank for French, the French Treebank, (later called FTB) (Abeille et al., 2003), containing 20,860 sen­tences and 540,648 words from the newspaper <i>Le Monde </i>(version of 2004). As this corpus is small, we used a <i>cross-validation </i>procedure for the eval­uation. This method consists in splitting the cor­pus into <i>p </i>equal parts, then we compute training on <i>p-1 </i>parts and evaluations on the remaining part. We can iterate this process <i>p </i>times. This allows us to calculate an average score for a sample as large as the initial corpus. In our case, we set the param­eter <i>p </i>to 10. We also used the part-of-speech tagset defined in (Crabbe and Candito, 2008) containing 28 different tags describing some complementary morphological and syntactic features (e.g. verb mood, clitics, ...)<footnote anchor="8"/>. Compound words have been merged in order to obtain a single token.</p><p>In the following experiments, we will test the im­pact of modifying the tagset of the training cor­pus, namely the addition of information from the Lexicon-Grammar described in the section 4. Re­sults on evaluation parts are reported using the standard protocol called PARSEVAL (Black et al., 1991) for sentences smaller than 40 words. The score f-measure (F1) takes into account the brack­eting and categories of nodes (including punctuation nodes). For each experiment, we have re­ported the <i>Baseline </i>results (i.e. the results of BKY trained on the original treebank without an­notations from the Lexicon-Grammar). We have also indicated the percentage of distinct annotated verbs and verbal nouns in the entire corpus for each annotation method<footnote anchor="9"/>.</p><footnote label="7">The number of non-ambiguous nouns is 6126 for 6175 entries.</footnote><footnote label="8">There are 6 distinct tags for verbs and 2 distinct tags for nouns.</footnote><subsection number="5.1" title="Annotation of verb tags"><p>We first conducted experiments on verbs described in section 4, namely <i>AnnotTable </i>and <i>AnnotVerbs. </i>The experimental results are shown in the table 4. In the case of the method <i>AnnotVerbs, </i>we varied two parameters, <i>Lvl </i>(for Level) indicating the level of the hierarchy used and <i>Amb. </i>(for Ambiguity) indicating that a tag of a verb is changed only if this verb belongs to a number of classes less than or equal to the number specified by this parameter.</p><doubt alpha="100.0" length="6" tooSmall="False" monospace="0.0">Method</doubt><p>Lvl/Amb. F1/Tagging  Absolute gains (F1)</p><doubt alpha="66.7" length="12" tooSmall="False" monospace="0.0">Baseline -/-</doubt><p>AnnotTable -/1</p><p>AnnotVerbs 1/1</p><p>AnnotVerbs 2/1</p><p>AnnotVerbs 3/1</p><p>AnnotVerbs 2/2</p><p>AnnotVerbs 3/2</p><doubt alpha="0.0" length="11" tooSmall="False" monospace="0.0">85.05/97.43</doubt><doubt alpha="0.0" length="11" tooSmall="False" monospace="0.0">84.49/97.29</doubt><doubt alpha="0.0" length="23" tooSmall="False" monospace="0.0">85.06/97.46 85.35/97.41</doubt><doubt alpha="0.0" length="11" tooSmall="False" monospace="0.0">85.39/97.49</doubt><doubt alpha="0.0" length="23" tooSmall="False" monospace="0.0">84.60/97.35 85.20/97.48</doubt><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">-0.5</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">0.0</doubt><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">+0.5</doubt><table caption="Table 4: Results from cross-validation evaluation according to verb annotation methods"></table><p>For non-ambiguous verbs, we observe that the ex­periment <i>AnnotTable </i>highly deteriorates perfor­mances.   This comes most probably from the grammar which is too fragmented because of the significant size of the part-of-speech tagset (as shown in table 5).<page local="5" global="367"/> However, the effect is reversed as soon as we use levels of the hierarchy of tables (levels 2 and 3 only). The use of the table hierar­chy causes the increase of the number of verbs an­notated as non-ambiguous and the decrease of the size of the tagset. Considering ambiguous verbs do not improve performances (results are shown only for levels 2 and 3 with maximal ambiguity of 2) because of the large size of the tagset (as for ex­periment <i>AnnotTable).</i></p><footnote label="9">The corpus contains 3058 distinct verbal forms and 17003 distinct nominal forms.</footnote><table caption="Table 3: Characteristics of the hierarchy of noun tables" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Level</p></td><td class="cell"><p>#classes</p></td><td class="cell"><p>AVG_1</p></td><td class="cell"><p>AVG_2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>0</p></td><td class="cell"><p>76</p></td><td class="cell"><p>162</p></td><td class="cell"><p>1.43</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p>3</p></td><td class="cell"><p>3,413</p></td><td class="cell"><p>1.2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 5: Size of tagset and percentage of annotated verbs according to verb annotation methods" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Method</p></td><td class="cell"><p>Size of tagset</p></td><td class="cell"><p>% annotated verbs</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Baseline</p></td><td class="cell"><p>28</p></td><td class="cell"><p>-</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>AnnotTable</p></td><td class="cell"><p>228</p></td><td class="cell"><p>18,6%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>AnnotVerbs 1/1</p></td><td class="cell"><p>89</p></td><td class="cell"><p>21,5%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>AnnotVerbs 2/1</p></td><td class="cell"><p>76</p></td><td class="cell"><p>22,5%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>AnnotVerbs 3/1</p></td><td class="cell"><p>47</p></td><td class="cell"><p>33,9%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>AnnotVerbs 2/2</p></td><td class="cell"><p>246</p></td><td class="cell"><p>44,7%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>AnnotVerbs 3/2</p></td><td class="cell"><p>75</p></td><td class="cell"><p>55,7%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><doubt alpha="100.0" length="6" tooSmall="False" monospace="0.0">Method</doubt><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">-0.5</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">0.0</doubt><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">+0.5</doubt><p>Figure 3: Absolute gains (F1) of verb annotation methods on evaluation parts (baseline is the hori­zontal line at 0 on y axis)</p><p>We can see on Figure 3 absolute gains according to verb annotation methods on evaluation parts. We have displayed curves for methods <i>AnnotTable </i>and <i>AnnotVerbsX, </i>where X is the level in the hi­erarchy (without ambiguity). Higher we are in the hierarchy of tables, the more we obtain better per­formances. Levels 2 and 3 are globally above the baseline for most of their evaluation parts. There­fore, this would mean that table identifiers of verbs and the hierarchy are a real help for parsing and do not produce a random effect. On table 6, we can see the top most error reductions according to phrase label, for the best verb annotation method <i>(AnnotVerbs </i>with level 3 of the hierarchy). For each phrase, the column called <i>Error reduction</i> indicates the average error reduction rate associ­ated with the corresponding average number of er­ror corrected (inside brackets). The NP and PP phrases are those that have the highest number of errors corrected (the low reduction rate can be ex­plained by the fact that these two phrases have the highest number of errors). Furthermore, they are linked to each other because, generally, a PP has a NP kernel. Therefore, if a NP is corrected, the corresponding PP is also corrected (if it is the only error).</p></subsection><subsection number="5.2" title="Annotation of noun tags"><p>For verbal nouns, we successively conducted sev­eral experiments <i>AnnotTable, AnnotNouns </i>and <i>An-notlN, </i>described in section 4. Results are given in table 7. As for verbs, we have reported the re­sults for the experiment <i>AnnotNouns </i>with respect to the parameter <i>Ambiguity </i>(the maximum number of classes being associated with a noun is 3).</p><p>Amb. F1/Tagging  Absolute gains (F1)</p><p>Baseline -</p><p>AnnotTable 1</p><p>AnnotNouns 1</p><p>AnnotNouns 2</p><p>AnnotNouns 3</p><p>AnnotIN -</p><doubt alpha="0.0" length="47" tooSmall="False" monospace="0.0">85.05/97.43 85.10/97.42 85.13/97.48 85.16/97.47</doubt><doubt alpha="0.0" length="11" tooSmall="False" monospace="0.0">85.05/97.41</doubt><doubt alpha="0.0" length="11" tooSmall="False" monospace="0.0">85.20/97.54</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">P P</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">P</doubt><table caption="Table 7: Results from cross-validation evaluation according to noun annotation methods"></table><p>Table 8: Size of tagset and percentage of anno­tated verbal nouns according to noun annotation methods</p><p>The various noun annotation methods slightly in­crease performances of the parser. Unlike verbs, the method <i>AnnotTable </i>does not degrade perfor­mances because there are much less nouns in the corpus belonging to the syntactic lexicon (less than 9% as shown in table 8), hence the limited<page local="6" global="368"/></p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Method</p></td><td class="cell"><p>Size of tagset</p></td><td class="cell"><p>% annotated nouns</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Baseline</p></td><td class="cell"><p>28</p></td><td class="cell"><p>-</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>AnnotTable</p></td><td class="cell"><p>98</p></td><td class="cell"><p>8,6%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>AnnotNouns 1</p></td><td class="cell"><p>33</p></td><td class="cell"><p>11,2%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>AnnotNouns 2</p></td><td class="cell"><p>38</p></td><td class="cell"><p>16,5%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>AnnotNouns 3</p></td><td class="cell"><p>39</p></td><td class="cell"><p>16,9%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>AnnotIN</p></td><td class="cell"><p>30</p></td><td class="cell"><p>16,9%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 6: Top most error reductions according to phrase label" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Phrase label</p></td><td class="cell"><p>Meaning</p></td><td class="cell"><p>Error reduction</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Ssub</p></td><td class="cell"><p>subordinate clause</p></td><td class="cell"><p>5,3% (52)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Sint</p></td><td class="cell"><p>internal clause</p></td><td class="cell"><p>3,6% (47)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PP</p></td><td class="cell"><p>prepositional phrase</p></td><td class="cell"><p>3,1% (272)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Srel</p></td><td class="cell"><p>relative clause</p></td><td class="cell"><p>2,2% (17)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NP</p></td><td class="cell"><p>nominal phrase</p></td><td class="cell"><p>2,1% (347)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>VPinf</p></td><td class="cell"><p>infinitive phrase</p></td><td class="cell"><p>2,1% (34)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Figure 4: Absolute gains (F1) of noun annotation methods on evaluation parts (baseline is the hori­zontal line at 0 on y axis) impact of the new tagset. The use of a simple hierarchy of the noun tables, through experiment <i>AnnotNouns, </i>achieves positive gains but, here, in-signiicant. Moreover, we obtain a slight improve­ment by annotating some ambiguous nouns. Sur­prisingly, the method which gives the best result, despite its simplicity, is <i>AnnotlN. </i>We can see in</p><p><b>evaluation parts</b></p><p>Figure 4 absolute gains according to noun annota­tion methods on all evaluation parts. Unlike verbs, absolute gains are closer to the baseline. The best method <i>AnnotlN </i>is able to improve significantly 4 of 10 evaluation parts (+0,4 to +0,8).</p></subsection><subsection number="5.3" title="Combination of annotations"><p>In a inal experiment with BKY, we combined the best methods of verb and verbal noun annotations, that are <i>AnnotlN </i>for verbal nouns and <i>AnnotVerbs </i>for verbs (level 3 without ambiguity). Results are shown in table 9.</p><p>Combination of annotations does not increase the gains obtained with the method <i>AnnotVerbs </i>and we even observe a slight decrease.</p></subsection><subsection number="5.4" title="Impact on a reranker"><p>We also experimented the integration of a dis­criminative reranker (cf. section 2). We prac­tically set to 10 the number of parses gener­ated by BKY for each sentence (therefore, the 10 most probable parses). The following exper­iment consists in evaluating the impact of the modiication of the tagset on a reranker. We called <i>Reranker(Baseline) </i>the experiment using the reranker with BKY trained on the original corpus (without annotations from the Lexicon-Grammar). <i>Reranker(AnnotVerbs) </i>is the experi­ment based on BKY that is trained on the corpus annotated by the best verb annotation method, <i>An-notVerbs </i>(level 3 of hierarchy without ambiguity). Results are shown in table 10. The column named <i>Oracle F1/Tagging </i>indicates oracle scores for f-measure and tagging accuracy. An oracle score is the best global score that we could obtain whether we choose, for each input sentence, the best parse from the n-best parses. With this score, we can estimate the performance limit of a parser and the global quality of parses generated.</p><p>First, we can see that <i>Reranker(Baseline) </i>im­proves performances with an absolute gain of +1,46 as compared with the baseline. These re­sults are comparable to scores obtained for En­glish (Charniak and Johnson, 2005). Then, we ob­serve that the experiment <i>Reranker(AnnotVerbs) </i>increases the f-measure by +0,2 compared with <i>Reranker(Baseline) </i>(and to a lesser extent, the tagging accuracy by +0,07). The power of the discriminative model of the reranker implies that the gap of performances between the two experi­ments based on the reranker is less than the one obtained from experiments only based on BKY (+0,2 against +0,34). In addition, the oracle f-measure is improved (+0,27), which means that analyses generated by BKY are slightly better. We can see on Figure 5 absolute gains given by the reranker on all evaluation parts according to the two methods described above. Globally, the method <i>Reranker(AnnotVerbs) </i>has a curve slightly above the one of <i>Reranker(Baseline). </i>Note that the irst one outperforms the latter on 8 of 10 eval­uation parts. All these observations conirm that the syntactic lexicon through the experiment <i>An-notVerbs </i>is able to improve performances on both<page local="7" global="369"/></p><table caption="Table 10: Results from cross-validation evaluation for reranking process." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Method</p></td><td class="cell"><p>F1/Tagging</p></td><td class="cell"><p>Oracle F1/Tagging</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>BKY(Baseline)</p></td><td class="cell"><p>85.05/97.43</p></td><td class="cell"><p>-</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>BKY(AnnotVerbs)</p></td><td class="cell"><p>85.39/97.49</p></td><td class="cell"><p>-</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Reranker(Baseline)</p></td><td class="cell"><p>86.51/97.42</p></td><td class="cell"><p>91,72/98.03</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Reranker(AnnotVerbs)</p></td><td class="cell"><p><b>86.71/97.49</b></p></td><td class="cell"><p><b>91.99/98.08</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 9: Results from cross-validation evaluation according to combination of annotations" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Method</p></td><td class="cell"><p>F1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Baseline</p></td><td class="cell"><p>85.05</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Combination</p></td><td class="cell"><p><b>85.32</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Figure 5: Absolute gains (F1) given by the reranker on evaluation parts (BKY(baseline) is the horizontal line at 0 on y axis) a generative parser based on a PCFG grammar (BKY), and a discriminative parser (reranker).</p></subsection></section><section number="6" title="Conclusions"><p>The work described in this paper shows that by adding some information from a syntactic lexicon like the Lexicon-Grammar, we are able to improve performances of several probabilistic parsers. These performances are mainly obtained thanks to a hierarchy of verb tables that can limit ambiguity in terms of number of classes associ­ated with a verb. This has the effect of increas­ing the coverage of verbs annotated according to the level of granularity used. However, once we include some ambiguity, performances drop. Re­sults obtained on verbal nouns with a simple hier­archy of tables are insigniicant but suggest a de­gree of progress with a more complex hierarchy as the one available for verbs. In the near future, we plan to reproduce these ex­periments by taking into account of word clus­tering methods introduced by (Koo et al., 2008; Candito and Crabbe, 2009; Candito and Seddah, 2010). Thanks to a semi-supervized algorithm, these methods can reduce the size of the lexi­con of the grammar by grouping words according to their behaviors in a treebank. These methods could be complementary to annotation methods described in this paper. Moreover, we plan to ex­ploit the LFG formalism in order to use a syntac­tic lexicon more easily than for PCFGs, as many works have reported performance improvements for these models (Cahill, 2004; Deoskar, 2008).</p></section><references><p>A. Abeille, L. Clement, andF. Toussenel. 2003. Build­ing a treebank for French. In Anne Abeille, editor, <i>Treebanks, </i>Kluwer, Dordrecht.</p><doubt alpha="65.4" length="52" tooSmall="False" monospace="0.0">E. Black, S.Abney, D. Flickinger, C. Gdaniec, R. Gr-</doubt><p>ishman, P. Harrison, D. Hindle, R. Ingria, F. Jelinek, J. Klavans, M. Liberman, M. Marcus, S. Roukos, B. Santorini, and T. Strzalkowski. 1991. A proce­dure for quantitatively comparing the syntactic cov­erage of english grammars. In <i>Proceedings of the DARPA Speech and Naturale Language Workshop, </i>pages 306-311.</p><p>T. Briscoe and J. Carroll. 1997. Automatic extraction of subcategorization from corpora. In <i>Fifth Con­ference on Applied Natural Language Processing, </i>pages 356-363, Washington DC, USA.</p><p>A. Cahill. 2004. <i>Parsing with Automatically Ac­quired, Wide-Coverage, Robust, Probabilistic LFG Approximations. </i>Ph.D. thesis, Dublin City Univer­sity, Dublin 9.</p><p>Marie Candito and B. Crabbe. 2009. Improving generative statistical parsing with semi-supervised word clustering. In <i>Proceedings of the 11th International Conference on Parsing Technology (IWPT'09), </i>pages 138-141.</p><p>Marie Candito and D. Seddah. 2010. Parsing word clusters. In <i>Proceedings of the first NAACL HLT Workshop on Morphologically-Rich Languages (SPRML2010), </i>pages 76-84, Los Angeles, Califor­nia.</p><p>J. Carroll and A. C. Fang. 2004. The automatic ac­quisition of verb subcategorisations and their impact on the performance of an HPSG parser. In <i>Proceed­ings of the 1st International Conference onNatural Language Processing, </i>Sanya City, China.</p><p>E. Charniak and M. Johnson. 2005. Coarse-to-fine n-best parsing and MaxEnt discriminative reranking.</p><p>In <i>Proceedings of the 43th Annual Meeting of the Association for Computational Linguistics (ACL05).</i></p><p>M. Collins. 2000. Discriminative reranking for natural language parsing. In <i>Proceedings of the 17th ICML, </i>pages 175-182.</p><p>M. Constant and E. Tolone. 2008. A generic tool to generate a lexicon for NLP from Lexicon-Grammar tables. In <i>Actes du 27eme Colloque Lexique et Grammaire, </i>L'Aquila, Italie.</p><p>B. Crabbe and Marie Candito. 2008. Experiences d'analyse syntaxique statistique du francais. In <i>Actes de la 15eme Conférence sur le Traitement Au­tomatique des Langues Naturelles (TALN'08), </i>pages 45-54, Avignon, France.</p><p>E. de La Clergerie. 2010. Building factorized TAGs with meta-grammars. In <i>Proceedings </i><i>ofthe</i><i> 10th International Conference on Tree Adjoining Gram­mars and Related Formalisms, </i>pages 111-118.</p><page local="8" global="370"/><p>T. Deoskar. 2008. Re-estimation of lexical param­eters for treebank PCFGs. In <i>Proceedings ofthe 22nd International Conference on Computational Linguistics (Coling 2008), </i>pages 193-200, Manch­ester, Great Britain.</p><p>J. Dubois and F. Dubois-Charlier. 1997. <i>Les verbes francais. </i>Larousse-Bordas.</p><p>K. Eynde and M. Piet. 2003. La valence: l'approche pronominale et son application au lexique verbal. <i>Journal of French Language studies, </i>pages 63-104.</p><p>M. Gross. 1994. Constructing Lexicon-grammars. In Atkins and Zampolli, editors, <i>Computational Ap­proaches to the Lexicon, </i>pages 213-263.</p><p>A. K. Joshi. 1987. An introduction to tree adjoin­ing grammars. In Alexis Manaster-Ramer, editor, <i>Mathematics of Language, </i>pages 87-115, Amster­dam/Philadelphia. John Benjamins Publishing Co.</p><p>R. Kaplan and J. Maxwell. 1994. <i>Grammar writer's workbench, version 2.0. </i>Rapport technique, Xerox Corporation.</p><doubt alpha="61.2" length="49" tooSmall="False" monospace="0.0">T. Koo, X. Carreras, and M. Collins. 2008. Simple</doubt><p>semi-supervised dependency parsing. In <i>Proceed­ings ofACL-08.</i></p><p>T. Matsuzaki, Y. Miyao, and J. Tsujii. 2005. Proba­bilistic cfg with latent annotations. In <i>Proceedings ofACL-05, </i>pages 75-82, Ann Arbor, USA.</p><p>R. O'Donovan, A. Cahill, A. Way, M. Burke, and J. van Genabith. 2005. Large-scale induction and evalua­tion of lexical resources from the Penn-II and Penn­III treebanks. In <i>Proceedings ofthe First Interna­tional Joint Conference on Natural Language Pro­cessing (IJCNLP 04).</i></p><p>S. Petrov, L. Barrett, R. Thibaux, and D. Klein. 2006.</p><p>Learning accurate, compact, and interpretable tree annotation. In <i>Proceedings ofthe 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computa­tional Linguistics, </i>Sydney, Australia.</p><p>S. Riezler, T. King, R. Kaplan, R. Crouch, J. Maxwell, and M. Johnson. 2002. Parsing the Wall Street Jour­nal using a lexical-functional grammar and discrim­inative estimation techniques. In <i>Proceedings ofthe Annual Meeting ofthe ACL, </i>University of Pennsyl­vania.</p><p>B. Sagot and E. Tolone. 2009. Integrer les ta­bles du Lexique-Grammaire a un analyseur syn­taxique robuste a grande echelle. In <i>Actes de la 15eme Conference sur le Traitement Automatique des Langues Naturelles (TALN 09), </i>Senlis, France.</p><p>B. Sagot. 2006. <i>Analyse automatique dufrancais: lex­iques, formalismes, analyseurs. </i>Ph.D. thesis, Uni­versite Paris VII.</p><p>B. Sagot. 2010. The lefff, a freely available, accurate and large-coverage lexicon for french. In <i>Proceed­ings ofLREC 2010, </i>La Valette, Malte.</p><p>N. Schluter and J. Van Genabith. 2008. Treebank-based Acquisition of LFG Parsing Resources for French. In <i>Proceedings of LREC08, </i>Marrakech, Morocco.</p><p>D. Seddah, Marie Candito, and B. Crabbe. 2009. Adaptation de parsers statistiques lexicalises pour le francais : Une evaluation complete sur corpus arbores. In <i>Actes de la 15eme Conference sur le Traitement Automatique des Langues Naturelles</i> <i>(TALN 09), </i>Senlis, France.</p><p>E. Tolone. 2011. <i>Analyse syntaxique a l'aide des ta­bles du Lexique-Grammaire dufrancais. </i>Ph.D. the­sis, Universite Paris-Est Marne-la-Vallee.</p></references></body></article>