<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="267"/><title>Exploiting Multi-Word Units in History-Based Probabilistic Generation</title><pubinfo>Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning,pp. 267-276, Prague, June 2007. ©2007 Association for Computational Linguistics</pubinfo><author surname="Hogan" givenname="Deirdre"><org  name="Dublin City University" country="Ireland" city="Dublin"/></author><author surname="Cafferkey" givenname="Conor"><org  name="Dublin City University" country="Ireland" city="Dublin"/></author><author surname="Cahill" givenname="Aoife"><org  name="Dublin City University" country="Ireland" city="Dublin"/></author><author surname="van Genabith" givenname="Josef"><org  name="Dublin City University" country="Ireland" city="Dublin"/></author></firstpageheader><frontmatter><p><b>Exploiting Multi-Word Units in History-Based Probabilistic Generation</b></p><p><b>Deirdre Hogan, Conor Cafferkey, Aoife Cahill* </b>and <b>Josef van Genabith</b></p><p>National Centre for Language Technology School of Computing, Dublin City University Dublin 9, Ireland dhogan,ccafferkey,josef@computing.dcu.ie</p></frontmatter><abstract>We present a simple history-based model for sentence generation from LFG f-structures, which improves on the accuracy of previous models by breaking down PCFG indepen­dence assumptions so that more f-structure conditioning context is used in the predic­tion of grammar rule expansions. In addi­tion, we present work on experiments with named entities and other multi-word units, showing a statistically significant improve­ment of generation accuracy. Tested on sec­tion 23 of the Penn Wall Street Journal Tree-bank, the techniques described in this paper improve BLEU scores from 66.52 to 68.82, and coverage from 98.18% to 99.96%. </abstract></header><body><section number="1" title="Introduction"><p>Sentence generation, or surface realisation, is the task of generating meaningful, grammatically cor­rect and fluent text from some abstract semantic or syntactic representation of the sentence. It is an im­portant and growing field of natural language pro­cessing with applications in areas such as transfer-based machine translation (Riezler and Maxwell, 2006) and sentence condensation (Riezler et al., 2003). While recent work on generation in restricted domains, such as (Belz, 2007), has shown promising results there remains much room for improvement particularly for broad coverage and robust genera­tors, like those of Nakanishi et al. (2005) and Cahill</p><p>* Now at the Institut für Maschinelle Sprachverarbeitung, Universität Stuttgart, Azenbergstrae 12, D-70174 Stuttgart, Germany. aoife.cahill@ims.uni-stuttgart.de</p><p>and van Genabith (2006), which do not rely on hand­crafted grammars and thus can easily be ported to new languages.</p><p>This paper is concerned with sentence genera­tion from Lexical-Functional Grammar (LFG) f-structures (Kaplan, 1995). We present improve­ments in previous LFG-based generation models firstly by breaking down PCFG independence as­sumptions so that more f-structure conditioning con­text is included when predicting grammar rule ex­pansions. This history-based approach has worked well in parsing (Collins, 1999; Charniak, 2000) and we show that it also improves PCFG-based genera­tion.</p><p>We also present work on utilising named entities and other multi-word units to improve generation results for both accuracy and coverage. There has been a limited amount of exploration into the use of multi-word units in probabilistic parsing, for ex­ample in (Kaplan and King, 2003) (LFG parsing) and (Nivre and Nilsson, 2004) (dependency pars­ing). We are not aware of any similar work on gen­eration. In the LFG-based generation algorithm pre­sented by Cahill and van Genabith (2006) complex named entities (i.e. those consisting of more than one word token) and other multi-word units can be fragmented in the surface realization. We show that the identification of such units may be used as a sim­ple measure to constrain the generation model's out­put.</p><p>We take the generator of (Cahill and van Gen­abith, 2006) as our baseline generator. When tested on f-structures for all sentences from Section 23 of the Penn Wall Street Journal (WSJ) treebank (Marcus et al., 1993), the techniques described in this pa­per improve BLEU score from 66.<page local="2" global="268"/>52 to 68.82. In addition, coverage is increased from 98.18% to al­most 100% (99.96%).</p><p>The remainder of the paper is structured as fol­lows: in Section 2 we review related work on sta­tistical sentence generation. Section 3 describes the baseline generation model and in Section 4 we show how the new history-based model improves over the baseline. In Section 5 we describe the source of the multi-word units (MWU) used in our experiments and the various techniques we employ to make use of these MWUs in the generation process. Section 6 gives experimental details and results.</p></section><section number="2" title="Related Work on Statistical Generation"><p>In (statistical) generators, sentences are generated from an abstract linguistic encoding via the appli­cation of grammar rules. These rules can be hand­crafted grammar rules, such as those of (Langkilde-Geary, 2002; Carroll and Oepen, 2005), created semi-automatically (Belz, 2007) or, alternatively, extracted fully automatically from treebanks (Ban­galore and Rambow, 2000; Nakanishi et al., 2005; Cahill and van Genabith, 2006).</p><p>Insofar as it is a broad coverage generator, which has been trained and tested on sections of the WSJ corpus, our generator is closer to the generators of (Bangalore and Rambow, 2000; Langkilde-Geary, 2002; Nakanishi et al., 2005) than to those designed for more restricted domains such as weather fore­cast (Belz, 2007) and air travel domains (Ratnaparkhi, 2000).</p><p>Another feature which characterises statistical generators is the probability model used to select the most probable sentence from among the space of all possible sentences licensed by the grammar. One generation technique is to first generate all possible sentences, storing them in a word lattice (Langkilde and Knight, 1998) or, alternatively, a generation for­est, a packed represention ofalternate trees proposed by the generator (Langkilde, 2000), and then select the most probable sequence of words via an n-gram language model.</p><p>Increasingly syntax-based information is being incorporated directly into the generation model. For example, Carroll and Oepen (2005) describe a sentence realisation process which uses a hand-crafted HPSG grammar to generate a generation forest. A selective unpacking algorithm allows the extraction of an n-best list of realisations where realisation ranking is based on a maximum entropy model. This unpacking algorithm is used in (Velldal and Oepen, 2005) to rank realisations with features defined over HPSG derivation trees. They achieved the best re­sults when combining the tree-based model with an n-gram language model.</p><p>Nakanishi et al. (2005) describe a treebank-extracted HPSG-based chart generator. Importing techniques developed for HPSG parsing, they apply a log linear model to a packed representation of all alternative derivation trees for a given input. They found that a model which included syntactic infor­mation outperformed a bigram model as well as a combination of bigram and syntax model.</p><p>The probability model described in this paper also incorporates syntactic information, however, unlike the discriminative HPSG models just described, it is a generative history- and PCFG-based model.</p><doubt alpha="55.6" length="45" tooSmall="False" monospace="0.0">While Belz (2007) and Humphreys et al. (2001)</doubt><p>mention the use of contextual features for the rules in their generation models, they do not provide de­tails nor do they provide a formal probability model. To the best of our knowledge this is the first paper providing a probabilistic generative, history-based generation model.</p></section><section number="3" title="Surface Realisation from f-Structures"><p>Cahill and van Genabith (2006) present a prob­abilistic surface generation model for LFG (Ka­plan, 1995). LFG is a constraint-based theory of grammar, which analyses strings in terms of c(onstituency)-structure and f(unctional)-structure (Figure 1). C-structure is defined in terms of CFGs, and f-structures are recursive attribute-value ma­trices which represent abstract syntactic functions (such as suBJect, OBJect, OBLique, coMPlement (sentential), ADJ(N)unct), agreement, control, long­distance dependencies and some semantic informa­tion (e.g. tense, aspect).</p><p>C-structures and f-structures are related in a pro­jection architecture in terms of a piecewise corre­spondence 0.<footnote anchor="1"/> The correspondence is indicated in</p><p>'Our formalisation follows (Kaplan, 1995).</p><page local="3" global="269"/><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">s</doubt><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">î=4</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">NP</doubt><doubt alpha="55.6" length="9" tooSmall="False" monospace="0.0">(îSUBJ)=4</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">NNP</doubt><p>Susan contacted</p><doubt alpha="64.7" length="34" tooSmall="False" monospace="0.0">(îpred)= 'Susan'(îpred)= 'contact'</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">PRP</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">her</doubt><doubt alpha="57.1" length="14" tooSmall="False" monospace="0.0">(îpred)= 'pro'</doubt><doubt alpha="61.1" length="18" tooSmall="False" monospace="0.0">(îNUM)=SG(îPERS)=3</doubt><doubt alpha="20.0" length="5" tooSmall="False" monospace="0.0">f2: '</doubt><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">f2:</doubt><doubt alpha="84.6" length="13" tooSmall="True" monospace="0.0">PRED NUM PERS</doubt><doubt alpha="100.0" length="2" tooSmall="True" monospace="0.0">SG</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">3</doubt><doubt alpha="71.4" length="7" tooSmall="False" monospace="0.0">'PRO'SG</doubt><figure caption="Figure 1: C- and f-structures with0links for the sentenceSusan contacted her."></figure><p>terms of the curvy arrows pointing from c-structure nodes to f-structure components in Figure 1. Given a c-structure node <i>ui, </i>the corresponding f-structure component fj is 0(ui). F-structures and the c-structure/f-structure correspondence are described in terms of functional annotations on c-structure nodes (CFG grammar rules). An equation of the form (|F) = j states that the f-structure associated with the mother of the current c-structure node (|) has an attribute (grammatical function) (F), whose value is the f-structure of the current node (j). The up-arrows and down-arrows are shorthand for 0(M(ui)) = 0(ui) where ui is the c-structure node annotated with the equation.<footnote anchor="2"/></p><p><i>Tree</i><i>best </i>:= <i>argmaxjreeP(Tree\F-Str) </i>(1)</p><doubt alpha="55.4" length="74" tooSmall="False" monospace="0.0">P(Tree\F-Str) :=Y\P(X—Y\X, Feats)(2)X—Y in Tree Feats={ai\3vj(4&gt;(X))at=Vj}</doubt><p>The generation model of (Cahill and van Gen­abith, 2006) maximises the probability of a tree given an f-structure (Eqn. 1), and the string gener­ated is the yield of the highest probability tree. The generation process is guided by <i>purely </i>local infor­mation in the input f-structure: f-structure annotated CFG rules (LHS —&gt; RHS) are conditioned on their LHSs and on the set of features/attributes <i>Feats </i>= <i>[ai</i><i>\3</i><i>vj</i>0(LHS)a^ = <i>Vj</i>}<footnote anchor="3"/> 0-linked to the LHS (Eqn.</p><footnote label="2">M is the mother function on CFG tree nodes.</footnote><footnote label="3">In words, Feats is the set of top level features/attributes (those attributes a t for which there is a value V i) of the f-structure 0 linked to the LHS.</footnote><p>2). Table 1 shows a generation grammar rule and conditioning features extracted from the example in Figure 1. The probability of a tree is decomposed into the product of the probabilities of the f-structure annotated rules (conditioned on the LHS and local <i>Feats) </i>contributing to the tree. Conditional proba­bilities are estimated using maximum likelihood es­timation.</p><p><u>grammar rule</u>_<u>|  local conditioning features</u>_</p><p>s(T=4)^NP(TsuBj=|)VP(t=4) | s(|=i), {subj,obj,pred,tense} Table 1: Example grammar rule (from Figure 1).</p><p>Cahill and van Genabith (2006) note that condi­tioning f-structure annotated generation rules on lo­cal features (Eqn. 2) can sometimes cause the model to make inappropriate choices. Consider the follow­ing scenario where in addition to the c-/f-structure in Figure 1, the training set contains the c-/f-structure displayed in Figure 2.</p><p>From Figures 1 and 2, the model learns (among others) the generation rules and conditional proba­bilities displayed in Tables 2 and 3.</p><table caption="Table 2: A sample of internal grammar rules ex­tracted from Figures 1 and 2."></table><p>Given the input f-structure (for She accepted) in Figure 3, (and assuming suit­able generation rules for intransitive VPs and accepted) the model would produce the inappro­priate highest probability tree of Figure 4 with an incorrect case for the pronoun in subject position.</p><doubt alpha="66.7" length="24" tooSmall="False" monospace="0.0">'CONTACT{(îSUBj)(îOBj))'</doubt><doubt alpha="71.4" length="7" tooSmall="False" monospace="0.0">'SUSAN'</doubt><p>(î num) = sg (î tense) = past<page local="4" global="270"/></p><doubt alpha="50.0" length="10" tooSmall="False" monospace="0.0">(îpers)= 3</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>--H</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>SUBJ</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>fi:</p></td><td class="cell"><p>OBJ</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>TENSE</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 2: A sample of internal grammar rules extracted from Figures 1 and 2." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>F-Struct Feats</p></td><td class="cell"><p>Grammar Rules</p></td><td class="cell"><p>Prob</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>{subj, obj, pred}</p></td><td class="cell"><p>S(Î=|)-NP(îsubj=4)VP(î=4)</p></td><td class="cell"><p>1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>{subj, obj, pred}</p></td><td class="cell"><p>vpC'=4)^V(T=4)NP(Tobj=4)</p></td><td class="cell"><p>1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>{num, per, gen}</p></td><td class="cell"><p>NP("subj=..) ^NNP(T=4)</p></td><td class="cell"><p>0.5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>{num, per, gen}</p></td><td class="cell"><p>np(Tsubj=4) -► PRP(T=4)</p></td><td class="cell"><p>0.5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>{num, per, gen}</p></td><td class="cell"><p>np(Tobj=4) ^PRP(T=4)</p></td><td class="cell"><p>1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">NP</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">PRP</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">her</doubt><doubt alpha="100.0" length="4" tooSmall="True" monospace="0.0">SUBJ</doubt><doubt alpha="84.6" length="13" tooSmall="True" monospace="0.0">PRED NUM PERS</doubt><doubt alpha="100.0" length="2" tooSmall="True" monospace="0.0">SG</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">S</doubt><doubt alpha="45.5" length="11" tooSmall="True" monospace="0.0">(î SUBJ)= |</doubt><doubt alpha="40.0" length="5" tooSmall="True" monospace="0.0">î = i</doubt><p>She hired</p><doubt alpha="58.6" length="29" tooSmall="False" monospace="0.0">(îpred)= 'pro'(îpred)= 'hire'</doubt><doubt alpha="54.3" length="35" tooSmall="False" monospace="0.0">(î PRED)= 'pro'(î NUM)=SG (îpers)=3</doubt><doubt alpha="61.9" length="21" tooSmall="False" monospace="0.0">'hire{(Îsubj)(Îobj))'</doubt><doubt alpha="60.0" length="5" tooSmall="False" monospace="0.0">'PRO'</doubt><doubt alpha="20.0" length="5" tooSmall="True" monospace="0.0">Î2- "</doubt><doubt alpha="33.3" length="3" tooSmall="True" monospace="0.0">Î2-</doubt><doubt alpha="66.7" length="3" tooSmall="True" monospace="0.0">SG3</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">B</doubt><figure caption="Figure 2: C- and f-structures with0links for the sentenceShe hired her."></figure><doubt alpha="90.0" length="10" tooSmall="True" monospace="0.0">PRED TENSE</doubt><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">past</doubt><figure caption="Figure 3: Input f-structure for Sheaccepted."></figure><p>To solve the problem, Cahill and van Gen­abith (2006) apply an automatic generation gram­mar transformation to their training data: they au­tomatically label CFG nodes with additional case information and the model now learns the new im­proved generation rules of Tables 4 and 5. Note how the additional case labelling subverts the prob­lematic independence assumptions of the probabil­ity model and communicates the fact that a subject NP has to be realised as nominative case from the <i>S — NP-nom VP </i>production, via the intermediate <i>NP-nom — PRP-nom, </i>down to the lexical produc­tion <i>PRP-nom — she. </i>The labelling guarantees that, given the example f-structure in Figure 3, the model generates the correct string she accepted.</p><doubt alpha="66.7" length="3" tooSmall="True" monospace="0.0">î=l</doubt><doubt alpha="60.0" length="10" tooSmall="False" monospace="0.0">(î SUBJ)=I</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">VP</doubt><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">î=l1</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">V</doubt><p>her accepted</p><doubt alpha="58.6" length="29" tooSmall="False" monospace="0.0">(tpred)= 'pro'(tpred)= 'hire'</doubt><doubt alpha="63.6" length="33" tooSmall="False" monospace="0.0">(tnum)=sg(ttense)= past(tpers)= 3</doubt><figure caption="Figure 4: Inappropriate output:her accepted."></figure><table caption="Table 5: Lexical item rules with case markings"></table></section><section number="4" title="A History-Based Generation Model"><p>The automatic generation grammar transform pre­sented in (Cahill and van Genabith, 2006) provides a solution to coarse-grained and (in fact) inappropri­ate independence assumptions in the basic genera­tion model. However, there is a sense in which the proposed cure improves on the symptoms, but not the cause of the problem: it weakens independence assumptions by multiplying and hence increasing the specificity of conditioning CFG category labels. There is another option available to us, and that is the option we will explore in this paper: instead of applying a generation grammar transform, we will improve the f-structure-based conditioning of the generation rule probabilities. In the original model, rules are conditioned on <i>purely local </i>f-structure con­text: the set of features/attributes ^-linked to the LHS of a grammar rule. As a direct consequence of this, the conditioning (and hence the model) can­not not distinguish between NP, PRP and NNP rules</p><p>(î num) = sg (î tense) = past<page local="5" global="271"/></p><doubt alpha="50.0" length="10" tooSmall="False" monospace="0.0">(îpers)= 3</doubt><table caption="Table 3: A sample of lexical item rules extracted from Figures 1 and 2." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>F-Struct Feats</p></td><td class="cell"><p>Grammar Rules</p></td><td class="cell"><p>Prob</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>{PRED=PRO,NUM=SG PER</b>=3, <b>GEN=FEM}</b></p></td><td class="cell"><p>PRP(T=|) -&gt; she</p></td><td class="cell"><p>0.33</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>{PRED=PRO,NUM=SG PER</b>=3, <b>GEN=FEM}</b></p></td><td class="cell"><p>PRP(T=|) -&gt; her</p></td><td class="cell"><p>0.66</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PRED</p></td><td class="cell"><p>pro</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NUM</p></td><td class="cell"><p>sg</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PERS</p></td><td class="cell"><p>3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>GEND</p></td><td class="cell"><p>fem</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>accept</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 4: Internal grammar rules with case markings." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>F-Struct Feats</p></td><td class="cell"><p>Grammar Rules</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>{pred=pro,num=sg per=3, gen=fem}</p></td><td class="cell"><p>PRP-nom(î=|)^ she</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>{pred=pro,num=sg per=3, gen=fem}</p></td><td class="cell"><p>PRP-acc(î=|) —» her</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>F-Struct Feats</p></td><td class="cell"><p>Grammar Rules</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>{subj, obj, pred}</p></td><td class="cell"><p>S(T=JJ     NP-nom(ÎSUBJ=JJ VP(T=JJ</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>{subj, obj, pred}</p></td><td class="cell"><p>VP(T=JJ V(T=J.)NP-acc(TOBJ=J.)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>{num, per, gen}</p></td><td class="cell"><p>NP-nom("SUBJ=..) —► PRP-nom(T=J.)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>{num, per, gen}</p></td><td class="cell"><p>NP-nom("SUBJ=..) —► NNP-nom(T=J.)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>{num, per, gen}</p></td><td class="cell"><p>NP-acc(TOBJ=J.) —»-PRP-acc(T=i)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>appropriate to e.g. subject (subj) or object con­texts (obj) in a given input f-structure. However, the required information can easily be incorporated into the generation model by uniformly conditioning generation rules on their <i>parent (mother) </i>grammati­cal function, in addition to the local ((-linked feature set. This additional conditioning has the effect of making the choice of generation rules sensitive to the <i>history </i>of the generation process, and, we argue, provides a simpler, more uniform, general, intuitive and natural probabilistic generation model obviating the need for CFG-grammar transforms in the origi­nal proposal of (Cahill and van Genabith, 2006).</p><p>In the new model, each generation rule is now conditioned on the LHS rule CFG category, the set of features ((-linked to LHS <i>and </i>the parent grammat­ical function of the f-structure ((-linked to LHS. In a given c-/f-structure pair, for a CFG node n, the par­ent grammatical function of the f-structure ((-linked to n is that grammatical function GF, which, if we take the f-structure ((-linked to the mother M(n), and apply it to GF, returns the f-structure ((-linked to n: (&lt;(M(n))GF) = &lt;(n).</p><p>The basic idea is best explained by way of an example. Consider again Figure 1. The mother grammatical function of the f-structure f2 asso­ciated with node NP(|subj=|) and its daughter NNP(|=D (via the |=j functional annotation) is subj, as (&lt;&lt;(M(n2))suBj) = &lt;&lt;(n2), or equivalently (fi subj) = f2.</p><p>Given Figures 1 and 2 as training set, the im­proved model learns the generation rules (the mother grammatical function of the outermost f-structure is assumed to be a dummy top grammatical function) of Tables 6 and 7.</p><p>Note, that for our example the effect of the uni­form additional conditioning on mother grammat­ical function has the same effect as the genera­tion grammar transform of (Cahill and van Gen­abith, 2006), but without the need for the grammar transform. Given the input f-structure in Fig­ure 3, the model will generate the correct string she accepted. In addition, uniform condition­ing on mother grammatical function is more general than the case-phenomena specific generation gram­mar transform of (Cahill and van Genabith, 2006), in that it applies to each and every sub-part of a recursive input f-structure driving generation, mak­ing available relevant generation history (context) to guide local generation decisions.</p><p>The new history-based probabilistic generation model is defined as:</p><doubt alpha="56.4" length="39" tooSmall="False" monospace="0.0">P(Tree\F-Str):=FJP(X—Y\X, Feats,GF) (3)</doubt><doubt alpha="49.1" length="57" tooSmall="False" monospace="0.0">X—Y in Tree Feats={ai\3vj(&lt;j&gt;(X ))at=Vj}(&lt;(M(X)))GF =&lt;(X)</doubt><p>Note that the new conditioning feature, the f-structure mother grammatical function, GF, is avail­able from structure previously generated in the c-structure tree. As such, it is part of the <i>history </i>of the tree, i.e. it has already been generated in the top-down derivation of the tree. In this way, the gen­eration model resembles history-based models for parsing (Black et al., 1992; Collins, 1999; Charniak, 2000). Unlike, say, the parent annotation for parsing of (Johnson, 1998) the parent GF feature for a par­ticular node expansion is not merely extracted from the parent node in the c-structure tree, but is some­times extracted from an ancestor node further up the c-structure tree via intervening |=| functional an­notations.</p><p>Section 6 provides evaluation results for the new model on section 23 of the Penn treebank.</p></section><section number="5" title="Multi-Word Units"><p>In another effort to improve generator accuracy over the baseline model we explored the use of multi­word units in generation. We expect that the identi­fication of MWUs may be useful in imposing word-order constraints and reducing the complexity of the generation task. Take, for example, the following<page local="6" global="272"/></p><table caption="Table 7: Lexical item rules." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>F-Struct Feats</p></td><td class="cell"><p>Grammar Rules</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>{pred=pro,num=sg per=3, gen=fem, SUBj}</p></td><td class="cell"><p>PRP(|=|) she</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>{pred=pro,num=sg per=3, gen=fem, OBj}</p></td><td class="cell"><p>PRP(|=|) her</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 6: Grammar rules with extra feature extracted from F-Structures." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>F-Struct Feats</p></td><td class="cell"><p>Grammar Rules</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>{subj, obj, pred, top}</p></td><td class="cell"><p>S(T=4)-np(Tsubj=4)VP(T=|)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>{subj, obj, pred, top}</p></td><td class="cell"><p>vpc=4)-V(T=|)NP(Tobj=4)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>{num, per, gen, subj}</p></td><td class="cell"><p>NP("subj=|) -&gt;prp(T=|)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>{num, per, gen, obj}</p></td><td class="cell"><p>NP(Tobj=|) -► PRP(T=|)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>{num, per, gen, subj}</p></td><td class="cell"><p>NP(Tsubj=|) -&gt;nnp(T=|)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>PRED 'New'</p><p>NUM sg PERS 3</p><p>PRED 'York' NUM sg PERS 3</p><p>PRED 'New.York' NUM sg PERS 3</p><p>PRED 'New'/NE1_1 NUM sg PERS 3</p><p>PRED 'York'/NEl_2 NUM sg PERS 3</p><p>Figure 5: Three different f-structure formats. From left to right: the original f-structure format; the MWU chunk format; the MWU mark-up format.</p><doubt alpha="100.0" length="7" tooSmall="True" monospace="0.0">ADJUNCT</doubt><doubt alpha="100.0" length="3" tooSmall="True" monospace="0.0">APP</doubt><p>two sentences which show the gold version of a sen­tence followed by the version of the sentence pro­duced by the generator:</p><doubt alpha="61.2" length="49" tooSmall="False" monospace="0.0">GoldBy this time , it was 4:30 a.m. in New York ,</doubt><p><i>and Mr. Smith fielded a call from a New York customer wanting an opinion on the British stock market , which had been having trou­bles of its own even before Friday's New York market break .</i></p><p><i>Test By this time , in New York, it was 4:30 a.m.</i></p><p><i>, and Mr. Smith fielded a call from </i>New <i>a customer </i>York <i>, wanting an opinion on the market British stock which had been having troubles of its own even before Friday 's New York market break .</i></p><p>The gold version of the sentence contains a multi­word unit, <i>New York, </i>which appears fragmented in the generator output. If multi-word units were either treated as one token throughout the generation pro­cess, or, alternatively, if a constraint were imposed on the generator such that multi-word units were al­ways generated in the correct order, then this should help improve generation accuracy. In Section 5.1 we describe the various techniques that were used to incorporate multi-word units into the generation process and in 5.2 we detail the different types and sources of multi-word unit used in the experiments. Section 6 provides evaluation results on test and de­velopment sets from the WSJ treebank.</p><subsection number="5.1" title="Incorporating MWUs into the Generation Process"><p>We carried out three types of experiment which, in different ways, enabled the generation process to respect the restrictions on word-order provided by multi-word units. For the first experiments (type 1), the WSJ treebank <i>training </i>and <i>test </i>data were altered so that multi-word units are concatenated into single words (for example, <i>New York </i>becomes <i>New.</i><i>York). </i>As in (Cahill and van Genabith, 2006) f-structures are generated from the (now altered) tree-bank and from this data, along with the treebank trees, the PCFG-based grammar, which is used for training the generation model, is extracted. Simi­larly, the f-structures for the test and development sets are created from Penn Treebank trees which have been modified so that multi-word units form single units. The leftmost and middle f-structures in Figure 5 show an example of an original f-structure format and a named-entity chunked format, respec­tively. Strings output by the generator are then post-processed so that the concatenated word sequences are converted back into single words.</p><p>In the second experiment (type 2) only the <i>test </i>data was altered with no concatenation of MWUs carried out on the training data.</p><p>In the final experiments (type 3), instead of con­catenating named entities, a constraint is introduced to the generation algorithm which penalises the gen­eration of sequences of words which violate the in­ternal word order of named entities. The input is marked-up in such a way that, although named en­tities are no longer chunked together to form single words, the algorithm can read which items are part of named entities. See the rightmost f-structure in Figure 5 for an example of an f-structure marked-up in this way. The tag <i>NE1J,</i><i> </i>for example, indi­cates that the sub-f-structure is part of a named iden­tity with id number 1 and that the item corresponds to the first word of the named entity. The baseline generation algorithm, following Kay (1996)'s work on chart generation, already contains the hard con­straint that when combining two chart edges they must cover disjoint sets of words. We added an ad­ditional constraint which prevents edges from being combined if this would result in the generation of a string which contained a named entity which was either incomplete or where the words in the named entity were generated in the wrong order.<page local="7" global="273"/></p></subsection><subsection number="5.2" title="Types of MWUs used in Experiments"><p>We carry out experiments with multi-word units from three different sources. First, we use the output of the maximum entropy-based named entity recog­nition system of (Chieu and Ng, 2003). This sys­tem identifies four types of named entity: person, organisation, location, and miscellaneous. Addition­ally we use a dictionary of candidate multi-word ex­pressions based on a list from the Stanford Multi­word Expression Project<footnote anchor="4"/>. Finally, we also carry out experiments with multi-word units extracted from the BBN Pronoun Coreference and Entity Type Cor­pus (Weischedel and Brunstein, 2005). This supple­ments the Penn WSJ treebank's one million words of syntax-annotated Wall Street Journal text with addi­tional annotations of 23 named entity types, includ­ing nominal-type named entities such as person, or­ganisation, location, etc. as well as numeric types such as date, time, quantity and money. Since the BBN corpus data is very comprehensive and is hand­annotated we take this be be a gold standard, repre­senting an upper bound for any gains that might be made by identifying complex named entities in our experiments.<footnote anchor="5"/> Table 8 gives examples of the various types of MWUs identified by the three sources.</p><p>For our purposes we are not concerned with the distinctions between different types of named enti­ties; we are merely exploiting the fact that they may be treated as atomic units in the generation model. In all cases we disregard multi-word units that cross the original syntactic bracketing of the WSJ treebank. An overview ofthe various types ofmulti-word units used in our experiments is presented in Table 9.</p></subsection></section><section number="6" title="Experimental Evaluation"><p>All experiments were carried out on the WSJ tree-bank with sections 02-21 for training, section 24 for development and section 23 for final test results. The LFG annotation algorithm of (Cahill et al., 2004) was used to produce the f-structures for develop­ment, test and training sets.</p><footnote label="4">mwe.stanford.edu</footnote><footnote label="5">Although it is possible there are other types of MWUs that may be more suitable to the task than the named entities identi­fied by BBN, so further gains might be possible.</footnote><p>Table 9: Average number of MWUs per sentence and average MWU length in the WSJ treebank grouped by MWU source.</p><p>Table 10 shows the final results for section 23. For each test we present BLEU score results as well as String Edit Distance and coverage. We measure sta­tistical significance using two different tests. First we use a bootstrap resampling method, popular for machine translation evaluations, to measure the sig­nificance of improvements in BLEU scores, with a resampling rate of 1000.<footnote anchor="6"/> We also calculated the significance of an increase in String Edit Distance by carrying out a paired t-test on the mean differ­ence of the String Edit Distance scores. In Table 10, » means significant at level 0.005. &gt; means signif­icant at level 0.05.</p><p>In Table 10, <i>Baseline </i>gives the results of the generation algorithm of (Cahill and van Genabith, 2006). <i>HB Model </i>refers to the improved model with the increased history context, as described in Section 4. The results, where for example the BLEU score rises from 66.52 to 67.24, show that even increasing the conditioning context by a limited amount increases the accuracy of the system signif­icantly for both BLEU and String Edit Distance.<page local="8" global="274"/> In addition, coverage goes up from 98.18% to 99.88%.</p><footnote label="6">Scripts for running the bootstrapping method carried out in our evaluation are available for download at projec-tile.is.cs.cmu.edu/research/public/tools/bootStrap/tutorial.htm</footnote><table caption="Table 8: Examples of some of the types of MWU from the three different sources." class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MWU type</p></td><td class="cell"><p>Examples</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Names</p></td><td class="cell"><p><i>Martha Matthews</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><i>Yoshio Hatakeyama</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Organisations</p></td><td class="cell"><p><i>Rolls-Royce Motor Cars Inc.</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><i>Washington State University</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Locations</p></td><td class="cell"><p><i>New York City</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><i>New Zealand</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Time expressions</p></td><td class="cell"><p><i>October 19th</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><i>two years ago</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><i>the 21st century</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Quantities</p></td><td class="cell"><p><i>$2.7 million to $3 million</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><i>about 25 %</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><i>60 mph</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Prepositional expressions</p></td><td class="cell"><p><i>in fact</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><i>at the time</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><i>on average</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>average number</p></td><td class="cell"><p>average length</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(Chieu and Ng, 2003)</p></td><td class="cell"><p>0.61</p></td><td class="cell"><p>2.40</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Stanford MWE Project</p></td><td class="cell"><p>0.10</p></td><td class="cell"><p>2.48</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>BBN Corpus</p></td><td class="cell"><p>1.15</p></td><td class="cell"><p>2.66</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p><i>+MWU Best Automatic </i>displays our best results using automatically identified named entities. These were achieved using experiment type 2, described in Section 5, with the MWUs produced by (Chieu and Ng, 2003). Results displayed in Table 10 up to this point are cumulative. The final row in Ta­ble 10, <i>MWU BBN, </i>shows the best results with BBN MWUs: the history-based model with BBN multi­word units incorporated in a type 1 experiment.</p><p>We now discuss the various MWU experiments in more detail. See Table 11 for a breakdown of the MWU experiment results on the development set, WSJ section 24. Our baseline for these exper­iments is the history-based generator presented in Section 4. For each experiment type described in Section 5.1 we ran three experiments, varying the source of MWUs. First, MWUs came from the auto­matic NE recogniser of (Chieu and Ng, 2003), then we added the MWUs from the Stanford list and fi­nally we ran tests with MWUs extracted from the BBN corpus.</p><p>Our first set of experiments (type 1), where both training data and development set data were MWU-chunked, produced the worst results for the automat­ically chunked MWUs. BLEU score accuracy actu­ally decreased for the automatically chunked MWU experiments. In an error analysis of type 1 ex­periments with (Chieu and Ng, 2003) concatenated MWUs, we inspected those sentences where accu­racy had decreased from the baseline. We found that for over half (51.5%) of these sentences, the in­put f-structures contained no multi-word units at all. The problem for these sentences therefore lay with the probabilistic grammar extracted from the MWU-chunked training data. When the source of MWU for the type 1 experiments was the BBN, however, accuracy improved significantly over the baseline and the result is the highest accuracy achieved over all experiment types. One possible reason for the low accuracy scores in the type 1 experiments with the (Chieu and Ng, 2003) MWU chunked data could be noisy MWUs which negatively affect the gram­mar. For example, the named entity recogniser of (Chieu and Ng, 2003) achieves an accuracy of 88.3% on section 23 of the Penn Treebank.</p><p>In order to avoid changing the grammar through concatenation of MWU components (as in exper­iment type 1) and thus risking side-effects which cause some heretofore likely constructions become less likely and vice versa, we ran the next set of ex­periments (type 2) which leave the original grammar intact and alter the input f-structures only. These experiments were more successful overall and we achieved an improvement over the baseline for both BLEU and String Edit Distance scores with all MWU types. As can be seen from Table 11 the best score for automatically chunked MWUs are decreases marginally when we added the Stanford MWUs. In our final set of experiments (type 3) al­though the accuracy for all three types of MWUs improves over the baseline, accuracy is a little be­low the type 2 experiments.</p><doubt alpha="65.9" length="44" tooSmall="False" monospace="0.0">with the (Chieu and Ng, 2003) MWUs. Accuracy</doubt><p>It is difficult to compare sentence generators since the information contained in the input varies greatly between systems, systems are evaluated on different test sets and coverage also varies considerably. In order to compare our system with those of (Nakan-ishi et al., 2005) and (Langkilde-Geary, 2002) we report our best results with automatically acquired MWUs for sentences of &lt; 20 words in length on section 23: our system gets coverage of 100% and a BLEU score of 71.39. For the same test set Nakan-ishi et al. (2005) achieved coverage of 90.75 and a BLEU score of 77.33. Langkilde-Geary (2002) reports 82.<page local="9" global="275"/>7% coverage and a BLEU score of 75.7% on the same test set with the 'permute,no dir' type input. Langkilde-Geary (2002) report results for ex­periments with varying levels of linguistic detail in the input given to the generator. As with Nakanishi et al. (2005) we find the 'permute,no dir' type of in­put is most comparable to the level of information contained in our input f-structures. Finally, the sym­bolic generator of Callaway (2003) reports a Sim­ple String Accuracy score of 88.84 and coverage of 98.7% on section 23 for all sentence lengths.</p><table caption="Table 10: Results on Section 23 for all sentence lengths." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Section 23 (2416 sentences)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Model</p></td><td class="cell"><p>BLEU</p></td><td class="cell"><p>StringEd</p></td><td class="cell"><p>Coverage</p></td><td class="cell"><p>BLEU Bootstrap Signif</p></td><td class="cell"><p>StringEd Paired T-Test</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1. Baseline</p></td><td class="cell"><p>66.52</p></td><td class="cell"><p>68.69</p></td><td class="cell"><p>98.18</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2. HB Model</p></td><td class="cell"><p>67.24</p></td><td class="cell"><p>69.89</p></td><td class="cell"><p>99.88</p></td><td class="cell"><p>&gt; 1</p></td><td class="cell"><p>&gt; 1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3. +MWU Best Automatic</p></td><td class="cell"><p>67.81</p></td><td class="cell"><p>70.36</p></td><td class="cell"><p>99.92</p></td><td class="cell"><p>&gt; 2</p></td><td class="cell"><p>&gt; 2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>4. MWU BBN</p></td><td class="cell"><p>68.82</p></td><td class="cell"><p>70.92</p></td><td class="cell"><p>99.96</p></td><td class="cell"><p>&gt;3</p></td><td class="cell"><p>&gt; 3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><section number="7" title="Conclusion and Future Work"><p>We have presented techniques which improve the ac­curacy of an already state-of-art surface generation model. We found that a history-based model that increases conditioning context in PCFG style rules by simply including the grammatical function of the f-structure parent, improves generator accuracy. In the future we will experiment with increasing condi­tioning context further and using more sophisticated smoothing techniques to avoid sparse data problems when conditioning is increased.</p><p>We have also demonstrated that automatically ac­quired multi-word units can bring about moderate, but significant, improvements in generator accuracy. For automatically acquired MWUs, we found that this could best be achieved by concatenating input items when generating the f-structure input to the generator, while training the input generation gram­mar on the original (i.e. non-MWU concatenated) sections of the treebank. Relying on the BBN cor­pus as a source of multi-word units, we gave an up­per bound to the potential usefulness of multi-word units in generation and showed that automatically acquired multi-word units, encouragingly, give re­sults not far below the upper bound.</p><table caption="Table 11: Results on Section 24, all sentence lengths." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>Section 24 (1346 sentences)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Model</p></td><td class="cell"><p>MWUs</p></td><td class="cell"><p>BLEU</p></td><td class="cell"><p>StringEd</p></td><td class="cell"><p>Coverage</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>HB Model</p></td><td class="cell"><p></p></td><td class="cell"><p>65.85</p></td><td class="cell"><p>69.93</p></td><td class="cell"><p>99.93</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>type 1</p></td><td class="cell"><p>(Chieu and Ng, 2003)</p></td><td class="cell"><p>65.81</p></td><td class="cell"><p>70.34</p></td><td class="cell"><p>99.93</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(training and test data chunked)</p></td><td class="cell"><p>+Stanford MWEs</p></td><td class="cell"><p>64.81</p></td><td class="cell"><p>69.67</p></td><td class="cell"><p>99.93</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>BBN</p></td><td class="cell"><p>67.24</p></td><td class="cell"><p>71.46</p></td><td class="cell"><p>99.93</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>type 2</p></td><td class="cell"><p>(Chieu and Ng, 2003)</p></td><td class="cell"><p>66.37</p></td><td class="cell"><p>70.26</p></td><td class="cell"><p>99.93</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(test data chunked)</p></td><td class="cell"><p>+Stanford MWEs</p></td><td class="cell"><p>66.28</p></td><td class="cell"><p>70.21</p></td><td class="cell"><p>99.93</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>BBN</p></td><td class="cell"><p>66.84</p></td><td class="cell"><p>70.74</p></td><td class="cell"><p>99.93</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>type 3</p></td><td class="cell"><p>(Chieu and Ng, 2003)</p></td><td class="cell"><p>66.30</p></td><td class="cell"><p>70.12</p></td><td class="cell"><p>100</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(internal generation constraint)</p></td><td class="cell"><p>+Stanford MWEs</p></td><td class="cell"><p>66.07</p></td><td class="cell"><p>70.02</p></td><td class="cell"><p>99.93</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>BBN</p></td><td class="cell"><p>66.45</p></td><td class="cell"><p>70.14</p></td><td class="cell"><p>99.93</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><references><p>Srinivas Bangalore and Owen Rambow. 2000. Exploit­ing a probabilistic hierarchical model for generation.</p><p>In <i>Proceedings of the 18th COLING.</i></p><p>AnjaBelz. 2007. Probabilistic generation of wether fore­cast texts. In <i>Proceedings ofNAACL-HLT.</i></p><p>Ezra Black, Fred Jelinek, John Lafferty, David M. Mager-man, Robert Mercer, and Salim Roukos. 1992. To­wards history-based grammars: Using richer models for probabilistic parsing. In <i>Proceeding of the 5th DARPA Speech and Language Workshop.</i></p><p>Aoife Cahill and Josef van Genabith. 2006. Robust PCFG-based generation using automatically acquired LFG approximations. In <i>Proceedings of the 44th ACL.</i></p><p>Aoife Cahill, Michael Burke, Ruth O'Donovan, Josef van Genabith, and Andy Way. 2004. Long-distance de­pendency resolution in automatically acquired wide-coverage PCFG-based LFG approximations. In <i>Pro­ceedings of the 42nd ACL.</i></p><p>Charles B. Callaway. 2003. Evaluating coverage for large symbolic NLG grammars. In <i>In Proceedings of the 18th IJCAI.</i></p><p>John A. Carroll and Stephan Oepen.  2005.  High ef­ficiency realization for a wide-coverage unification grammar. In <i>Proceedings ofIJCNLP.</i></p><p>Eugene Charniak. 2000. A maximum entropy-inspired parser. In<i>Proceedingsofthe 1st NAACL.</i></p><p>Hai Leong Chieu and Hwee Tou Ng. 2003. Named en­tity recognitionwith a maximumentropy approach. In <i>Proceedings ofthe CoNLL.</i><i></i></p><page local="10" global="276"/><p>Michael Collins. 1999. <i>Head-Driven Statistical Models for Natural Language Parsing. </i>Ph.D. thesis, Univer­sity ofPennsylvania.</p><p>Kevin Humphreys, Mike Calcagno, and David Weise. 2001. Reusing a statistical language model for gen­eration. In <i>Proceedings ofthe 8th European Workshop on Natural Language Generation (EWNLG).</i></p><p>Mark Johnson. 1998. PCFG models of linguistic tree representations. <i>Computational Linguistics, </i>24.</p><p>Ronald M. Kaplan and Tracy Holloway King. 2003. Low-level mark-up and large-scale LFG grammar pro­cessing. In <i>Proceedings ofthe Lexical Functional Grammar Conference.</i></p><p>Ron Kaplan. 1995. The formal architecture of lexical-functional grammar. In Dalrymple, Kaplan, Maxwell, and Zaenen, editors, <i>Formal Issues in Lexical-Functional Grammar, </i>pages 7-27. CSLI Pub­lications.</p><p>Martin Kay. 1996. Chart generation. In <i>Proceedings of the 34th ACL.</i></p><p>Irene Langkilde and Kevin Knight. 1998. Generation that exploits corpus-based statistical knowledge. In <i>Proceedings of the 36th Annual Meeting of the Asso­ciation for Computational Linguistics and the 17th In­ternational Conference on Computational Linguistics</i> <i>(ACL-COLING).</i><i></i></p><p>Irene Langkilde-Geary. 2002. An empirical verification of coverage and correctness for a general-purpose sen­tence generator. In <i>Proceedings ofthe 2nd INLG.</i></p><p>Irene Langkilde. 2000. Forest-based statistical sentence generation. In <i>Proceedings of the 1st NAACL.</i></p><p>Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated cor­pus of English: the Penn Treebank. <i>Computational Linguistics, </i>19(2):313-330.</p><p>Hiroko Nakanishi, Yusuke Miyao, and Jun'ichi Tsujii. 2005. Probabilistic models for disambiguation of an HPSG-based chart generator. In <i>Proceedings ofthe 9th IWPT.</i></p><p>Joakim Nivre and Jens Nilsson. 2004. Multiword units in syntactic parsing. In <i>Workshop on Methodologies and Evaluation ofMultiword Units in Real-World Ap­plications.</i></p><p>Adwait Ratnaparkhi. 2000. Trainable methods for sur­face natural language generation. In <i>Proceedings of the 1st NAACL.</i></p><p>Stefan Riezler and John T. Maxwell. 2006. Grammat­ical machine translation.  In <i>Proceedings ofthe 6th</i> <i>NAACL.</i><i></i></p><p>Stefan Riezler, Tracy H. King, Richard Crouch, and An­nie Zaenen. 2003. Statistical sentence condensation using ambiguity packing and stochastic disambigua­tion methods for lexical-functional grammar. In <i>Pro­ceedings ofthe 3rd NAACL.</i></p><p>Erik Velldal and Stephan Oepen. 2005. Maximum en­tropy models for realization ranking. In <i>Proceedings ofthe MT-Summit.</i></p><p>Ralph Weischedel and Ada Brunstein, 2005. <i>BBN pro­noun coreference and entity type corpus. </i>Technical Report.</p></references></body></article>