<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="1611"/><title>A Probabilistic Forest-to-String Model for Language Generation from Typed Lambda Calculus Expressions</title><pubinfo>Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing,pages 1611-1622, Edinburgh, Scotland, UK, July 27-31, 2011. ©2011 Association for Computational Linguistics</pubinfo><author surname="Lu" givenname="Wei"><org  name="National University of Singapore" country="Singapore"/></author><author surname="Ng" givenname="Hwee Tou"><org  name="Konan University" country="Japan" city="Kobe"/></author></firstpageheader><frontmatter><p><b>A Probabilistic Forest-to-String Model for Language Generation from Typed Lambda Calculus Expressions</b></p><p><b>Wei Lu </b>and <b>Hwee Tou Ng</b></p><p>Department of Computer Science</p><p>School of Computing National University of Singapore</p><p>{luwei,nght}@comp.nus.edu.sg</p></frontmatter><abstract>This paper describes a novel probabilistic ap­proach for generating natural language sen­tences from their underlying semantics in the form of typed lambda calculus. The approach is built on top of a novel reduction-based weighted synchronous context free grammar formalism, which facilitates the transforma­tion process from typed lambda calculus into natural language sentences. Sentences can then be generated based on such grammar rules with a log-linear model. To acquire such grammar rules automatically in an unsuper-vised manner, we also propose a novel ap­proach with a generative model, which maps from sub-expressions of logical forms to word sequences in natural language sentences. Ex­periments on benchmark datasets for both En­glish and Chinese generation tasks yield sig­nificant improvements over results obtained by two state-of-the-art machine translation models, in terms of both automatic metrics and human evaluation. </abstract></header><body><section number="1" title="Introduction"><p>This work focuses on the task of generating natu­ral language sentences from their underlying mean­ing representations in the form of formal logical ex­pressions (typed lambda calculus). Many early ap­proaches to generation from logical forms make use of rule-based methods (Wang, 1980; Shieber et al., 1990), which concern surface realization (ordering and inflecting of words) but largely ignore lexical ac­quisition. Recent approaches start to employ corpus-based probabilistic methods, but many of them as­sume the underlying meaning representations are of specific forms such as variable-free tree-structured representations (Wong and Mooney, 2007a; Lu et al., 2009) or database entries (Angeli et al., 2010).</p><p>While these algorithms usually work well on spe­cific semantic formalisms, it is unclear how well they could be applied to a different semantic formal­ism. In this work, we propose a general probabilis­tic model that performs generation from underlying formal semantics in the form of typed lambda calcu­lus expressions (we refer to them as <i>X-expressions </i>throughout this paper), where both lexical acquisi­tion and surface realization are integrated in a single framework.</p><p>One natural proposal is to adopt a state-of-the-art statistical machine translation approach. However, unlike text to text translation, which has been ex­tensively studied in the machine translation commu­nity, translating from logical forms into text presents additional challenges. Specifically, logical forms such as X-expressions may have complex internal structures and variable dependencies across sub­expressions. Problems arise when performing auto­matic acquisition of a translation lexicon, as well as performing lexical selection and surface realization during generation.</p><p>In this work, we tackle these challenges by mak­ing the following contributions:</p><p><i>• A novel forest-to-string generation algorithm: </i>Inspired by the work of Chiang (2007), we in­troduce a novel reduction-based weighted bi­nary synchronous context-free grammar for­malism for generation from logical forms (X-expressions), which can then be integrated with a probabilistic forest-to-string generation algo-</p><page local="2" global="1612"/><p>rithm.</p><p><i>• A novel grammar induction algorithm: </i>To au­tomatically induce such synchronous grammar rules, we propose a novel generative model that establishes phrasal correspondences be­tween logical sub-expressions and natural lan­guage word sequences, by extending a previ­ous model proposed for parsing natural language into meaning representations (Lu et al., 2008).</p><p>To our best knowledge, this is the first probabilis­tic model for generating sentences from the lambda calculus encodings of their underlying formal mean­ing representations, that concerns both surface real­ization and lexical acquisition. We demonstrate the effectiveness of our model in Section 5.</p></section><section number="2" title="Related Work"><p>The task of language generation from logical forms has a long history. Many early works do not rely on probabilistic approaches. Wang (1980) presented an approach for generation from an extended predicate logic formalism using hand-written rules. Shieber et al. (1990) presented a semantic head-driven ap­proach for generation from logical forms based on rules written in Prolog. Shemtov (1996) presented a system for generation of multiple paraphrases from ambiguous logical forms. Langkilde (2000) pre­sented a probabilistic model for generation from a packed forest meaning representation, without con­cerning lexical acquisition. Specifically, we are not aware of any prior work that handles both automatic unsupervised lexical acquisition and surface realiza­tion for generation from logical forms in a single framework.</p><p>Another line of research efforts focused on the task of language generation from other meaning rep­resentation formalisms. Wong and Mooney (2007a) as well as Chen and Mooney (2008) made use of synchronous grammars to transform a variable­free tree-structured meaning representation into sen­tences. Lu et al. (2009) presented a language gener­ation model using the same meaning representation based on tree conditional random fields. Angeli et al. (2010) presented a domain-independent proba­bilistic approach for generation from database en­tries. All these models are probabilistic models.</p><p>Recently there are also substantial research efforts on the task of mapping natural language to meaning representations in various formalisms - the inverse task of language generation called <i>semantic parsing.</i><i> </i>Examples include Zettlemoyer and Collins (2005; 2007; 2009), Kate and Mooney (2006), Wong and Mooney (2007b), Lu et al. (2008), Ge and Mooney (2009), as well as Kwiatkowski et al. (2010).</p><p>Of particular interest is our prior work Lu et al. (2008), in which we presented a joint generative pro­cess that produces a hybrid tree structure containing words, syntactic structures, and meaning represen­tations, where the meaning representations are in a variable-free tree-structured form. One important property of the model in our prior work is that it induces a hybrid tree structure automatically in an unsupervised manner, which reveals the correspon­dences between natural language word sequences and semantic elements. We extend our prior model in the next section, so as to support X-expressions. The model in turn serves as the basis for inducing the synchronous grammar rules later.</p></section><section number="3" title="A -Hybrid Tree"><p>In Lu et al. (2008), a generative model was pre­sented to model the process that jointly generates both natural language sentences and their underly­ing meaning representations of a variable-free tree-structured form. The model was defined over a <i>hybrid tree, </i>which consists of meaning representa­tion tokens as internal nodes and natural language words as leaves. One limitation of the hybrid tree model is that it assumes a single fixed tree struc­ture for the meaning representation. However, X-expressions exhibit complex structures and variable dependencies, and thus it is not obvious how to rep­resent them in a single tree structure.</p><p>In this section, we present a novel X<i>-hybrid tree </i>model that provides the following extensions over the model of Lu et al. (2008):</p><p>1. The internal nodes of a meaning representation tree involve X-expressions which are not neces­sarily of variable-free form;</p><p>2. The meaning representation has a packed forest representation, rather than a single determinis­tic tree structure.</p><subsection number="3.1" title="Packed X -Meaning Forest"><p>We represent a X-expression with a packed forest of meaning representation trees (called X<i>-meaning for-</i> <i>est).</i><page local="3" global="1613"/><i> </i>Multiple different meaning representation trees (called <i>X-meaning trees) </i>can be extracted from the same X-meaning forest, but they all convey equiva­lent semantics via reductions, as discussed next.</p><p>Constructing a X-meaning forest for a given X-expression requires decomposition of a complete X-expression into semantically complete and syntacti­cally correct sub-expressions in a principled man­ner. This can be achieved with a process called higher order unification (Huet, 1975). The process was known to be very complex and was shown to be undecidable in unrestricted form (Huet, 1973). Re­cently a restricted form of higher order unification was applied to a semantic parsing task (Kwiatkowski et al., 2010). In this work, we employ a similar tech­nique for building the X-meaning forest.</p><p>For a given X-expression e, our algorithm finds ei­ther two expressions h and f such that (h f ) = e, or three expressions h, f, and g such that ((h f ) g) = e, where the symbol = is interpreted as <i>a-equivalent </i>after <i>reductions<footnote anchor="1"/> </i>(Barendregt, 1985). We then build the X-meaning forest based on the expressions h, f, and g. In practice, we develop a BUILDFOREST(e) procedure which recursively builds X-forests by ap­plying restricted higher-order unification rules on top of the X-expression e. Each node of the X-forest is called a X-production, to which we will give more details in Section 3.2. For example, once a candi­date triple (h, f, g) as in ((h f) g) = e has been identified, the procedure creates a X-forest with the root node being a X-production involving h, and two sets of child X-forests given by BUILDFOREST(f) and BUILDFOREST(g) respectively. For restricted higher-order unification, besides the similar assump­tions made by Kwiatkowski et al. (2010), we also impose one additional assumption: limited free vari­able, which states that the expression h must contain no more than one free variable. Note that this pro­cess provides a semantically equivalent packed for­est representation of the original X-expression, with­out altering its semantics in any way.</p><footnote label="1">In this work, for reductions, we consider a-conversions (changing bound variables) and ß-conversions (applying func­tors to their arguments).</footnote><p>ample, the expression Xx.state(x) A loc(6osion, x) can be represented as the functional application form of [Xf.Xx.f (x) A loc(6osion, x)] &lt; Xx.state(x).<footnote anchor="2"/></p><p>Such a packed forest representation contains ex­ponentially many tree structures which all convey the same semantics. We believe such a semantic representation is more advantageous than the sin­gle fixed tree-structured representation. In fact, one could intuitively regard a different decomposition path as a different way of interpreting the same se­mantics. Thus, such a representation could poten­tially accommodate a wider range of natural lan­guage expressions, which all share the same seman­tics but with very different word choices, phrase or-derings, and syntactic structures (like paraphrases). It may also alleviate the non-isomorphism issue that was commonly faced by researchers when mapping meaning representations and sentences (Wong and Mooney, 2007b). We will validate our belief later through experiments.</p></subsection><subsection number="3.2" title="The Joint Generative Process"><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">I</doubt><doubt alpha="57.1" length="14" tooSmall="False" monospace="0.0">Ta :Ta&lt; Tb&lt; Tc</doubt><doubt alpha="65.2" length="23" tooSmall="False" monospace="0.0">WlTb:1Tb&lt; TdW2Tc: TicW3</doubt><doubt alpha="13.3" length="15" tooSmall="False" monospace="0.0">...       W4.W5</doubt><p>Figure 1: The joint generative process of both A-meaning tree and its corresponding natural language sentence, which results in a A-hybrid tree.</p><p>The generative process for a sentence together with its corresponding X-meaning tree is illustrated in Figure 1, which results in a <i>X-hybrid tree. </i>Internal nodes of a X-hybrid tree are called <i>X-productions, </i>which are building blocks of a X-forest. Each X-production in turn has at most two child X-productions. A X-production has the form <i>ra : na &lt; </i>Tb, where Ta is the expected <i>type<footnote anchor="3"/> </i>after type evalu­ation of the terms to its right, na is a X-expression (serves as the functor), and Tb are types of the child X-productions (as the arguments). The leave nodes</p><footnote label="2">Throughout this paper, we abuse this notation a bit by al­lowing the arguments to be types rather than actual expressions, such as Ay.Ax.loc(y, x)) &lt; e, which indicates that the functor Ay.Ax.loc(y, x) expects an expression of type e to serve as its argument.</footnote><footnote label="3">This work considers basic types: e (entities) and t (truth values). It also allows function types, e.g., (e, t) is the type assigned to functions that map from entities to truth values.</footnote><p>For better readability, we introduce the symbol &lt; as an alternative notation for functional appli­cation. In other words, h &lt; f refers to (h f ) or h(f ), and h &lt; f &lt; g refers to ((h f ) g). For ex-<page local="4" global="1614"/></p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">I</doubt><doubt alpha="40.0" length="10" tooSmall="False" monospace="0.0">r:&lt;e»t&gt;[T]</doubt><doubt alpha="85.7" length="7" tooSmall="False" monospace="0.0">give me</doubt><doubt alpha="39.2" length="51" tooSmall="False" monospace="0.0">(e, t&gt; g :Xg.Xf.Xx.g(x) Af (x) &lt; (e, t&gt;rf] &lt; (e,t&gt;^</doubt><doubt alpha="61.9" length="21" tooSmall="False" monospace="0.0">(e,t^jr :Xx.state(x)I</doubt><p>the states states that <i>:</i><i> miss-v </i>runs through the mississippi</p><doubt alpha="37.6" length="149" tooSmall="False" monospace="0.0">(e,      :Xf.Xg.Xx3y.g(y) A(f(x) y)&lt; (e, (e, t»rf] &lt; (e,t&gt;^ (e, (e, t»r^ : Xy.Xx.nextJbo(x, y)    (e, t&gt;:Xg.Xf .Xx.g(x) A f (x) &lt; (e, t&gt;^^ &lt; (e, t&gt;g]</doubt><doubt alpha="57.6" length="59" tooSmall="False" monospace="0.0">bordering(e,t&gt;^:Xx.state(x)    (e,t&gt;^j:Xy.Xx.loc(y, x) &lt;eQj</doubt><p>Figure 2: One example A-hybrid tree for the sentence "give me the states bordering states that the mississippi runs through" together with its logical form "Axo.state(xo) A <i>3xl.[loc(miss-r,xl) A state(xl) </i>A <i>nextJto(xl, </i>xo)]".</p><p>w are contiguous word sequences. The model re­peatedly generates X<i>-hybrid sequences, </i>which con­sist of words intermixed with X-productions, from each X-production at different levels.</p><p>Consider part of the example X-hybrid tree in Fig­ure 2. The probability associated with generation of the subtree that spans the sub-sentence "that the mis­sissippi runs through" can be written as:</p><p><i>P (^Ax.loc(missjr, x), </i>that the mississippi runs through</p><doubt alpha="61.3" length="93" tooSmall="False" monospace="0.0">=&lt;\&gt;(m—wYw|pl)xi/&gt;(that eg runs through|pl)xp(p2\pl,argl)x&lt;\&gt;(m—w|p2)xi/&gt;(the mississippi|p2)</doubt><p>where <i>p1 </i>= e : miss_r.</p><doubt alpha="45.7" length="35" tooSmall="False" monospace="0.0">(e, t) : Xy.Xx.loc(y, x) &lt; e^,andp2</doubt><p>Following the work of Lu et al. (2008), the gener­ative process involves three types of parameters <i>Ö = </i>(0, p}: 1) pattern parameters 0, which model in what way the words and child X-productions are in­termixed; 2) emission parameters which model the generation process of words from X-productions, where either a unigram or a bigram assumption can be made (Lu et al., 2008); and 3) meaning repre­sentation (MR) model parameters p, which model the generation process from one X-production to its child X-productions. An analogous inside-outside algorithm (Baker, 1979) used there is employed here. Since we allow a packed X-meaning forest rep­resentation rather than a fixed tree structure, the MR model parameters p in this work should be estimated with the inside-outside algorithm as well, rather than being estimated directly from the training data by simple counting, as was done in Lu et al. (2008).</p></subsection></section><section number="4" title="The Language Generation Algorithm"><p>Now we present the algorithm for language gener­ation. We introduce the grammar first, followed by the features we use. Next, we present the method for grammar induction, and then discuss the decoder.</p><subsection number="4.1" title="The Grammar"><p>We use a weighted synchronous context free gram­mar (SCFG) (Aho and Ullman, 1969), which was previously used in Chiang (2007) for hierarchical phrase-based machine translation. The grammar is defined as follows:</p><doubt alpha="20.0" length="20" tooSmall="False" monospace="0.0">T —    (p\ ,hw,~)(1)</doubt><p>where T is the type associated with the X-production pX <footnote anchor="4"/>, and hw is a sequence consisting of natural lan­guage words intermixed with types. The symbol ~ denotes the one-to-one correspondence between nonterminal occurrences (i.e., in this case types of X-expressions) in both pX and hw.</p><p>We allow a maximum of two nonterminal sym­bols in each synchronous rule, as was also assumed in Chiang (2007), which makes the grammar a bi­nary SCFG. Two example rules are:</p><doubt alpha="62.3" length="77" tooSmall="False" monospace="0.0">(e,t) — (\y.Ax.loc(y,x)&lt;,that eg runs through^)e   — (missjr,the mississippi^</doubt><p>where the boxed indices give the correspondences between nonterminals.</p><p>A derivation with the above two synchronous rules results in the following X-expression paired with its natural language counterpart:</p><footnote label="4">Since type is already indicated by T, we avoid redundancy by omitting it when writing p\, without loss of information.</footnote><page local="5" global="1615"/><doubt alpha="66.7" length="6" tooSmall="False" monospace="0.0">Type1:</doubt><doubt alpha="57.1" length="7" tooSmall="False" monospace="0.0">Type 2:</doubt><doubt alpha="57.1" length="7" tooSmall="False" monospace="0.0">Type 3:</doubt><doubt alpha="31.2" length="16" tooSmall="False" monospace="0.0">(e, (e,t)) (e,t)</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">—</doubt><doubt alpha="26.7" length="15" tooSmall="False" monospace="0.0">(e,t) — (e,t) —</doubt><doubt alpha="64.5" length="31" tooSmall="False" monospace="0.0">^\y.\x.nextJo(x, y) ,borderingy</doubt><doubt alpha="36.9" length="65" tooSmall="False" monospace="0.0">v\g.\f.\x.g(x) A f(x) &lt; (e, t)rj &lt; (e, t)^ ,  (e, t)rg(e, t) rrj)</doubt><p><i>\x.loc(miss_r, x) A state(x) , </i>states that the mississippi runs through ) <i>\x.loc(missjr, x) , </i>that the mississippi runs through^ (e, t) <i>— </i>^<i>Ax.</i><i>loc(rnissjr, x) , </i>that the mississippi runs through^)</p><doubt alpha="59.1" length="171" tooSmall="False" monospace="0.0">(e, t) —s-(Xf.Xx.state(x)A3y.[f(y) A nextJto(y,x)\ &lt;(e,t)rrj,the states bordering(e,t)rrj (e,t)     —     (^Xy.Xx.loc(y, x) A state(x) &lt;erjj,states that erf] runs through^</doubt><figure caption="Figure 3: Example synchronous rules that can be extracted from the A-hybrid tree of Figure 2."></figure><p>where the source side A-expression is constructed from the application <i>Ay.Ax.loc(y, </i><i>x)</i><i> &lt; miss-V </i>fol­lowed by a reduction <i>(ß</i>-conversion). Assuming the A-expression to be translated is <i>Ax.loc(miss-V,x), </i>the above rule in fact gives one candidate translation "that the mississippi runs through".</p></subsection><subsection number="4.2" title="Features"><p>Following the work of Chiang (2007), we assign scores to derivations with a log-linear model, which are essentially weighted products of feature values.</p><p>For generality, we only consider the following four simple features in this work:</p><p>2.</p><p>3.</p><p>4.</p><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">w(D)</doubt><doubt alpha="54.5" length="11" tooSmall="False" monospace="0.0">(nn/i(r)wi)</doubt><doubt alpha="55.6" length="9" tooSmall="False" monospace="0.0">\reD   iI</doubt><doubt alpha="66.7" length="15" tooSmall="False" monospace="0.0">/i(r)wMxplm(S)v</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(2)</doubt><p>where <i>v </i><i>G D </i>refers to a rule <i>v </i>that appears in the derivation <i>D, ss </i>is the target side (sentence) as­sociated with the derivation D, and <i>fi </i>is a rule-specific feature (one of features 1-3 above) which is weighted with <i>w</i><i>i.</i><i> </i>The language model feature is weighted with <i>wLM</i>.</p><p>Once the feature values are computed, our goal is to find the optimal weight vector <i>W* </i>that maximizes a certain evaluation metric when used for decoding, as we will discuss in Section 4.4.</p><p>Following popular approaches to learning feature weights in the machine translation community (Och and Ney, 2004; Chiang, 2005), we use the minimum error rate training (MERT) (Och, 2003) algorithm to learn the feature weights that directly optimize cer­tain automatic evaluation metric. Specifically, the Z-MERT (Zaidan, 2009) implementation of the al­gorithm is used in this work.</p></subsection><subsection number="4.3" title="Grammar Induction"><p>Automatic induction of the grammar rules as de­scribed above from training data (which consists of pairs of <i>A</i>-expressions and natural language sen­tences) is a challenging task. Current state-of-the-art string-based translation systems (Koehn et al., 2003; Chiang, 2005; Galley and Manning, 2010) typically begin with a word-aligned corpus to con­struct phrasal correspondences. Word-alignment in­formation can be estimated from alignment models, such as the IBM alignment models (Brown et al., 1993) and HMM-based alignment models (Vogel et al., 1996; Liang et al., 2006). However, unlike texts, logical forms have complex internal structures and variable dependencies across sub-expressions. It is not obvious how to establish alignments between logical terms and texts with such alignment models.</p><p>Fortunately, the generative model for <i>A</i>-hybrid tree introduced in Section 3 explicitly models the mappings from <i>A</i>-sub-expressions to (possibly dis­contiguous) word sequences with a joint genera­tive process. This motivates us to extract grammar rules from the A-hybrid trees. Thus, we first find the Viterbi <i>A</i>-hybrid trees for all training instances,</p><p>1. <i>p(hw\p\): </i>the relative frequency estimate of a hybrid sequence <i>hw </i>given the A-production <i>p\; </i> <i>p(p</i><i>x</i><i>\hw</i><i>,t</i>): the relative frequency estimate of a A-production <i>p\ </i>given the phrase <i>hw </i>and the type <i>t </i>; exp(—<i>wc(hw</i>)): the number of words generated, where <i>wc(hw) </i>refers to the number of words in <i>h</i><i>w </i>(i.e., word penalty); and  <i>Plm</i>(s): the language model score of the gen­erated sentence s. The first three features, which are also widely used in state-of-the-art machine translation models (Koehn et al., 2003; Chiang, 2007), are rule-specific and thus can be computed before decoding. The last feature is computed during the decoding phase in combination with the sibling rules used. We score a derivation <i>D </i>with a log-linear model:</p><page local="6" global="1616"/><doubt alpha="100.0" length="6" tooSmall="False" monospace="0.0">Source</doubt><doubt alpha="100.0" length="6" tooSmall="False" monospace="0.0">Target</doubt><doubt alpha="58.4" length="250" tooSmall="False" monospace="0.0">:     (substitution)Xy'. \Xg.Xf.Xx.g(x) A f(x)&lt; [Xy.Xx.loc(y,x)&lt;y']&lt; Xx.state(x)^ &lt;epg (two ß-conversions)=&gt; Xy'.[Xf .Xx.loc(y', x)af(x)&lt; Xx.state(x)] &lt;erg (ß-conversion)=&gt;Xy' .Xx.loc(y',x)A state(x) &lt;ejg (a-conversion)=&gt;Xy.Xx.loc(y,x)A state(x) &lt;erg</doubt><p>:    "states that <i>eg </i>runs through"</p><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">Rule</doubt><doubt alpha="66.2" length="68" tooSmall="False" monospace="0.0">(e,t)— {Xy.Xx.loc(y,x)A state(x) &lt;erg , states that erg runs through</doubt><p>Figure 4: Construction of a two-level A-hybrid sequence rule via substitution and reductions from a tree fragment. Note that the subtree rooted by erg : <i>missjr </i>gets "abstracted" by its type e. The auxiliary variable y' of type e is thus introduced to facilitate the construction process.</p><p>based on the learned parameters of the generative A­hybrid tree model.</p><p>Next, we extract grammar rules on top of these A-hybrid trees. Specifically, we extract the follow­ing three types of synchronous grammar rules, with examples given in Figure 3:</p><p>g. A-hybrid sequence rules: They are the conven­tional rules constructed from one A-production and its corresponding A-hybrid sequence.</p><p>2. Subtree rules: These rules are constructed from a complete subtree of the A-hybrid tree. Each rule provides a mapping between a complete sub-expression and a contiguous sub-sentence.</p><p>3. Two-level A-hybrid sequence rules: These rules are constructed from a tree fragment with one of its grandchild subtrees (the subtree rooted by one of its grandchild nodes) being abstracted with its type only. These rules are constructed via substitution and reductions.</p><p>Figure 4 gives an example based on a tree frag­ment of the A-hybrid tree in Figure 2. Note that the first step makes use of the auxiliary vari­able <i>y'</i><i> </i>of type e to represent the grandchild subtree. Ay' is introduced so as to allow any A-expression of type e serving as this expres­sion's argument to replace y'. In fact, if the semantics conveyed by the grandchild subtree serves as its argument, we will obtain the exact complete semantics of the current subtree. As we can see, the resulting rule is more general, and is able to capture longer structural depen­dencies. Such rules are thus potentially more useful.</p><p>The overall algorithm for learning the grammar rules is sketched in Figure 5.</p></subsection><subsection number="4.4" title="Decoding"><p>Our goal in decoding is to find the most probable sentence <i>s </i>for a given A-expression e:</p><p><i>s = </i>s<b>i   </b>arg max <b>w</b>(D) <b>I</b></p><doubt alpha="46.7" length="15" tooSmall="False" monospace="0.0">\ds.t.e(D) = eJ</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(3)</doubt><p>where <i>e(D) </i>refers to the source side (A-expression) of the derivation D, and s(D) refers to the target side (natural language sentence) of D.</p><p>A conventional CKY-style decoder as used by Chiang (2007) is not applicable to this work since the source side does not exhibit a linear structure. As discussed in Section 3.g, A-expressions are rep­resented as packed A-meaning forests. Thus, in this work, we make use of a bottom-up dynamic programming chart-parsing algorithm that works di­rectly on translating forest nodes into target natural language words. The algorithm is similar to that of Langkilde (2000) for generation from an underly­ing packed semantic forest. Language models are incorporated when scoring the n-best candidates at each forest node, where the cube-pruning algorithm of Chiang (2007) is used. In order to accommodate type 2 and type 3 rules as discussed in Section 4.3, whose source side A-productions are not present in the nodes of the original A-meaning forest, new A-productions are created (via substitution and reduc­tions) and attached to the original A-meaning forest.</p><page local="7" global="1617"/><p><b>Procedures</b></p><p>• <i>f </i><i>—</i><i> </i>BUILDFOREST(e)</p><p>It takes in a A-expression e and outputs its A-meaning forest f. (Sec. 3.g)</p><p>• é&gt; — TRAINGENMODEL<b>(f, s)</b></p><p>It takes in A-meaning forest-sentence pairs <b>(f, s), </b>performs EM training of the generative model, and outputs the parameters <i>9. </i>(Sec. 3.2)</p><p>• h <i>—</i><i> </i>FINDHYBRIDTREE(f, <i>s,9)</i></p><p>It finds the most probable A-hybrid tree h contain­ing the given f-s pair, under the generative model parameters 9 . (Sec. 4.3)</p><p>• <i>T</i><i>h</i><i> </i><i>—</i><i> </i>ExtractRules (h)</p><p>It takes in a A-hybrid tree h, and extracts a set of grammar rules T<i>h</i><i> </i>out of it. (Sec. 4.3) <b>Algorithm</b></p></subsection></section><section number="1." title="Inputs and initializations:"><doubt alpha="65.9" length="44" tooSmall="False" monospace="0.0">•A training set(e, s),an empty rule set T =0</doubt></section><section number="2." title="Learn the grammar:"><p>• For each ei e <b>e, </b>find its A-meaning forest: fi = BuILDFoREST(ei). This gives the set</p><p><b>(f, s).</b></p><p>• Learn the generative model parameter :</p><p>Ö* = TRAINGENMODEL<b>(f, s).</b></p><p>• For each (fi; si) e <b>(f, s), </b>find the most proba­ble A-hybrid tree h<i>i, </i>and then extract the gram­mar rules from it:</p><p>hi = FINDHYBRIDTREE(fi,si,ö*) T = T U EXTRACTRULES(hi)</p><p>3. Output the learned grammar rule set T. Figure 5: The algorithm for learning the grammar rules</p></section><section number="5" title="Experiments"><p>For experiments, we evaluated on the Geoquery dataset, which consists of 880 queries on U.S. geog­raphy. The dataset was manually labeled with A-expressions as their semantics in Zettlemoyer and Collins (2005). It was used in many previous re­search efforts on semantic parsing (Zettlemoyer and Collins, 2005; Wong and Mooney, 2006; Zettle­moyer and Collins, 2007; Kwiatkowski et al., 2010). The original dataset was annotated with English sen­tences only. In order to assess the generation per­formance across different languages, in our work the entire dataset was also manually annotated with Chinese by a native Chinese speaker with linguistics background<footnote anchor="5"/>.</p><p>For all the experiments we present in this sec­tion, we use the same split as that of Kwiatkowski et al. (2010), where 280 instances are used for test­ing, and the remaining instances are used for learn­ing. We further split the learning set into two por­tions, where 500 instances are used for training the models, which includes induction of grammar rules, training a language model, and computing feature values, and the remaining 100 instances are used for tuning the feature weights.</p><footnote label="5">The annotator created annotations with both A-expressions and corresponding English sentences available as references.</footnote><p>As we have mentioned earlier, we are not aware of any previous work that performs generation from formal logical forms that concerns both lexical ac­quisition and surface realization. The recent work by Angeli et al. (2010) presented a generation sys­tem from database records with an additional focus on content selection (selection of records and their subfields for generation). It is not obvious how to adopt their algorithm in our context where content selection is not required but the more complex log­ical semantic representation is used as input. Other earlier approaches such as the work of Wang (1980) and Shieber et al. (1990) made use of rule-based approaches without automatic lexical acquisition.</p><p>We thus compare our system against two state-of-the-art machine translation systems: a phrase-based translation system, implemented in the Moses toolkit (Koehn et al., 2007)<footnote anchor="6"/>, and a hierarchical phrase-based translation system, implemented in the Joshua toolkit (Li et al., 2009), which is a reim­plementation of the original Hiero system (Chiang, 2005; Chiang, 2007). The state-of-the-art unsuper-vised Berkeley aligner (Liang et al., 2006) with de­fault setting is used to construct word alignments. We train a trigram language model with modified Kneser-Ney smoothing (Chen and Goodman, 1996) from the training dataset using the SRILM toolkit (Stolcke, 2002), and use the same language model for all three systems. We use an n-best list of size 100 for all three systems when performing MERT.</p><subsection number="5.1" title="Automatic Evaluation"><p>For automatic evaluation, we measure the original IBM BLEU score (Papineni et al., 2002) (4-gram precision with brevity penalty) and the TER score (Snover et al., 2006) (the amount of edits required to change a system output into the reference)<footnote anchor="7"/>. Note that TER measures the translation error rate, thus a smaller score indicates a better result.<page local="8" global="1618"/> For clarity, we report 1—TER scores. Following the tuning pro­cedure as conducted in Galley and Manning (2010), we perform MERT using BLEU as the metric.</p><footnote label="6">We used the default settings, and enabled the default lexi-calized reordering model, which yielded better performance. 7 We used tercom version 0.7.25 with the default settings.</footnote><p>We compare our model against state-of-the-art statistical machine translation systems. As a base­line, we first conduct an experiment with the fol­lowing naive approach: we treat the A-expressions as plain texts. All the bound variables (e.g., x in <i>Ax.state(x)) </i>which do not convey semantics are removed, but free variables (e.g., <i>state </i>in <i>Ax.state(x)) </i>which might convey semantics are left intact. Quantifiers and logical connectives are also left intact. While this naive approach might not ap­pear very sensible, we merely want to treat it as our simplest baseline.</p><p>Alternatively, analogous to the work of Wong and Mooney (2007a), we could first parse the A-expressions into binary tree structures with a deter­ministic procedure, and then linearize the tree struc­ture as a sequence. Since there exists different ways to linearize a binary tree, we consider preorder, in­order, and postorder traversal of the trees, and lin­earize them in these three different ways.</p><p>As for our system, during the grammar learning phase, we initialize the generative model parame­ters with output from the IBM alignment model 1 (Brown et al., 1993)<footnote anchor="8"/>, and run the A-hybrid tree gen­erative model with the unigram emission assumption for 10 iterations, followed by another 10 iterations with the bigram assumption. Grammar rules are then extracted based on the <i>A</i>-hybrid trees obtained from such learned generative model parameters.</p><p>Since MERT is prone to search errors, we run each experiment 5 times with randomly initialized fea­ture weights, and report the averaged scores. Ex­perimental results for both English and Chinese are presented in Table 1. As we can observe, the way that a meaning representation tree is linearized has a significant impact on the translation performance. Interestingly, for both Moses and Joshua, the pre-order setting yields the best performance for En­glish, whereas it is inorder that yields the best per­formance for Chinese. This is perhaps due to the fact that Chinese presents a very different syntactic structure and word ordering from English.</p><footnote label="8">We assume word unigrams are generated from free vari­ables, quantifiers, and logical connectives in IBM model 1.</footnote><p>Our system, on the other hand, employs a packed forest representation for <i>A</i>-expressions. Therefore, it eliminates the ordering constraint by encompass­ing exponentially many possible tree structures dur­ing both the alignment and decoding stage. As a result, our system obtains significant improvements in both BLEU and 1—TER using the significance test under the paired bootstrap resampling method of Koehn (2004). We obtain <i>p &lt; </i>0.01 for all cases, except when comparing against Joshua-preorder for English, where we obtain p &lt; 0.05 for both metrics.</p><p>Table 1: Performance on generating English and Chinese from A-expressions with automatic evaluation metrics (we report per­centage scores).</p></subsection><subsection number="5.2" title="Human Evaluation"><p>We also conducted human evaluation with 5 eval-uators each on English and Chinese. We randomly selected about 50% (139) test instances and obtained output sentences from the three systems. Moses and Joshua were run with the top-performing settings in terms of automatic metrics (i.e., preorder for En­glish and inorder for Chinese). Following Angeli et al. (2010), evaluators are instructed to give scores based on language fluency and semantic correctness, on the following scale:</p><p>For each test instance, we first randomly shuffled the output sentences of the three systems, and pre­sented them together with the correct reference to the evaluators. The evaluators were then asked to score all the output sentences at once. This eval­uation process not only ensures that the annotators have no access to which system generated the out-<page local="9" global="1619"/></p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>English</p></td><td class="cell"><p>Chinese</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Bleu</p></td><td class="cell"><p>1-Ter</p></td><td class="cell"><p>Bleu</p></td><td class="cell"><p>1-Ter</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>text</p></td><td class="cell"><p>48.93</p></td><td class="cell"><p>61.08</p></td><td class="cell"><p>43.23</p></td><td class="cell"><p>51.71</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>preorder Moses       . , inorder</p></td><td class="cell"><p><i>51.13</i></p></td><td class="cell"><p><i>63.73</i></p></td><td class="cell"><p>42.08</p></td><td class="cell"><p>50.43</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>46.72</p></td><td class="cell"><p>57.59</p></td><td class="cell"><p><i>48.03</i></p></td><td class="cell"><p><i>55.29</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>postorder</p></td><td class="cell"><p>44.30</p></td><td class="cell"><p>55.05</p></td><td class="cell"><p>46.36</p></td><td class="cell"><p>54.59</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>text</p></td><td class="cell"><p>37.40</p></td><td class="cell"><p>48.97</p></td><td class="cell"><p>36.60</p></td><td class="cell"><p>46.20</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>preorder Joshua       . , inorder</p></td><td class="cell"><p><i>51.40</i></p></td><td class="cell"><p><i>64.69</i></p></td><td class="cell"><p>40.05</p></td><td class="cell"><p>49.70</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>40.31</p></td><td class="cell"><p>50.47</p></td><td class="cell"><p><i>48.32</i></p></td><td class="cell"><p><i>54.64</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>postorder</p></td><td class="cell"><p>31.10</p></td><td class="cell"><p>42.44</p></td><td class="cell"><p>41.31</p></td><td class="cell"><p>49.71</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>This work (t)</p></td><td class="cell"><p><b>54.58</b></p></td><td class="cell"><p><b>67.65</b></p></td><td class="cell"><p><b>55.11</b></p></td><td class="cell"><p><b>63.77</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(t) w/o type 2 rules</p></td><td class="cell"><p>53.77</p></td><td class="cell"><p>66.43</p></td><td class="cell"><p>54.30</p></td><td class="cell"><p>62.49</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(t) w/o type 3 rules</p></td><td class="cell"><p>53.68</p></td><td class="cell"><p>66.17</p></td><td class="cell"><p>50.96</p></td><td class="cell"><p>60.13</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Score</p></td><td class="cell"><p>Language Fluency</p></td><td class="cell"><p>Semantic Correctness</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>5</p></td><td class="cell"><p>Flawless</p></td><td class="cell"><p>Perfect</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>4</p></td><td class="cell"><p>Good</p></td><td class="cell"><p>Near Perfect</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3</p></td><td class="cell"><p>Non-native</p></td><td class="cell"><p>Minor Errors</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2</p></td><td class="cell"><p>Disfluent</p></td><td class="cell"><p>Major Errors</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p>Gibberish</p></td><td class="cell"><p>Completely Wrong</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Table 2: Human evaluation results on English and Chinese generation. Flu: language fluency; Sem: semantic correctness.</p><p>put, but also minimizes bias associated with scor­ing different outputs for the same input. The de­tailed and averaged results (with one standard devi­ation) for human evaluation are presented in Table 2 for English and Chinese respectively. For both lan­guages, our system achieves a significant improve­ment over Moses and Joshua (p &lt; 0.01 with paired t-tests), in terms of both language fluency and se­mantic correctness. This set of results is important, as it demonstrates that our system produces more fluent texts with more accurate semantics when per­ceived by real humans.</p></subsection><subsection number="5.3" title="Additional Experiments"><p>We also performed the following additional experi­ments. First, we attempted to increase the number of EM iterations (to 100) when training the model with the bigram assumption, so as to assess the ef­fect of the number of EM iterations on the final gen­eration performance. We observed similar perfor­mance. Second, in order to assess the importance of the two types of novel rules - subtree rules (type 2) and two-level <i>A</i>-hybrid sequence rules (type 3), we also conducted experiments without these rules for generation. Experiments show that these two types of rules are important. Specifically, type 3 rules, which are able to capture longer structural depen­dencies, are of particular importance for generating Chinese. Detailed results for these additional exper­iments are presented in Table 1.</p></subsection><subsection number="5.4" title="Experiments on Variable-free Meaning Representations"><p>Finally, we also assess the effectiveness of our model on an alternative meaning representation for­malism in the form of variable-free tree structures. Specifically, we tested on the Robocup dataset (Kuhlmann et al., 2004), which consists of 300 English instructions for coaching robots for soccer games, and a variable-free version of the Geo-query dataset. These are the standard datasets used in the generation tasks of Wong and Mooney (2007a) and Lu et al. (2009). Similar to the tech­nique introduced in Kwiatkowski et al. (2010), our proposed algorithm could still be applied to such datasets by writing the tree-structured representa­tions as function-arguments forms. The higher order unification-based decomposition algorithm could be applied on top of such forms accordingly. For exam­ple, <i>midfield(opp) = Ax.midfield(x) </i>&lt; opp. See Kwiatkowski et al. (2010) for more details. How­ever, since such forms present monotonous struc­tures, and thus give less alternative options in the higher-order unification-based decomposition pro­cess, it prevents the algorithm from creating many disjunctive nodes in the packed forest. It is thus hy­pothesized that the advantages of the packed forest representation could not be fully exploited with such a meaning representation formalism.</p><p>Following previous works, we performed 4 runs of 10-fold cross validation based on the same split as that of Wong and Mooney (2007a) and Lu et al. (2009), and measured standard Bleu percent­age and Nist (Doddington, 2002) scores. For ex­perimentation on each fold, we trained a trigram language model on the training data of that fold, and randomly selected 70% of the training data for grammar induction, with the remaining 30% for learning of the feature weights using Mert. Next, we performed grammar induction with the complete training data of that fold, and used the learned feature weights for decoding of the test in­stances. The averaged results are shown in Ta­ble 3. our approach outperforms the previous sys­tem Wasp-1++ (Wong and Mooney, 2007a) sig­nificantly, and achieves comparable or slightly bet­ter performance as compared to Lu et al. (2009). This set of results is particularly striking. We note<page local="10" global="1620"/></p><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>English</p></td><td class="cell"><p>Judge E1</p><p>Flu Sem</p></td><td class="cell"><p>Judge E2</p><p>Flu Sem</p></td><td class="cell"><p>Judge E3</p><p>Flu Sem</p></td><td class="cell"><p>Judge E4</p><p>Flu Sem</p></td><td class="cell"><p>Judge E5</p><p>Flu Sem</p></td><td class="cell"><p>Average Flu Sem</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Moses preorder</p></td><td class="cell"><p>4.56</p></td><td class="cell"><p>4.57</p></td><td class="cell"><p>4.58</p></td><td class="cell"><p>4.54</p></td><td class="cell"><p>4.52</p></td><td class="cell"><p>4.52</p></td><td class="cell"><p>4.48</p></td><td class="cell"><p>4.14</p></td><td class="cell"><p>4.28</p></td><td class="cell"><p>4.22</p></td><td class="cell"><p>4.48 ± 0.12</p></td><td class="cell"><p>4.40 ± 0.20</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Joshua preorder</p></td><td class="cell"><p>4.50</p></td><td class="cell"><p>4.43</p></td><td class="cell"><p>4.49</p></td><td class="cell"><p>4.29</p></td><td class="cell"><p>4.44</p></td><td class="cell"><p>4.36</p></td><td class="cell"><p>4.46</p></td><td class="cell"><p>4.04</p></td><td class="cell"><p>4.12</p></td><td class="cell"><p>4.06</p></td><td class="cell"><p>4.40 ± 0.16</p></td><td class="cell"><p>4.24 ± 0.18</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>This work</p></td><td class="cell"><p><b>4.76</b></p></td><td class="cell"><p><b>4.73</b></p></td><td class="cell"><p><b>4.73</b></p></td><td class="cell"><p><b>4.70</b></p></td><td class="cell"><p><b>4.68</b></p></td><td class="cell"><p><b>4.60</b></p></td><td class="cell"><p><b>4.64</b></p></td><td class="cell"><p><b>4.37</b></p></td><td class="cell"><p><b>4.49</b></p></td><td class="cell"><p><b>4.44</b></p></td><td class="cell"><p><b>4.66 </b>± 0.10</p></td><td class="cell"><p><b>4.57 </b>± 0.16</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Chinese</p></td><td class="cell"><p>Judge C1</p><p>Flu Sem</p></td><td class="cell"><p>Judge C2</p><p>Flu Sem</p></td><td class="cell"><p>Judge C3</p><p>Flu Sem</p></td><td class="cell"><p>Judge C4</p><p>Flu Sem</p></td><td class="cell"><p>Judge C5</p><p>Flu Sem</p></td><td class="cell"><p>Average Flu Sem</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Moses inorder</p></td><td class="cell"><p>4.38</p></td><td class="cell"><p>4.22</p></td><td class="cell"><p>3.95</p></td><td class="cell"><p>3.99</p></td><td class="cell"><p>4.01</p></td><td class="cell"><p>3.80</p></td><td class="cell"><p>4.27</p></td><td class="cell"><p>4.19</p></td><td class="cell"><p>4.09</p></td><td class="cell"><p>4.01</p></td><td class="cell"><p>4.14 ± 0.18</p></td><td class="cell"><p>4.04 ± 0.17</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Joshua inorder</p></td><td class="cell"><p>4.32</p></td><td class="cell"><p>4.04</p></td><td class="cell"><p>3.74</p></td><td class="cell"><p>3.91</p></td><td class="cell"><p>3.76</p></td><td class="cell"><p>3.55</p></td><td class="cell"><p>4.21</p></td><td class="cell"><p>4.04</p></td><td class="cell"><p>3.96</p></td><td class="cell"><p>3.97</p></td><td class="cell"><p>4.00 ± 0.26</p></td><td class="cell"><p>3.90 ± 0.21</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>This work</p></td><td class="cell"><p><b>4.61</b></p></td><td class="cell"><p><b>4.47</b></p></td><td class="cell"><p><b>4.53</b></p></td><td class="cell"><p><b>4.43</b></p></td><td class="cell"><p><b>4.50</b></p></td><td class="cell"><p><b>4.31</b></p></td><td class="cell"><p><b>4.71</b></p></td><td class="cell"><p><b>4.55</b></p></td><td class="cell"><p><b>4.57</b></p></td><td class="cell"><p><b>4.32</b></p></td><td class="cell"><p><b>4.59 </b>± 0.08</p></td><td class="cell"><p><b>4.42 </b>± 0.10</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Figure 6: Sample English outputs for various datasets. For the variable-present dataset, we also show outputs from Moses and Joshua.</p><p>that the algorithm of Lu et al. (2009) is capable of modeling dependencies over phrases, which gives global optimization over the sentence generated, and works by building conditional random fields (Laf-ferty et al., 2001) over trees. But the algorithm of Lu et al. (2009) is also limited to handling tree-structured meaning representation, and is therefore unable to accept inputs such as the variable ver­sion of <i>A</i>-expressions. our algorithm works well by introducing additional new types of synchronous rules that are able to capture longer range depen­dencies. Wasp-1++, on the other hand, also makes use of a synchronous parsing-based statistical ma­chine translation approach. Their system, however, requires linearization of the tree structure for both alignment and translation. In contrast, our model directly performs alignment and translation from a packed forest representation to a sentence. As a result, though Wasp-1++ made use of additional features (lexical weights), our system yielded bet­ter performance. Sample English output sentences are given in Figure 6.</p></subsection></section><section number="6" title="Conclusions and Future Work"><p>in this work, we presented a novel algorithm for gen­erating natural language sentences from their underlying semantics in the form of typed lambda calcu­lus. We tackled the problem by introducing a novel reduction-based weighted synchronous context-free grammar formalism, which allows sentence genera­tion with a log-linear model. in addition, we pro­posed a novel generative model that jointly gener­ates lambda calculus expressions and natural lan­guage sentences. The model is then used for auto­matic grammar induction. Empirical results show that our model outperforms state-of-the-art machine translation models, for both English and Chinese, in terms of both automatic and human evaluation. Furthermore, we have demonstrated that the model can also effectively handle inputs with a variable­free version of meaning representation.</p><p>We believe the algorithm used for inducing the reduction-based synchronous grammar rules may find applications in other research problems, such as statistical machine translation and phrasal syn­chronous grammar induction. We are interested in exploring further along such directions in the future.</p></section><section title="Acknowledgments"><p>This research was done for CSIDM Project No.</p><p>CSiDM-200804 partially funded by a grant from the National Research Foundation (NRF) adminis­tered by the Media Development Authority (MDA) of Singapore. We would like to thank Tom Kwiatkowski and Luke Zettlemoyer for sharing their dataset, and Omar F. Zaidan for his help with Z-</p><p>MERT.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Variable-present dataset</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>A-expression :</p><p>Reference : Moses : Joshua :</p><p>This work :</p></td><td class="cell"><p><i>argmax(x, river(x) A 3y.[state(y) </i>a <i>nextJ,o(y, indias) </i>a <i>loc(x, y)], len(x))</i></p><p>what is the longest river that flows through a state that borders indiana</p><p>what is the states that border long indiana</p><p>what is the longest river surrounding states border indiana</p><p>what is the longest river in the states that border indiana</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>A-expression : Reference : Moses : Joshua : This work :</p></td><td class="cell"><p><i>density (ix.loc(argmax(y, loc(y, usci-co) A river(y), size(y)), x) A state(x)) </i>which is the density of the state that the largest river in the united states runs through what is the population density in lie on the state with the smallest state in the us what is the population density of states lie on the smallest state in the us what is the population density of the state with the largest river in the us</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Variable-free datasets</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>A-expression : Reference : This work :</p></td><td class="cell"><p><i>populati&lt;on(largest-&lt;one-density (state-all))</i></p><p>what is the population of the state with the highest population density</p><p>how many people live in the state with the largest population density</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>A-expression :</p><p>Reference : This work :</p></td><td class="cell"><p><i>rule(and(bpos(fr&lt;om-goalJine(our, jnum(n0.0, n32.0))),not(bpos(left(penalty_area(our))))),-dont(player_our(n3), intercept))</i></p><p>player 3 should not intercept the ball if the ball is within 32 meters of our goal line and not in our left penalty area</p><p>if the ball is within 32 meters from our goal line and not on the left side of our penalty area then player 3 should not</p><p>intercept it</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 3: Performance on variable-free representations" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Robocup</p><p>Bleu Nist</p></td><td class="cell"><p>Geoquery Bleu Nist</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Wasp-1++</p><p>Lu et al. (2009)</p><p>This work</p></td><td class="cell"><p>60.22 6.8976 62.20 6.9845 <b>62.45 7.0011</b></p></td><td class="cell"><p>53.70 6.4808 57.33 <b>6.7459 57.62 </b>6.6867</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="11" global="1621"/></section><references><p>A.V. Aho and J.D. Ullman. 1969. Syntax directed trans­lations and the pushdown assembler. <i>Journal of Com­puter and System Sciences, </i>3(1):37-56.</p><p>G. Angeli, P. Liang, and D. Klein. 2010. A simple domain-independent probabilistic approach to gener­ation. In <i>Proc. EMNLP, </i>pages 502-512.</p><p>J. K. Baker. 1979. Trainable grammars for speech recog­nition. <i>The Journal </i><i>ofthe</i><i> Acoustical Society ofAmer-</i>ica, 65:S132.</p><p>H. P. Barendregt. 1985. <i>The Lambda Calculus, Its Syntax and Semantics (Studies in Logic and the Foundations of Mathematics, Volume 103). Revised Edition. </i>North-</p><p>Holland.</p><p>P. F. Brown, V. J. Della Pietra, S. A. Della Pietra, and R. L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. <i>Computa­tional Linguistics, </i>19:263-311. S. F. Chen and J. Goodman. 1996. An empirical study of smoothing techniques for language modeling. In <i>Proc. ACL, </i>pages 310-318. D. L. Chen and R. J. Mooney.   2008.   Learning to sportscast: a test of grounded language acquisition. In <i>Proc. ICML, </i>pages 128-135. D. Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In <i>Proc. ACL, </i>pages 263-270.</p><p>D. Chiang. 2007. Hierarchical phrase-based translation.</p><p><i>Computational Linguistics, </i>33:201-228. G. Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statis­tics. In <i>Proc. HLT, </i>pages 138-145. M. Galley and C. D. Manning.    2010. Accurate non-hierarchical phrase-based translation.   In <i>Proc.</i></p><p><i>HLT/NAACL, </i>pages 966-974.</p><p>R. Ge and R. J. Mooney. 2009. Learning a compositional semantic parser using an existing syntactic parser. In <i>Proc. ACL/IJCNLP, </i>pages 611-619. G. P. Huet. 1973. The undecidability of unification in third order logic. <i>Information and Control, </i>22(3):257267.</p><p>G. P. Huet. 1975. A unification algorithm for typed A-calculus. <i>Theoretical Computer Science, </i>1(1):27-57. R. J. Kate and R. J. Mooney. 2006. using string-kernels for learning semantic parsers. In <i>Proc. COLING/ACL, </i>pages 913-920.</p><doubt alpha="59.6" length="52" tooSmall="False" monospace="0.0">P. Koehn, F. J. och, and D. Marcu. 2003. Statistical</doubt><p>phrase-based translation. In <i>Proc. NAACL/HLT, </i>pages 48-54.</p><doubt alpha="65.2" length="198" tooSmall="False" monospace="0.0">P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer, o. Bojar, A. Constantin, and E. Herbst. 2007. Moses: open source toolkit for</doubt><p>statistical machine translation. In <i>Proc. ACL(Demon-stration Sessions), </i>pages 177-180. P. Koehn. 2004. Statistical significance tests for machine translation evaluation. In <i>Proc. EMNLP, </i>pages 388­395.</p><p>G. Kuhlmann, P. Stone, R. Mooney, and J. Shavlik. 2004. Guiding a reinforcement learner with natural language advice: Initial results in RoboCup soccer. In <i>Proc. AAAI Workshop on Supervisory Control ofLearning and Adaptive Systems.</i></p><p>T. Kwiatkowski, L. Zettlemoyer, S. Goldwater, and M. Steedman. 2010. Inducing probabilistic CCG grammars from logical form with higher-order unifi­cation. In <i>Proc. EMNLP, </i>pages 1223-1233.</p><p>J. D. Lafferty, A. McCallum, and F. C. N. Pereira. 2001.</p><p>Conditional random fields: Probabilistic models for segmenting and labeling sequence data.   In <i>Proc.</i></p><p><i>ICML, </i>pages 282-289.</p><p>I. Langkilde. 2000. Forest-based statistical sentence gen­eration. In <i>Proc. NAACL, </i>pages 170-177.</p><p>Z. Li, C. Callison-Burch, C. Dyer, J. Ganitkevitch, S. Khudanpur, L. Schwartz, W. N. G. Thornton, J. Weese, and o. F. Zaidan. 2009. Joshua: an open source toolkit for parsing-based machine translation. In <i>Proc. WMT, </i>pages 135-139.</p><p>P. Liang, B. Taskar, and D. Klein. 2006. Alignment by agreement. In <i>Proc. HLT/NAACL, </i>pages 104-111.</p><p>W. Lu, H. T. Ng, W. S. Lee, and L. Zettlemoyer. 2008.</p><p>A generative model for parsing natural language to meaning representations.   In <i>Proc. EMNLP, </i>pages 783-792.</p><p>W. Lu, H. T. Ng, and W. S. Lee. 2009. Natural lan­guage generation with tree conditional random fields.</p><p>In <i>Proc. EMNLP, </i>pages 400-409.</p><p>F. J. och and H. Ney. 2004. The alignment template approach to statistical machine translation. <i>Computa­tional Linguistics, </i>30(4):417-449.</p><p>F. J.Och. 2003. Minimum error rate training in statistical machine translation. In <i>Proc. ACL, </i>pages 160-167.</p><p>K. Papineni, S. Roukos, T. Ward, and W. J. Zhu. 2002.</p><p>BLEu: a method for automatic evaluation of machine translation. In <i>Proc. ACL, </i>pages 311-318.</p><p>H. Shemtov. 1996. Generation of paraphrases from am­biguous logical forms. In <i>Proc. COLING, </i>pages 919­924.</p><p>S. M. Shieber, G. van Noord, F. C. N. Pereira, and R. C. Moore. 1990. Semantic-head-driven generation. <i>Computational Linguistics, </i>16(1):30-42.</p><p>M. Snover, B. Dorr, R. Schwartz, L. Micciulla, and J. Makhoul. 2006. A study of translation edit rate with targeted human annotation.  In <i>Proc. AMTA, </i>pages 223-231.</p><p>A. Stolcke. 2002. SRILM-an extensible language mod­eling toolkit. In <i>Proc. ICSLP, </i>pages 901-904.</p><page local="12" global="1622"/><doubt alpha="60.0" length="50" tooSmall="False" monospace="0.0">S. Vogel, H. Ney, and C. Tillmann. 1996. HMM-based</doubt><p>word alignment in statistical translation.   In <i>Proc.</i></p><p><i>COLING, </i>pages 836-841.</p><p>J. Wang. 1980. on computational sentence generation from logical form. In <i>Proc. COLING, </i>pages 405-411.</p><p>Y. W. Wong and R. J. Mooney. 2006. Learning for se­mantic parsing with statistical machine translation. In <i>Proc.</i><i> HLT/NAACL, </i>pages 439-446.</p><p>Y. W. Wong and R. J. Mooney. 2007a. Generation by in­verting a semantic parser that uses statistical machine translation. In <i>Proc. NAACL/HLT, </i>pages 172-179.</p><p>Y. W. Wong and R. J. Mooney. 2007b. Learning syn­chronous grammars for semantic parsing with lambda calculus. In <i>Proc. ACL, </i>pages 960-967.</p><p>O. F. Zaidan. 2009. Z-MERT: A fully configurable open source tool for minimum error rate training of machine translation systems. <i>The Prague Bulletin ofMathe-matical Linguistics, </i>91:79-88.</p><p>L. Zettlemoyer and M. Collins. 2005. Learning to map sentences to logical form: Structured classifica­tion with probabilistic categorial grammars. In <i>Proc.</i></p><p><i>UAI, </i>pages 658-666.</p><p>L. Zettlemoyer and M. Collins. 2007. Online learning of relaxed CCG grammars for parsing to logical form. In <i>Proc.</i><i> EMNLP-CoNLL, </i>pages 678-687.</p><p>L. Zettlemoyer and M. Collins. 2009. Learning context-dependent mappings from sentences to logical form.</p><p>In <i>Proc. ACL/IJCNLP, </i>pages 976-984.</p></references></body></article>