<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="455"/><title>A Confidence Model for Syntactically-Motivated Entailment Proofs</title><pubinfo>Proceedings of Recent Advances in Natural Language Processing,pages 455^-62, Hissar, Bulgaria, 12-14 September 2011.</pubinfo><author surname="Stern" givenname="Asher"><org  name="Bar Ilan University" country="Israel" city="Ramat Gan"/></author><author surname="Dagan" givenname="Ido"><org  name="Bar Ilan University" country="Israel" city="Ramat Gan"/></author></firstpageheader><frontmatter><p><b>A Confidence Model for Syntactically-Motivated Entailment Proofs</b></p><p><b>Asher Stern Ido Dagan</b></p><p>Dept. of Computer Science Dept. of Computer Science</p><p>Bar-Ilan University Bar-Ilan University</p><p>Ramat Gan, Israel Ramat Gan, Israel</p><p>astern7@gmail.com dagan@cs.biu.ac.il</p></frontmatter><abstract>This paper presents a novel method for recognizing textual entailment which de­rives the hypothesis from the text through a sequence of parse tree transforma­tions. Unlike related approaches based on tree-edit-distance, we employ trans­formations which better capture linguis­tic structures of entailment. This is achieved by (a) extending an earlier deter­ministic knowledge-based algorithm with syntactically-motivated on-the-fly trans­formations, and (b) by introducing an al­gorithm that uniformly learns costs for all types of transformations. Our evaluations and analysis support the validity ofthis ap­proach. </abstract></header><body><section number="1" title="Introduction"><p><i>Recognizing Textual Entailment </i>(RTE) is the task of determining whether a given textual statement (a hypothesis), <b>H, </b>can be inferred by a given text passage, <b>T </b>(Dagan et al., 2005). In recent years, the task has attracted considerable interest, with research evolving around the six RTE challenges, organized by PASCAL<footnote anchor="1"/> and later under the NIST Text Analysis Conference (TAC)<footnote anchor="2"/>. While some of the proposed RTE systems employed quite shal­low and ad-hoc techniques, a few principled ap­proaches for modeling entailment inference began emerging as well.</p><p>This paper focuses on an appealing approach attempted in several previous works, which, like most RTE systems, utilizes parse-based represen­tations of the text and hypothesis. Within this approach the parse-tree of <b>H </b>is explicitly gen­erated from that of <b>T </b>by applying a sequence of tree transformation operations. In analogy to logic, that sequence can be referred to as a <i>proof, </i>by which the target proposition, represented as a parse tree, is generated from the given text propo­sitions using appropriate proof steps.</p><footnote label="1">http://pascallin.ecs.soton.ac.uk/Challenges/RTE/  2 http://www.nist.gov/tac/</footnote><doubt alpha="66.7" length="84" tooSmall="False" monospace="0.0">In one line of these works (Wang and Man­ning, 2010; Heilman and Smith, 2010; Mehdad</doubt><p>and Magnini, 2009) the tree-transformation op­erations followed mostly traditional <i>tree edit dis­tance </i>operations, such as node insertion, deletion and substitution, and learned their costs according to the given RTE training data. As described in more detail in Section 2, these transformations do not necessarily capture the syntactic structure of entailment-preserving transformations.</p><p>On the other hand, a rich inventory of knowledge-based operations was employed by Bar-Haim et al. (2007a). Their operations enable transforming complete sub-trees which do capture the syntactic structure of entailment inferences. Nevertheless, their work did not include a learn­ing component for estimating proof costs and their tree-transformations were based only on available knowledge resources, without providing on-the-fly operations that could compensate for some in­evitably missing knowledge.</p><p>In this work we aim to combine the com­plementing advantages of the above mentioned works while filling in some missing gaps. We utilize knowledge-based sub-tree transformations, following (Bar-Haim et al., 2007a), but augment them with a set of <i>on the fly </i>transformations that correspond to syntactically-motivated entailment inferences, whose reliability can be learned us­ing syntactic features. We further apply a cost model for entailment proofs and introduce an it­erative learning scheme that estimates reliability weights based on the "best" (lowest-cost) proofs for the training pairs.</p><p>Evaluations show that our current implemen­tation, including an initial set of knowledge re­sources and linguistic analysis, achieves comparable results to other proof-based systems.<page local="2" global="456"/> We conclude the paper by pointing at the generality and flexibility of our framework and suggest sev­eral research directions which can be naturally in­tegrated into it.</p></section><section number="2" title="Background"><p>As pointed above, a promising approach in RTE research is applying a sequence of operations (a proof) on the given text, <b>T, </b>to reveal whether and how it entails the hypothesis, <b>H. </b>The advantage of this approach is that it allows composition of knowledge, where in many cases information from one knowledge resource becomes relevant only af­ter a previous operation was performed.</p><p>Methods that follow this approach should deal with three main aspects. First, they have to decide how to represent <b>T </b>and <b>H. </b>Second, they have to define the set of proof operations. Third, they have to define a method to estimate the likelihood that a generated sequence of operations indeed preserves entailment.</p><p>Raina et al. (2005) used a logical representation, and accordingly defined the set of operations by a commonly used theorem proving method (reso­lution refutation). However, since state-of-the-art methods that transform a text into logical repre­sentation are less robust than syntactic parsers, the logical representation is rarely used.</p><p>Syntactic parse trees provide a common rep­resentation in text understanding systems in gen­eral, and for RTE in particular. The corresponding proof operations are thus tree-transformations that subsequently change the parse tree of <b>T </b>until <b>H</b>'s parse tree is obtained.</p><p>Mostly, the selected tree-transformations fol­lowed standard ("insert", "delete", "substitute") or custom tree edit distance operations (Mehdad and Magnini, 2009; Wang and Manning, 2010; Heil-man and Smith, 2010) However, those sets of op­erations are often not linguistically-motivated and thus do not necessarily reflect the nature of the RTE problem. In addition, utilizing knowledge resources (both linguistic knowledge and world knowledge) is limited in such systems. Consider, for example, transformation of a parse-tree from a passive form to an active form. Such transforma­tion can be done by a sequence of mostly dele­tion and insertion operations, however, such se­quence misses to capture the syntactic structure of the transformation. Similarly, resources that in-</p><p>Fi gure 1: The figure demonstrates the rule <i>X listen to Y </i><i>—►</i><i> X hear Y, </i>along with its application to the sentence "I listen to the music." dicate semantic similarity of two sub-trees, e.g. DIRT (Lin and Pantel, 2001), would be utilized naturally by substitution of a complete sub-tree by another, which cannot be performed by the above tree-edit-distance operations.</p><p>In contrast, a set of linguistically-motivated op­erations was proposed independently by Harmelthe set of operations defined by Harmeling (2009) was limited and included mostly ad-hoc heuristics, the operations defined by Bar-Haim et al. (2007a) were designed to capture a broad range of linguis­tic and world knowledge. Their primary opera­tions are applications of <i>Entailment Rules, </i>which substitute complete sub-trees and generate new parse-trees, based on knowledge resources (see Figure 1 and Table 1).</p><doubt alpha="52.1" length="48" tooSmall="False" monospace="0.0">ing (2009) and by Bar-Haim et al. (2007a). While</doubt><p>However, unlike other methods that followed the proof-style approach, the method of Bar-Haim et al. (2007a) does not estimate the likelihood that a generated proof is valid. Another problem is that in most cases, there is no sequence of operations that completely generates <b>H. </b>Rather, starting from <b>T, </b>the operations generate new trees that become more similar, but not identical to <b>H<footnote anchor="3"/> </b>(Bar-Haim, 2010).</p><p>To summarize, two main challenges are in­volved in transformational proof-style inferences over syntactic parse trees. The first is defining a method to estimate the likelihood that a given proof preserves entailment. The second is to de­fine operations that are linguistically-motivated and reflect the RTE problem space. So far, all proof-style systems addressed either the first or the<page local="3" global="457"/></p><footnote label="3">On the RTE datasets, a hybrid framework was intro­duced by Bar-Haim et al. (2007b), which uses an approximate match mechanism for final classifications.</footnote><p>Table 1: Types of Entailment Rules. Note that for simplicity the examples are presented as strings, though the actual definition and implementation are based on sub-trees, as in Figure 1 second challenge, but not both. As described next, in this work we propose a principled integrated so­lution to those two challenges.</p></section><section number="3" title="A Cost-based Proof Model"><p>In our framework we adopt the linguistically-motivated entailment operations proposed by Bar-Haim et al. (2007a), and extend them with syntactically-motivated on-the-fly operations to enable generation of complete proofs (Sec. 3.1). The extended framework is then integrated with a learning method similar to the one proposed for logic representations by (Raina et al., 2005) as fol­lows. We propose a cost model, which assigns a cost for each entailment proof (Sec. 3.2), and introduce a search algorithm that finds the "best proof" with respect to the cost model (Sec. 3.3). Finally we describe a method to iteratively learn the parameters of the cost model (Sec. 3.4).</p><subsection number="3.1" title="Inference Formalism"><p>The model presented here assumes a single-sentence hypothesis, similar to the RTE chal­lenges, though it can be easily adjusted to multi-sentence hypotheses as well.</p><p>Given a <b>(T,H) </b>pair, the system first constructs the dependency parse trees<footnote anchor="4"/> of <b>T </b>and <b>H. </b>Each node in those trees contains information about one lexical item (i.e. a word or a multi-word expres­sion), which includes its lemma and its part-of-speech, and optionally other information, such as Named Entity type<footnote anchor="5"/>. Each edge is labelled with a dependency relation (e.g. <i>subject, object).</i></p><p>Let T be a set of dependency parse trees that were constructed for <b>T</b>'s sentences, and let h be the dependency parse tree constructed for <b>H. </b>The system iteratively extends T with additional trees, by applying <i>tree generation operations, </i>until there exists a tree t G T, such that h is embedded in t.</p><footnote label="4">We used the Minipar parser (Lin, 1998b)</footnote><footnote label="5">We used Stanford NE recognizer (Finkel et al., 2005)</footnote><p>We will use the following notations: Let T be a set of trees, o be a tree generation operation, and t be a tree. T ho t denotes that t can be generated from T using the operation o. We will use the h notation also for the resulting extended set of trees, that is:</p><p><i>Th</i><i>o </i><i>Tu</i><i>{t]</i></p><p>Let <i>O = (oi,o2 </i>... <i>om) </i>be a sequence of op­erations. The notation T =O T' means that T' can be generated from T by applying iteratively the operations in O. Finally, a sequence of opera­tions is called a <i>proof, </i><i>P</i>, if T =P T' such that h is embedded in one of the trees in T'.</p><p>Although a more accurate definition of a proof would require that h would be identical to one of the trees in T', rather than being embedded in one of them, our relaxed definition is a common heuristic simplifying the proof construction pro­cess.</p><subsubsection number="3.1.1" title="Entailment rules"><p>The primary operations in Bar-Haim et al. (2007a) are applications of <i>Entailment Rules. </i>An entail-ment rule is composed of two sub-trees, named <i>left hand side (lhs) </i>and <i>right hand side (rhs), </i>in­tended to capture an entailment relation between its two sides (See Table 1). For example, a simple lexical rule is "music — art", where both sub-trees consist of single nodes.</p><p>Let <i>r = (lhs, rhs) </i>be a rule and t be a parse-tree, such that <i>lhs </i>is embedded in t. An <i>appli­cation </i>of r on t is a generation of a new tree, t', which is identical to t, but with the instance of <i>lhs </i>in t being replaced by <i>rhs. </i>If the underlying meaning of t entails the meaning of t', then we would consider the application of r as <i>valid. </i>It should be noted that in (Bar-Haim et al., 2007a) all rule-applications, based on the set of rules given to the system, were considered valid for any arbitrary <b>(T,H) </b>pair, an assumption which we relax in our cost-based model.</p><p>A rule's <i>lhs </i>and <i>rhs </i>may contain <i>variables, </i>i.e.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Rule Type</b></p></td><td class="cell"><p><b>Description</b></p></td><td class="cell"><p><b>Examples</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Lexical Rules</p></td><td class="cell"><p>Substitution of a single node, capturing lexical entailment. Both <i>lhs </i>and <i>rhs </i>are single nodes.</p></td><td class="cell"><p>novel — book walk — go</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Lexical Syntactic Rules</p></td><td class="cell"><p>Tree transformations that change the tree's lexical items as well as the tree's structure.</p></td><td class="cell"><p>"X file lawsuit against Y" — "X accuse Y" "X listen to Y" — "X hear Y"</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Generic Syntactic Rules</p></td><td class="cell"><p>Tree structure transformations. Capture linguistic phenomena (e.g. passive-active).</p></td><td class="cell"><p>X <i>V(active) </i>Y — Y is <i>V(passive) </i>by X</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="4" global="458"/><p>nodes in which the lemma is not specified. When such a rule is applied, the system first instantiates the variables with actual lemmas, according to the original tree, and then replaces the <i>lhs </i>by the in­stantiated <i>rhs </i>(As exemplified in Figure 1). As de­scribed in Section 2 and Table 1, such entailment rules are able to capture a broad range of linguis­tic and world knowledge. It should be noted that in our current implementation generic-syntactic rules were not integrated yet. Incorporating and extend­ing the set of generic-syntactic rules is currently under work.</p></subsubsection><subsubsection number="3.1.2" title="Co-reference Operations"><p><i>Co-reference Substitution </i>is a tree manipulation that is performed according to co-reference in­formation, given by an external co-reference re-solver<footnote anchor="6"/>. Given two mentions <i>mi </i>and m2 of the same entity, not necessarily in the same parse-tree, we define the operation of replacing the sub-tree rooted by mi by the sub-tree rooted by m2 as <i>Co-reference Substitution.</i></p></subsubsection><subsubsection number="3.1.3" title="On The Fly operations"><p>As described in Section 2, the original scheme of Bar-Haim et al. (2007a) recognized a <b>(T,H) </b>pair as entailing if and only if <b>H </b>could be generated by a sequence of co-reference substitutions and appli­cations of rules from the given set of knowledge resources. Inevitably that scheme suffers very lim­ited recall<footnote anchor="7"/>.</p><p>Utilizing our learning scheme as described be­low, we are able to overcome that difficulty, by adding an additional set of <i>on the fly </i>tree-transformations. Though those operations are not justified by a pre-given knowledge base, an es­timation of their correctness likelihood can be learned, based on syntactic features. For example, moving a complete sub-tree is defined as an atomic operation, in contrast to the regular tree-edit-distance operations, in which such transformation requires a sequence of "insert" and "delete" oper­ations.</p><p>An initial set of on-the-fly operations which is implemented in our system is specified in Table 2. The validity of applying such operations is esti­mated by the cost-model, described next, using the</p><footnote label="6">We used BART co-reference resolver (Versley et al., 2008)</footnote><footnote label="7">As mentioned earlier, to increase recall in practical RTE datasets, a hybrid framework was introduced by Bar-Haim et al. (2007b), which uses an approximate match mechanism for final classifications.</footnote><p>Table 2: <i>on-the-fly </i>operations in our system.</p><p>features listed in Table 3. Those operations rep­resent simple transformations required to handle differences between two dependency-parse-trees, and are applied when parts of the hypothesis tree are missing in a given tree in T.</p><p>This set of operations can be extended in the fu­ture by using additional linguistic resources, e.g. by identifying the semantic role of the inserted and moved nodes, or by adding on-the-fly substi­tutions, scored by distributional similarity.</p></subsubsection></subsection><subsection number="3.2" title="Cost Model"><p>Given a proof P, we want to estimate its cor­rectness likelihood. Under the assumption that some or all of the operations in P might be in­correct - for example due to inaccuracies of the knowledge bases, wrong co-reference resolution or incorrect on-the-fly operations - we define a <i>cost model </i>to quantify the proof's likelihood to be correct. Following the cost model applied by Raina et al. (2005) to logic proofs, we use an additive linear model in which each operation is characterized by a set of features and the oper­ation's total cost is a weighted linear combina­tion of those features. Formally, let o G P, let F<i>(o) </i>= <i>(F(o),F(o),</i>... <i>F{°])T </i>be a feature vec­tor characterizing o, and let w be a correspond­ing <i>weight vector. </i>The total cost of o (denoted by Cw(o)) is defined as:</p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">D</doubt><doubt alpha="42.9" length="28" tooSmall="False" monospace="0.0">Cw(o)â£Wi• F(o)= wT• F(o)(1)</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">i=i</doubt><p>The cost of a sequence of operations (and in par­ticular of a proof) is naturally defined as the sum of costs of all operations. Thus, given a proof<page local="5" global="459"/></p><table class="main" frame="box" rules="all" border="1" regular="True"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Operation-Name</p></td><td class="cell"><p>Operation-Description</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Insert Node</p></td><td class="cell"><p>Insert a new node in an arbitrary posi­tion in a parse tree.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Move sub tree</p></td><td class="cell"><p>Disconnect a sub tree rooted by n from its parent <i>p(n) </i>and connect it as a child of another node in the tree, <i>p'(n).</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Change Relation</p></td><td class="cell"><p>Change the relation (the edge label) between a node n and its parent <i>p(n)</i>.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Flip Part-Of-Speech</p></td><td class="cell"><p>Change a node's part-of-speech.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Cut Multi-Word</p></td><td class="cell"><p>Remove some of the words from a multi-word expression, as identified by the parser</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Single-Word to Multi-Word</p></td><td class="cell"><p>Replace a word by a multi word ex­pression containing it, e.g. "Bond" — "James Bond".</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">i=i</doubt><doubt alpha="57.4" length="47" tooSmall="False" monospace="0.0">P = (oi, o2,... om), its total cost, denoted by</doubt><doubt alpha="50.0" length="10" tooSmall="False" monospace="0.0">Cw(P), is:</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">m</doubt><doubt alpha="42.1" length="19" tooSmall="False" monospace="0.0">Cw(P ) =J]Cw(oj)(2)</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">j=i</doubt><doubt alpha="57.7" length="52" tooSmall="False" monospace="0.0">Let F(P)=YJ^jLiF(oj). Combining (1) and (2), we get:</doubt><doubt alpha="38.7" length="31" tooSmall="False" monospace="0.0">Cw(P) â £ wi• F(P)= wT• F(P)(3)</doubt><p>The last equation provides a way to represent a complete proof by a single feature-vector, which is simply the sum of all operations' vectors. We will use this feature representation in the learning and classification phases.</p><p>Foreach<b>(T,H) </b>pairtheremightbe manyproofs. However, for positive pairs, we assume there ex­ists a "correct" proof, i.e. a proof that is composed of only valid operations (though many other in­correct proofs exist as well), while for negative pairs non of the proofs is correct. An optimal weight vector, w*, would assign low costs to cor­rect proofs while incorrect proofs will be assigned high costs. Therefore, distinguishing between pos­itive pairs and negative pairs should be done by examining their lowest-cost proofs.</p><p>In the next sub-sections we describe how to search for lowest-cost proofs ("best proofs") and how to learn the optimal weight vector.</p><subsubsection number="3.2.1" title="Modelling Operations by Features"><p>As a convention, all features are assigned zero-or-negative values, interpreted as penalty. For each value vi assigned to a feature Fi, vi = 0 means that no penalty is implied by that feature, while » 0 implies a high penalty by that feature. Follow­ing that convention, all weights should be assigned zero-or-positive values, since adding an operation cannot improve the confidence of a proof. This implies that an operation's total cost Cw(o), and a proof's total cost Cw (P) are zero-or-negative. The higher the absolute cost value, the lower the like­lihood of the proof's correctness.</p><p>Features were defined for each knowledge re­source, for co-reference substitution and for on-the-fly operations, as summarized in Table 3. For knowledge resources, features were defined as fol­lows. Many knowledge resources provide nu­merical scores, indicating rules' reliability, which we use for the corresponding feature value. The knowledge resources that provide such scores and were used in the current system are DIRT (Lin and Pantel, 2001), Wikipedia rules (Shnarch et al., 2009), Lin similarity (Lin, 1998a), and Directional-Similarity<footnote anchor="8"/> (Kotlerman et al., 2010). For knowledge resources that do not provide a nu­merical information about rule reliability, the cor­responding feature-value is set to (—1). In the cur­rent system, WordNet<footnote anchor="9"/> (Fellbaum, 1998; Miller, 1995), an in-house Geographical data-base, and</p><p>VerbOcean<footnote anchor="10"/> (Chklovski and Pantel, 2004) were included.</p><p>Some on-the-fly operations incorporate numer­ical information that reflects how likely it is that the meaning of the text is changed by applying them. As an example, for the insert-node oper­ation we use the "Maximum Likelihood Estima­tion" (MLE) of the occurrence probability of the inserted word in a large news corpus<footnote anchor="11"/>. The un­derlying assumption here is that it is more likely that inserting frequent words would still preserve entailment than inserting rare words.</p></subsubsection></subsection><subsection number="3.3" title="Searching for the best proof"><p>Searching for the best proof is done iteratively. Starting from T as the original text's trees, and a given weight vector, the system adds all the trees that can be generated by applying any generation­operation on T. Since that scheme makes T grow exponentially, we use a simple beam search prun­ing approach as follows.</p><p>A constant beam size K is predetermined. In each iteration T is pruned such that its number of trees will be no more than K. Since every tree in T was generated by a sequence of operations, we define the cost of a tree as the cost of the se­quence that was used to generate that tree. We use that cost, in addition to estimations about the dif­ference between a given tree to the hypothesis tree, in order to decide which tree should be pruned out, such that after each iteration |T| &lt; K. Finally, the lowest cost generated tree which embeds h is re­turned.</p><footnote label="8">A rule-base of lexical entailment rules automatically ex­tracted by means of directional distributional similarity.</footnote><footnote label="9">We used the following WordNet relations: hypernymy, holonymy, verb-entailment and synonymy</footnote><footnote label="10">Only the relation "stronger" was used.</footnote><footnote label="11">We used Reuters Corpus, Volume 1+2 (RCV1-2). Avail­able at http://trec.nist.gov/data/reuters/reuters.html</footnote><page local="6" global="460"/><p>Table 3: Features and their values for each (knowledge and on-the-fly) operation. Note that all values are negative.</p></subsection><subsection number="3.4" title="Iterative Weight Estimation"><p>We would like to classify a proof P, represented by a feature vector F, as "correct" if its cost is low.</p><p>Formally, let (w,b) be a weight vector and a threshold. P is classified as correct if and only and as incorrect otherwise. The goal of parameter estimation is thus finding optimal (w*, b*).</p><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">if</doubt><doubt alpha="21.4" length="14" tooSmall="False" monospace="0.0">w • F + b&gt;0(4)</doubt><p>If our training set was a set of binary-labelled vectors (Fi, i G {1... n}, we could apply di­rectly a linear training algorithm to find (w*, b*). However, our training set is a set of labelled text pairs, for which the proofs that determine the cor­responding feature vectors should be constructed by the system. Yet, as explained at the end of Section 3.2, only the lowest-cost proofs should be considered to distinguish between positive and negative pairs, while finding those proofs through the search algorithm ofSection 3.3 requires know­ing the optimal weight vector.</p><p>We therefore use an iterative learning scheme to <b>Algorithm 1 </b>Parameters Estimation</p><p><b>Require: </b>Training set: <b>(T</b>i<b>,H</b>i,Zi)... <b>(Tn,H</b>n,1n) 1: (w0 , b0) — a reasonable guess of weights vector and threshold 3: <b>repeat</b> 5:     Construct   the   corresponding   feature vectors</p><doubt alpha="12.5" length="8" tooSmall="False" monospace="0.0">2: i — 0</doubt><doubt alpha="61.8" length="55" tooSmall="False" monospace="0.0">4:     FindP1...Pnby the method described in 3.3, using</doubt><doubt alpha="57.1" length="7" tooSmall="False" monospace="0.0">(wi,bi)</doubt><doubt alpha="54.5" length="11" tooSmall="False" monospace="0.0">F(Pi)_F(Pn\</doubt><p>6: use (F(Pi... (F(Pn),1n) as a training set to a linear classifier, resulting new parameters (wi+i,bi+i).</p><doubt alpha="12.5" length="16" tooSmall="False" monospace="0.0">7:     i — i + 1</doubt><p>8: <b>until </b>convergence</p><p>overcome this circularity problem, as follows (see Algorithm 1). We start with an initial weight vec­tor and threshold, (wo, bo), set manually by a rea­sonable guess. Using the algorithm in Section 3.3 we find a lowest-cost proof for each pair, resulting in n labelled feature vectors, (Fi, 1i)... (Fn, where 1i is the binary entailment annotation. Next, we use a standard linear learning algorithm to learn new parameters, (wi, bi). We iteratively im­prove the weights vectors and the proofs until con-<page local="7" global="461"/></p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>#</b></p></td><td class="cell"><p><b>Feature</b></p></td><td class="cell"><p><b>Value</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>1</b></p></td><td class="cell"><p>Wikipedia</p></td><td class="cell"><p>log(m), where m is the estimated accuracy of the method used to learn the given Wikipedia rule, as described in (Shnarch et al., 2009). 0 &lt; m &lt; 1.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>2</b></p></td><td class="cell"><p>Lin Similarity</p></td><td class="cell"><p>log(sim), where <i>sim </i>is the similarity score given for that rule ac­cording to (Lin, 1998a). 0 &lt; sim &lt; 1.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>3</b></p></td><td class="cell"><p>Directional-Similarity</p></td><td class="cell"><p>log(sim), where sim is the similarity score given for that rule ac­cording to (Kotlerman et al., 2010). 0 &lt; sim &lt; 1.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>4</b></p></td><td class="cell"><p>DIRT</p></td><td class="cell"><p>log(sim), where sim is the similarity score given for that rule ac­cording to (Lin and Pantel, 2001). Note that 0 &lt; sim &lt; 1.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>5</b></p></td><td class="cell"><p>WordNet</p></td><td class="cell"><p>-1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>6</b></p></td><td class="cell"><p>VerbOcean</p></td><td class="cell"><p>-1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>7</b></p></td><td class="cell"><p>Geographical Database</p></td><td class="cell"><p>-1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>8 9 10 11</b></p></td><td class="cell"><p>Insert Verb</p><p>Insert non-verb content word Insert non-content word insert Named Entity</p></td><td class="cell"><p>log(f ), where f is the MLE of the occurrence probability for the inserted lemma in the Reuters news corpus.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>12, 13, 14,</b></p><p><b>15</b></p></td><td class="cell"><p>Insert verb / content word / non-content word / Named Entity - that exist in other parts of the</p><p>text</p></td><td class="cell"><p>log(f ), where f is the MLE of the occurrence probability for the inserted lemma in the Reuters news corpus.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>16</b></p></td><td class="cell"><p>Change relation of a node to its parent, from "subject" to "object" or vice versa</p></td><td class="cell"><p>-1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>17 18</b></p></td><td class="cell"><p>Move Sub Tree rooted by n from <i>p(n) </i>to <i>p'</i><i> (n), </i>s.t. the path from n to p'(n) contains a verb All other "move Sub Tree" operations</p></td><td class="cell"><p>-1, where l is the length of the path between n and p'(n) in the original tree.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>19</b></p></td><td class="cell"><p>Single-word to Multi-word</p></td><td class="cell"><p>log(min/6F(f )) where F is the set of MLE of the occurrence probabilities corresponding to the added words. The probabilities were calculated using the Reuters News corpus.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>20</b></p></td><td class="cell"><p>Cut Multi-word</p></td><td class="cell"><p>-1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>21</b></p></td><td class="cell"><p>Flip part-of-speech</p></td><td class="cell"><p>-1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>22</b></p></td><td class="cell"><p>Co-reference</p></td><td class="cell"><p>-1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Table 4: Accuracy of proof-based systems on RTE datasets, followed by median results and best results of all systems participated in those challenges.</p><p>vergence. Since there is no theoretical bound on the convergence rate, we limit the number of iter­ations by a predefined constant. In practice, how­ever, only few iterations are required for conver­gence.</p></subsection></section><section number="4" title="Evaluation"><p>We ran experiments on the first, second, third and fifth RTE datasets<footnote anchor="12"/> (Dagan et al., 2005; Bar-Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009) and compared our system to other proof-style systems. Each dataset consists of 600 to 800 <b>(T,H) </b>pairs, half of them are positive, and the other half are negative. For the underlying lin­ear classifier, required by Algorithm 1, we used linear-SVM<footnote anchor="13"/>. The value of K, described in Sec­tion 3.3, was set to 135, according to tuning done on the training set. The results for our system, pre­sented in Table 4, show comparable performance to other systems on most datasets, with notably higher performance in RTE-3.</p><p>Table 5: The average value of certain features in posi­tive pairs and negative pairs, taken from an experiment on the</p><p>RTE-2 test set.</p><p>As indicated by the results, our system indeed assigns, on average, higher costs to negative pairs than to positive ones. Further insight into this be­havior is obtained by Table 5. The table presents a sample of features' average values. The up­per rows of the table present features whose aver­age absolute value in negative pairs is significantly higher than in positive pairs, while the features in the lower rows have similar average values in pos­itive and negative pairs.</p><footnote label="12">The RTE-4 dataset had no training dataset</footnote><footnote label="13">We     used     SVM-Light,      available at</footnote><footnote>http://svmlight.joachims.org/</footnote><p>The former features indicate that there are some operations that tend to be part of the "best" proof for negative pairs more frequently than for posi­tive pairs. A reasonable explanation for this phe­nomenon is that the system learned that some op­erations are less reliable than other operations, and tried to avoid them whenever possible. However, these operations could not be avoided in negative pairs, resulting in higher feature values.</p></section><section number="5" title="Conclusions and Future Work"><p>Two main concepts underlie this paper. The first is automatic estimation of the quality of proofs required to recognize textual-entailment. The second concept is a complete framework of linguistically-motivated proof operations for recognising textual-entailment. The main contri­bution of this paper is showing how those two con­cepts can be integrated, to leverage the advantages of both.</p><p>The linguistically-motivated framework pre­sented here is based on the framework proposed by (Bar-Haim et al., 2007a), with a significant extension of on-the-fly operations required for making it robust and complete. Many additional linguistically-motivated entailment operations can be naturally integrated into this framework. For example, lexical, syntactic and semantic attributes like verb-tense and polarity (negation) can be eas­ily handled, much like part-of-speech and named-entity (Bar-Haim et al., 2007a). Another exam­ple is temporal inference (e.g. "this afternoon — today") which can be integrated easily by proper substitutions based on an appropriate knowledge resource (similar to the one proposed by Wang and Zhang (2008)).<page local="8" global="462"/> Yet another example is ad­dressing more types of co-reference based opera­tions (Mirkin et al., 2010). Finally, as noted ear­lier, the current set of on-the-fly operations may be extended, which will likely improve the system's performance.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>System</b></p></td><td class="cell"><p><b>RTE-1</b></p></td><td class="cell"><p><b>RTE-2</b></p></td><td class="cell"><p><b>RTE-3</b></p></td><td class="cell"><p><b>RTE-5</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Learning and abductive reasoning (Raina et al., 2005)</p></td><td class="cell"><p>57.0 %</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Probabilistic Calculus of Tree Transformations (Harmeling, 2009)</p></td><td class="cell"><p></p></td><td class="cell"><p>56.39 %</p></td><td class="cell"><p>57.88 %</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Probabilistic Tree Edit model (Wang and Manning, 2010)</p></td><td class="cell"><p></p></td><td class="cell"><p>63.0 %</p></td><td class="cell"><p>61.10 %</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Deterministic Entailment Proofs (Bar-Haim et al., 2007b)</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>61.12%</p></td><td class="cell"><p>63.80 %</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Our System Accuracy (Recall % / Precision %)</p></td><td class="cell"><p>57.13%</p></td><td class="cell"><p>61.63%</p></td><td class="cell"><p>67.13%</p></td><td class="cell"><p>63.50%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(81.0/54.8)</p></td><td class="cell"><p>(76.2/59.0)</p></td><td class="cell"><p>(87.2/63.3)</p></td><td class="cell"><p>(75.7/60.9)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Median of all submissions in challenge</p></td><td class="cell"><p>55.20 %</p></td><td class="cell"><p>58.13 %</p></td><td class="cell"><p>61.75 %</p></td><td class="cell"><p>61.00 %</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Best System in challenge</p></td><td class="cell"><p>58.6 %</p></td><td class="cell"><p>75.3 %</p></td><td class="cell"><p>80.0 %</p></td><td class="cell"><p>73.5 %</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Operation</b></p></td><td class="cell"><p><b>avg. in posi­tives</b></p></td><td class="cell"><p><b>avg. in negatives</b></p></td><td class="cell"><p><b>ratio</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Insert Named Entity</p></td><td class="cell"><p>-0.006</p></td><td class="cell"><p>-0.016</p></td><td class="cell"><p>2.67</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Insert content word</p></td><td class="cell"><p>-0.038</p></td><td class="cell"><p>-0.094</p></td><td class="cell"><p>2.44</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>DIRT</p></td><td class="cell"><p>-0.013</p></td><td class="cell"><p>-0.023</p></td><td class="cell"><p>1.73</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>"subject" "object"</p></td><td class="cell"><p>-0.025</p></td><td class="cell"><p>-0.040</p></td><td class="cell"><p>1.60</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Flip part-of-speech</p></td><td class="cell"><p>-0.098</p></td><td class="cell"><p>-0.101</p></td><td class="cell"><p>1.03</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Lin similarity</p></td><td class="cell"><p>-0.084</p></td><td class="cell"><p>-0.072</p></td><td class="cell"><p>0.86</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>WordNet</p></td><td class="cell"><p>-0.064</p></td><td class="cell"><p>-0.052</p></td><td class="cell"><p>0.81</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><section title="Acknowledgements"><p>This work was partially supported by the Israel Science Foundation grant 1112/08 and by the PASCAL-2 Network of Excellence of the Euro­pean Community FP7-ICT-2007-1-216886.</p></section><references><p>Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second pascal recognising textual entailment challenge. In <i>Proceedings of the Second PASCAL Challenges Workshop on Recognis­ing Textual Entailment.</i></p><p>Roy Bar-Haim, Ido Dagan, Iddo Greental, and Eyal Shnarch. 2007a. Semantic inference at the lexical-syntactic level. In <i>Proceedings ofAAAI.</i></p><p>Roy Bar-Haim, Ido Dagan, Iddo Greental, Idan Szpek-tor, and Moshe Friedman. 2007b. Semantic infer­ence at the lexical-syntactic level for textual entail-ment recognition. In <i>Proceedings ofACL-PASCAL Workshop on Textual Entailment and Paraphrasing.</i></p><p>Roy Bar-Haim. 2010. <i>Semantic Inference at the Lexical-Syntactic Level. </i>Ph.D. thesis, Bar-Ilan Uni-veristy.</p><p>Luisa Bentivogli, Bernardo Magnini, Ido Dagan, H.T. Dang, and D. Giampiccolo. 2009. The fifth pas­cal recognizing textual entailment challenge. In <i>pro­ceeding ofTAC.</i></p><p>Timothy Chklovski and Patrick Pantel. 2004. Verbo-cean: Mining the web for fine-grained semantic verb relations. In <i>Proceedings ofEMNLP.</i></p><p>Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment challenge. In Joaquin Quifionero Candela, Ido Da­gan, Bernardo Magnini, and Florence d'Alche Buc, editors, <i>MLCW, </i>volume 3944 of <i>Lecture Notes in Computer Science, </i>pages 177-190. Springer.</p><p>Christiane Fellbaum, editor. 1998. <i>WordNet An Elec­tronic Lexical Database. </i>The MIT Press, Cam­bridge, MA ; London, May.</p><p>Jenny Rose Finkel, Trond Grenager, and Christo­pher D. Manning. 2005. Incorporating non-local information into information extraction systems by Gibbs Sampling. In <i>Proceedings ofACL.</i></p><p>Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third pascal recogniz­ing textual entailment challenge. In <i>RTE '07 Pro­ceedings ofthe ACL-PASCAL Workshop on Textual Entailment and Paraphrasing.</i></p><p>Stefan Harmeling. 2009. Inferring textual entailment with a probabilistically sound calculus. <i>Natural Language Engineering, </i>15(4):459—477.</p><p>Michael Heilman and Noah A. Smith. 2010. Tree edit models for recognizing textual entailments, para­phrases and answers to questions. In <i>HLT-NAACL.</i></p><p>Lili Kotlerman, Ido Dagan, Idan Szpektor, and Maayan Zhitomirsky-geffet. 2010. Directional distributional similarity for lexical inference. <i>Natural Language Engineering, </i>16:359-389.</p><p>Dekang Lin and Patrick Pantel. 2001. DIRT - discov­ery of inference rules from text. In <i>Proceedings of ACM SIGKDD Conference on Knowledge Discovery and Data Mining.</i></p><p>Dekang Lin. 1998a. Automatic retrieval and clustering of similar words. In <i>Proceedings of COLING-ACL.</i></p><p>Dekang Lin. 1998b. Dependency-based evaluation of minipar. In <i>Proceedings ofthe Workshop on Eval­uation of Parsing Systems at LREC 1998, </i>Granada, Spain.</p><p>Yashar Mehdad and Bernardo Magnini. 2009. Op­timizing textual entailment recognition using parti­cle swarm optimization. In <i>Proceedings of the 2009 Workshop on Applied Textual Inference.</i></p><p>George A. Miller. 1995. Wordnet: A lexical database forenglish. <i>Commun. ACM, </i>38(11):39-41.</p><p>Shachar Mirkin, Ido Dagan, and Sebastian Pado. 2010. Assessing the role of discourse references in entail-ment inference. In <i>Proceedings ofACL.</i></p><p>Rajat Raina, Andrew Y. Ng, and Christopher D. Man­ning. 2005. Robust textual inference via learning and abductive reasoning. In <i>Proceedings ofAAAI.</i></p><p>Eyal Shnarch, Libby Barak, and Ido Dagan. 2009. Ex­tracting lexical reference rules from Wikipedia. In <i>ACL-IJCNLP.</i></p><p>Yannick Versley, Simone Paolo Ponzetto, Massimo Poesio, Vladimir Eidelman, Alan Jern, Jason Smith, Xiaofeng Yang, and Ro Moschitti. 2008. BART: A modular toolkit for coreference resolution. In <i>Pro­ceedings of ACL, Demo Session.</i></p><p>Mengqiu Wang and Christopher D. Manning. 2010. Probabilistic tree-edit models with structured latent variables for textual entailment and question answer­ing. In <i>Proceedings ofCOLING.</i></p><p>Rui Wang and Yajing Zhang. 2008. Recognizing tex­tual entailment with temporal expressions in natural language texts. In <i>Proceedings ofthe IWSCA.</i></p></references></body></article>