A necessary (if not sufficient) condition for true natural language understanding is a mastery of open-domain natural language inference (NLI): the task of determining whether a natural-language hypothesis can be inferred from a given premise. Indeed, NLI can enable more immediate applica­tions, such as semantic search and question an-

© 2008. Licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported li­cense (http://creativecommons.Org/licenses/by-nc-sa/3.0/). Some rights reserved.

swering (Harabagiu and Hickl, 2006). In recent years a spectrum of approaches to robust, open-domain NLI have been explored within the con­text of the Recognizing Textual Entailment chal­lenge (Dagan et al., 2005). Up to now, the most successful approaches have used fairly shallow semantic representations, relying on measures of lexical or semantic overlap (Jijkoun and de Ri-jke, 2005), pattern-based relation extraction (Ro­mano et al., 2006), or approximate matching of predicate-argument structure (Hickl et al., 2006). Such methods, while robust and often effective, are at best partial solutions, unable to explain even simple forms of logical inference. For example, most shallow approaches would fail to license the introduction of large in the following example:

(1) Every firm saw costs grow more than expected, even after adjusting for inflation. Every large firm saw costs grow.

At the other extreme, some researchers have ap­proached NLI as logical deduction, building on work in theoretical semantics to translate sentences into first-order logic (FOL), and then applying a theorem prover or model builder (Akhmatova, 2005; Fowler et al., 2005). Regrettably, such ap­proaches tend to founder on the myriad complexi­ties of full semantic interpretation, including tense, aspect, causality, intensionality, modality, vague­ness, idioms, indexicals, ellipsis, and many other issues. (What is the right FOL representation of (1), for example?) FOL-based systems that have attained high precision (Bos and Markert, 2006) have done so at the cost of very poor recall.

This work explores a middle way, by develop­ing a computational model of what Lakoff (1970) called natural logic, which characterizes valid pat­terns of inference in terms of syntactic forms resembling natural language as much as possible. For example, natural logic might sanction (1) by observing that: in ordinary (upward monotone) contexts, deleting modifiers preserves truth; in downward monotone contexts, inserting modifiers preserves truth; and every is downward monotone in its restrictor NP. A natural logic system can thus achieve the expressivity and precision needed to handle a great variety of simple logical inferences, while sidestepping the difficulties of full semantic interpretation.

The natural logic approach originated in traditional logic (e.g., Aristotle's syllogisms), and was re­vived in a formal form by van Benthem (1986) and Sanchez Valencia (1991), who proposed a natural logic based on categorial grammar to handle infer­ences involving containment relations and upward and downward mono tonicity, such as (1). Their monotonicity calculus explains inferences involv­ing even nested inversions of monotonicity, but be­cause it lacks any representation of exclusion (as opposed to containment), it cannot explain simple inferences such as (38) and (205) in table 2, below.

Another model which arguably follows the nat­ural logic tradition (though not presented as such) was developed by Nairn et al. (2006) to explain in­versions and nestings of implicative (and factive) predicates, as in Ed did not forget to force Dave to leave |= Dave left. Their implication projection al­gorithm bears some resemblance to the monotonic­ity calculus, but does not incorporate containment relations or explain interactions between implica-tives and monotonicity, and thus fails to license John refused to dance |= John didn't tango.

We propose a new model of natural logic which generalizes the monotonicity calculus to cover in­ferences involving exclusion, and (partly) unifies it with Nairn et al.'s model of implicatives. We (1) augment the set of entailment relations used in monotonicity calculus to include representations of exclusion; (2) generalize the concept of mono­tonicity to one of projectivity, which describes how the entailments of a compound expression depend on the entailments of its parts; and (3) describe a weak proof procedure based on composing entail­ment relations across chains of atomic edits.

Entailment relations. We employ an inventory of seven mutually exclusive basic entailment rela­tions, defined by analogy with set relations: equiv­alence {couch = sofa); forward entailment {crow rz bird) and its converse {European □ French); negation, or exhaustive exclusion {human ~ non-human); alternation, or non-exhaustive exclusion {cat | dog); cover, or non-exclusive exhaustion {an­imal ^ nonhuman); and independence {hungry # hippo), which covers all other cases. As in the monotonicity calculus, we define these relations for expressions of every semantic type: sentences, common and proper nouns, transitive and intran­sitive verbs, adjectives, and so on. For example, among generalized quantifiers, we find that all = every, every rz some, some ~ no, no \ every, at least four ^ at most six, and most # ten or more. Projectivity. In order to explain the entailments of a compound expression as a function of the entailments of its parts, we categorize semantic functions according to their projectivity class, a concept which generalizes both Sanchez Valen­cia's monotonicity classes (upward, downward, and non-monotone) and the nine implication sig­natures of Nairn et al. The projectivity class of a function / describes how the entailment rela­tion between f(x) and f(y) depends on the en­tailment relation between x and y. Consider sim­ple negation {not). Like most functions, it projects = and # without change {not happy = not glad and isn't swimming # isn't hungry). As a down­ward monotone function, it swaps IZ and □ {didn't kiss □ didn't touch). But we can also establish that it projects " without change {not human " not nonhuman) and swaps | and ^ {not French ^ not German, not more than 4 \ not less than 6). By contrast, an implicative like refuse, though it also swaps rz and □ {refuse to tango □ refuse to dance), projects " as | {refuse to stay \ refuse to go) and projects both | and ^ as # {refuse to tango # refuse to waltz).

Projectivity thus allows us to determine the en­tailments of a compound expression recursively, by propagating entailments upward through a se­mantic composition tree according to the projec­tivity class of each node on the path to the root. For example, the semantics of Nobody can enter with- out a shirt might be represented by the tree {no­body {can {{without {a shirt)) enter))). Since shirt rz clothes, and since without is downward mono­tone, we have without shirt □ without clothes. Since nobody is also downward monotone, it fol­lows that Nobody can enter without a shirt rz No­body can enter without clothes.

Inference. Let x' = e{x) be the result of ap­plying an atomic edit e (the insertion, deletion, or substitution of a subexpression) to a compound ex­pression x. The entailment relation between x and x' is found by projecting the entailment relation generated by e upward through x's semantic com­position tree. Substitutions generate relations ac­cording to the meanings of the substituends. Most deletions generate the rz relation {red socks rz socks). (Insertions are symmetric: they typically generate □.) However, some items have special behavior. For example, deleting (or inserting) not generates" {not hungry" hungry).

If two expressions are connected by a chain of atomic edits, we can determine the entailment re­lation between them by composing (as in Tarskian relation algebra) the entailment relations generated by each edit. The result may be a basic entailment relation, or may be a union of such relations, with larger unions conveying less information about en­tailment. This possibility, coupled with the need to find a chain of atomic edits which preserves rel­evant entailment relations, limits the power of the proof procedure described.

Implicatives. The account of implicatives and factives given by Nairn et al. hinges on a classi­fication of implicative and factive operators into nine implication signatures, according to their implications—positive (+), negative (-), or null (o)—in both positive and negative contexts. Thus refuse has implication signature -lo, because it car­ries a negative implication in a positive context {re­fused to dance implies didn't dance), and no impli­cation in a negative context {didn't refuse to dance implies neither danced nor didn't dance).

Most of the phenomena observed by Nairn et al. can be explained within our framework by spec­ifying, for each signature, the relation generated when an operator of that signature is deleted from a compound expression. For example, deleting sig­nature -lo generates | {Jim refused to dance \ Jim danced); under negation, this is projected as ^ {Jim didn't refuse to dance ^ Jim didn't dance).

By contrast, deleting signature o/- generates □ {Jim attempted to dance □ Jim danced); under negation, this is projected as rz {Jim didn't attempt to dance rz Jim didn't dance)?

We can also account for monotonicity ef­fects of implicative and factive operators by describing the projectivity properties of each implication signature: signatures +/-, +/o, and ol- are upward monotone {attempt to tango rz attempt to dance); signatures -/+, -lo, and o/+ are downward monotone {refuse to dance rz refuse to tango); and signatures +/+, -/-, and o/o are non-monotone {think dancing is fun # think tangoing is fun).

Our implementation of natural logic, the NatLog system, uses a multi-stage architecture like those of (Marsi and Krahmer, 2005; MacCartney et al., 2006), comprising (1) linguistic analysis, (2) align­ment, (3) lexical entailment classification, (4) en­tailment projection, and (5) entailment composi­tion. We'll use the following inference as a run­ning example:

(2) Jimmy Dean refused to move without blue jeans. James Dean didn't dance without pants.

The example is admittedly contrived, but it com­pactly exhibits containment, exclusion, and im-plicativity. How the NatLog system handles this example is depicted in table 1.

Linguistic analysis. Relative to other NLI sys­tems, the NatLog system does comparatively lit­tle linguistic pre-processing. We rely on the Stan­ford parser (Klein and Manning, 2003), a Penn Treebank-trained statistical parser, for tokeniza-tion, lemmatization, part-of-speech tagging, and phrase-structure parsing.

By far the most important analysis performed at this stage, however, is projectivity marking, in which we compute the effective projectivity for each token span in each input sentence. In the premise of (2), for example, we want to determine that the effective projectivity is upward monotone

unary operator: without pattern: IN < / " [Ww] ithout$/ argument 1: projectivity j on dominating pp pattern: _ > PP=proj binary operator: most pattern: JJS < /"[Mm]ost$/ !> QP argument 1: projectivity 1/1 on dominating np argument 2: projectivity | on dominating s

for Jimmy Dean and refused to, downward mono­tone for move and without, and upward monotone for blue and jeans. Our choice of a Treebank-trained parser (driven by the goal of broad cov­erage) complicates this effort, because the nesting of constituents in phrase-structure parses does not always correspond to the structure of idealized se­mantic composition trees. Our solution is imper­fect but effective. We define a list of operator types affecting projectivity (e.g., implicatives like refuse to, prepositions like without), and for each type we specify its arity and a Tregex tree pattern (Levy and Andrew, 2006) which permits us to identify its oc­currences in our Treebank parses. We also specify, for each argument position of each type, both the projectivity class and another Tregex pattern which helps us to determine the sentence span over which the operator's effect is projected. (Figure 1 shows some example definitions.) The marking process computes these projections, performs projectivity composition where needed, and marks each token span with its final effective projectivity.

Alignment. Next, we establish an alignment be­tween the premise P and hypothesis H, repre­sented by a sequence of atomic edits over spans of word tokens. This alignment representation is symmetric and many-to-many, and is general enough to include various other alignment representations as special cases. We define four edit types: deletion (del) of a span from P, insertion (ins) of a span into H, substitution (sub) of an H span for a P span, and match (mat) of an H span to a P span. Each edit is parameterized by the to­ken indices at which it operates, and edit indices may "cross", permitting representation of move­ment. The first four lines of table 1 depict a possi­ble alignment for our example problem.

An alignment decomposes an inference problem into a sequence of atomic inference problems, one for each atomic edit. Note that edits are ordered, and that this ordering defines a path from P to H through intermediate forms. (Edit order need not correspond to sentence order, though it does in our example.) The relative ordering of certain kinds of edits (e.g., the insertion of not) may influence the effective projectivity applicable for other edits; consequently, the NatLog system can reorder edits to maximize the benefit of the projectivity marking performed during linguistic analysis.

This paper does not present new algorithms for alignment; we focus instead on identifying en­tailment relations between aligned sentence pairs. The experiments described in sections 4 and 5 use alignments from other sources.

Lexical entailment classification. Much of the heavy lifting in the NatLog system is done by the lexical entailment model, which uses a classifier to predict an entailment relation for each atomic edit based solely on features of the lexical items in­volved, independent of context. (For example, this model should assign the entailment relation □ to the edit s\JB(move, dance), regardless of whether the effective projectivity at the locus of the edit is upward monotone, downward monotone, or some­thing else.) In the case of a sub edit, the features include:

• WordNet-derived measures of synonymy, hyponymy, and antonymy between sub-

The FraCaS test suite (Cooper et al., 1996) con­tains 346 NLI problems, divided into nine sections, each focused on a specific category of semantic phenomena (listed in table 3). Each problem con­sists of one or more premise sentences, a question sentence, and one of three answers: yes (the union of rz and =), no (the union of | andor unknown (the union of □, ^, and #). Table 2 shows some example problems.

To facilitate comparison with previous work, we have evaluated our system using a version of the FraCas data prepared by (MacCartney and Man­ning, 2007), in which multiple-premise problems (44% of the total) and problems lacking a hypoth­esis or a well-defined answer (3% of the total) are excluded; question sentences have been converted

Table 3: Performance on FraCaS problems (three-way classification). The columns show the number of problems, precision and recall for the yes class, and accuracy. Results for NatLog are broken out by section.

to declarative hypotheses; and alignments between premise and hypothesis have been automatically generated and manually corrected.

Results are shown in table 3. We achieve over­all accuracy of 70.49%, representing a 27% error reduction from (MacCartney and Manning, 2007). In the section concerning quantifiers, which is both the largest and the most amenable to natural logic, all problems but one are answered correctly. We also answer all but one problems correctly in the (admittedly small) section on attitudes, which in­volves implicatives and factives. Unsurprisingly, performance is mediocre in four sections concern­ing semantic phenomena (e.g., ellipsis) not rele­vant to natural logic and not modeled by the sys­tem. But in the other five sections (about 60% of the problems), we achieve accuracy of 87.04%, an error reduction of 61% from (MacCartney and

Manning, 2007). What's more, precision is high in nearly every section: even outside its areas of ex­pertise, the system rarely predicts entailment when none exists.

Since the NatLog system was developed with FraCaS problems in mind, these results do not con­stitute a proper evaluation on unseen test data. On the other hand, the system does no training on Fra­CaS data, and has had no opportunity to learn its biases. (Otherwise, accuracy on §4 could not fall so far below the baseline.) The system not only an­swers most problems correctly, but usually does so for valid reasons, particular within its areas of ex­pertise. All in all, the results fulfill our main goal in testing on FraCaS: to demonstrate the represen­tational and inferential adequacy of our model of natural logic.

The confusion matrix shown in table 4 reveals an interesting property of the NatLog system. The commonest confusions are those where the answer is yes but we guess unknown. This reflects both the bias toward yes in the FraCaS data, and the system's tendency to predict unknown (entailment relation 4f) when confused: given the composition rules for entailment relations, the system can pre­dict yes only if all atomic-level predictions are ei­ther rz or =.

NLI problems from the PASCAL RTE Challenge (Dagan et al., 2005) differ from FraCaS problems in several important ways. (See table 5 for ex­amples.) Instead of textbook examples of semantic phenomena, RTE problems are more natural-seeming, with premises collected "in the wild" from newswire. The premises are much longer, averaging 35 words (vs. 11 words for FraCaS). Also, RTE aims at binary classification: the RTE no combines the no and unk answers in FraCaS.

Due to the character of RTE problems, we do not expect NatLog to be a good general-purpose solution to solving all RTE problems. First, most RTE problems depend on forms of inference, such as paraphrase, temporal reasoning, or relation ex­traction, which NatLog is not designed to address. Second, in most RTE problems, the edit distance between premise and hypothesis is relatively large. More atomic edits means a greater chance that er­rors made in lexical entailment classification or projection will propagate, via entailment compo­sition, to the system's final output. Rather, in ap­plying NatLog to RTE, we hope to make reliable predictions on a subset of RTE problems, trading recall for precision. If we succeed, then we may be able to hybridize with a broad-coverage RTE system to obtain better results than either system individually—the same strategy that was adopted by (Bos and Markert, 2006) for their FOL-based system. For this purpose, we have chosen to use the Stanford RTE system described in (de Marn-effe et al., 2006). We also use the Stanford system to generate alignments when evaluating NatLog on RTE problems.

Table 6 shows the performance of NatLog on RTE3 data. Relative to the Stanford sys­tem, NatLog achieves high precision on its yes predictions—above 70%—suggesting that hy­bridizing may be effective. For comparison, the FOL-based system reported in (Bos and Markert, 2006) attained a similarly high precision of 76% on RTE2 problems, but was able to make a pos­itive prediction in only about 4% of cases. Nat-Log makes positive predictions far more often—in about 25% of cases.

The Stanford system makes yes/no predictions

Table 6: Performance of various systems on RTE3 (two-way classification). The columns show the data set used (800 problems each), the proportion of yes predictions, precision and recall for the yes class, and accuracy.

by thresholding a real-valued inference score. To construct a hybrid system, we adjust the Stanford inference scores by +A or —A, depending on whether or not NatLog predicts yes. We choose A by optimizing development set accuracy, while adjusting the threshold to generate balanced pre­dictions (that is, equal numbers of yes and no pre­dictions). As an additional experiment, we fix A at this value and then adjust the threshold to op­timize development set accuracy, resulting in an excess of yes predictions. (Since this optimiza­tion is based solely on development data, its use on test data is fully legitimate.) Results for these two cases are shown in table 6. The parameters tuned on development data gave good results on test data. The optimized hybrid system attained an absolute accuracy gain of 4% over the Stanford system, corresponding to an extra 32 problems an­swered correctly. This result is statistically signifi­cant (p < 0.05, McNemar's test, 2-tailed).

The gains attributable to NatLog are exempli­fied by problem 788 (table 5). NatLog sanctions the deletion of a restrictive modifier and an appos-itive from the premise, and recognizes that delet­ing a negation generates a contradiction; thus it correctly answers no. On the other hand, there are many RTE problems where NatLog's precision works against it. For example, NatLog answers no to problem 71 because it cannot account for the insertion of acts as in the hypothesis. Fortunately, both the Stanford system and the hybrid system an­swer this problem correctly.

We do not claim natural logic to be a universal solution for NLI. Many important types of infer­ence are not amenable to natural logic, includ­ing paraphrase (Eve was let go |= Eve lost her job), verb alternation (he drained the oil |= the oil drained), relation extraction (Aho, a trader at UBS, ... |= Aho works for UBS), common-sense reasoning (the sink overflowed \= the floor got wet), and so on.

Moreover, because natural logic has a weaker proof theory than FOL, some inferences lie beyond its deductive power. For example, it cannot explain inferences involving De Morgan's laws for quanti­fiers, as in Not all birds fly = Some birds don't fly.

However, by incorporating semantic contain­ment, semantic exclusion, and implicativity, the model of natural logic developed in this paper suc­ceeds in explaining a great variety of everyday pat­terns of inference. Ultimately, open-domain NLI is likely to require combining disparate reasoners, and a facility for natural logic inference is a good candidate to be a component of such a solution.

Acknowledgements The authors wish to thank the anonymous reviewers for their helpful com­ments. This work was supported in part by ARDA's Advanced Question Answering for Intel­ligence (AQUAINT) Program.