<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="11"/><title>It's a Contradiction – no, it's not: A Case Study using Functional Relations</title><author surname="Ritter" givenname="Alan"><org  name="University of Washington" country="USA" city="Seattle"/></author><author surname="Downey" givenname="Doug"><org  name="University of Washington" country="USA" city="Seattle"/></author><author surname="Soderland" givenname="Stephen"><org  name="University of Washington" country="USA" city="Seattle"/></author><author surname="Etzioni" givenname="Oren"><org  name="University of Washington" country="USA" city="Seattle"/></author></firstpageheader><frontmatter><p><b>It's a Contradiction—No, it's Not: A Case Study using Functional Relations</b></p><p><b>Alan Ritter, Doug Downey, Stephen Soderland and Oren Etzioni</b></p><p>Turing Center Department of Computer Science and Engineering University of Washington</p><p>Box 352350 Seattle, WA 98195, USA {aritter,ddowney,soderlan,etzioni}@cs.washington.edu</p></frontmatter><abstract>Contradiction Detection (CD) in text is a difficult NLP task. We investigate CD over functions <i>(e.g., </i>BornIn(Person)=Place), and present a domain-independent algorithm that automatically discovers phrases denoting functions with high precision. Previous work on CD has investigated hand-chosen sentence pairs. In contrast, we automatically harvested from the Web pairs of sentences that <i>appear </i>contradictory, but were surprised to find that most pairs are in fact consistent. For example, "Mozart was born in Salzburg" does <i>not </i>con­tradict "Mozart was born in Austria" despite the functional nature of the phrase "was born in". We show that background knowledge about meronyms (e.g., Salzburg is in Austria), synonyms, functions, and more is essential for success in the CD task. </abstract></header><body><section number="1" title="Introduction and Motivation"><p>Detecting contradictory statements is an important and challenging NLP task with a wide range of potential applications including analysis of politi­cal discourse, of scientific literature, and more (de Marneffe et al., 2008; Condoravdi et al., 2003; Harabagiu et al., 2006). De Marneffe et <i>al. </i>present a model of CD that defines the task, analyzes different types of contradictions, and reports on a CD system. They report 23% precision and 19% recall at detect­ing contradictions in the RTE-3 data set (Voorhees, 2008). Although RTE-3 contains a wide variety of contradictions, it does not reflect the prevalence of <i>seeming </i>contradictions and the paucity of <i>genuine </i>contradictions, which we have found in our corpus.</p><subsection number="1.1" title="Contradictions and World Knowledge"><p>Our paper is motivated in part by de Marneffe <i>et al. </i>'s work, but with some important differences. First, we introduce a simple logical foundation for the CD task, which suggests that extensive world knowl­edge is essential for building a domain-independent CD system. Second, we automatically generate a large corpus of <i>apparent </i>contradictions found in ar­bitrary Web text. We show that most of these appar­ent contradictions are actually consistent statements due to meronyms (Alan Turing was born in London <i>and </i>in England), synonyms (George Bush is mar­ried to <i>both </i>Mrs. Bush and Laura Bush), hypernyms (Mozart died of <i>both </i>renal failure and kidney dis­ease), and reference ambiguity (one John Smith was born in 1997 and a <i>different </i>John Smith in 1883). Next, we show how background knowledge enables a CD system to discard seeming contradictions and focus on genuine ones.</p><p>De Marneffe <i>et al. </i>introduced a typology of con­tradiction in text, but focused primarily on contra­dictions that can be detected from linguistic evi­dence <i>(e.g. </i>negation, antonymy, and structural or lexical disagreements). We extend their analysis to a class of contradictions that can only be detected utilizing background knowledge. Consider for ex­ample the following sentences:</p><p>1) "Mozart was born in Salzburg."</p><p>2) "Mozart was born in Vienna."</p><p>3) "Mozart visited Salzburg."</p><p>4) "Mozart visited Vienna."</p><p>Sentences 1 &amp; 2 are contradictory, but 3 &amp; 4 are not. Why is that? The distinction is not syntactic. Rather, sentences 1 and 2 are contradictory because</p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">11</doubt><p><i>Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, </i>pages 11-20, Honolulu, October 2008. ©2008 Association for Computational Linguistics<page local="2" global="12"/></p><p>the relation expressed by the phrase "was born in" can be characterized here as a <i>function </i>from peo­ple's names to their unique birthplaces. In contrast, "visited" does not denote a functional relation.<footnote anchor="1"/></p><p>We cannot assume that a CD system knows, in advance, all the functional relations that might ap­pear in a corpus. Thus, a central challenge for a function-based CD system is to determine which re­lations are functional based on a corpus. Intuitively, we might expect that "functional phrases" such as "was born in" would typically map person names to unique place names, making function detection easy. But, in fact, function detection is surprisingly difficult because name ambiguity (e.g., John Smith), common nouns (e.g., "dad" or "mom"), definite de­scriptions (e.g., "the president"), and other linguistic phenomena can mask functions in text. For example, the two sentences "John Smith was born in 1997." and "John Smith was born in 1883." can be viewed as either evidence that "was born in" does not de­note a function or, alternatively, that "John Smith" is ambiguous.</p></subsection><subsection number="1.2" title="A CD System Based on Functions"><p>We report on the AuCONTRAIRE CD system, which addresses each of the above challenges. First, AU-Contraire identifies "functional phrases" statis­tically (Section 3). Second, AUCONTRAIRE uses these phrases to automatically create a large cor­pus of apparent contradictions (Section 4.2). Fi­nally, AUCONTRAIRE sifts through this corpus to find genuine contradictions using knowledge about synonymy, meronymy, argument types, and ambi­guity (Section 4.3).</p><p>Instead of analyzing sentences directly, AUCON-TRAIRE relies on the TEXTRUNNER Open Informa­tion Extraction system (Banko et al., 2007; Banko and Etzioni, 2008) to map each sentence to one or more tuples that represent the entities in the sen­tences and the relationships between them <i>(e.g., wasJ?ornJn(Mozart,Salzburg)). </i>Using extracted tu­ples greatly simplifies the CD task, because nu­merous syntactic problems <i>(e.g., </i>anaphora, rela­tive clauses) and semantic challenges (e.g., quantifi­cation, counterfactuals, temporal qualification) are delegated to TEXTRUNNER or simply ignored. Nev­ertheless, extracted tuples are a convenient approxi­mation of sentence content, which enables us to fo­cus on function detection and function-based CD. Our contributions are the following:</p><footnote label="1">Although we focus on function-based CD in our case study, we believe that our observations apply to other types of CD as well.</footnote><p>• We present a novel model of the Contradiction Detection (CD) task, which offers a simple log­ical foundation for the task and emphasizes the central role of background knowledge.</p><p>• We introduce and evaluate a new EM-style al­gorithm for detecting whether phrases denote functional relations and whether nouns (e.g., "dad") are ambiguous, which enables a CD sys­tem to identify functions in arbitrary domains.</p><p>• We automatically generate a corpus of seem­ing contradictions from Web text, and report on a set of experiments over this corpus, which provide a baseline for future work on statistical function identification and CD. <footnote anchor="2"/></p></subsection></section><section number="2" title="A Logical Foundation for CD"><p>On what basis can a CD system conclude that two statements T and H are contradictory? Logically, contradiction holds when T |= —H. As de Marneffe <i>et al. </i>point out, this occurs when <i>T </i>and <i>H </i>contain antonyms, negation, or other lexical elements that suggest that <i>T </i>and <i>H </i>are directly contradictory. But other types of contradictions can only be detected with the help of a body of background knowledge K: In these cases, T and H <i>alone </i>are mutually con­sistent. That is,</p><doubt alpha="45.5" length="11" tooSmall="False" monospace="0.0">T|=—HaH|=—T</doubt><p>A contradiction between T and H arises only in the context of K. That is:</p><doubt alpha="39.1" length="23" tooSmall="False" monospace="0.0">((KaT)|=—H)v((KaH)|=—T)</doubt><p>Consider the example of Mozart s birthplace in the introduction. To detect a contradiction, a CD system must know that A) "Mozart" refers to the same entity in both sentences, that B) "was born in" denotes a functional relation, and that C) Vienna and Salzburg are inconsistent locations.</p><footnote label="2">The corpus is available at http://www.cs . washington.edu/research/aucontraire/</footnote><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">12</doubt><page local="3" global="13"/><p>Of course, world knowledge, and reasoning about text, are often uncertain, which leads us to associate probabilities with a CD system's conclusions. Nev­ertheless, the knowledge base K is essential for CD.</p><p>We now turn to a probabilistic model that helps us simultaneously estimate the functionality of re­lations (B in the above example) and ambiguity of argument values (A above). Section 4 describes the remaining components of AuContraire.</p></section><section number="3" title="Detecting Functionality and Ambiguity"><p>This section introduces a formal model for comput­ing the probability that a phrase denotes a function based on a set of extracted tuples. An extracted tuple takes the form R(x, y) where (roughly) x is the sub­ject of a sentence, y is the object, and R is a phrase denoting the relationship between them. If the re­lation denoted by <i>R </i>is functional, then typically the object <i>y </i>is a function of the subject <i>x. </i>Thus, our dis­cussion focuses on this possibility, though the anal­ysis is easily extended to the symmetric case.</p><p>Logically, a relation <i>R </i>is functional in a vari­able <i>x </i>if it maps it to a unique variable <i>y: </i>vx,yi,y2 R(x,yi) a R(x,y2) == yi = y2. Thus, given a large random sample of ground instances of R, we could detect with high confidence whether R is functional. In text, the situation is far more com­plex due to ambiguity, polysemy, synonymy, and other linguistic phenomena. Deciding whether R is functional becomes a probabilistic assessment based on aggregated textual evidence.</p><p>The main evidence that a relation R(x, y) is func­tional comes from the distribution of y values for a given x value. If R denotes a function and x is unambiguous, then we expect the extractions to be predominantly a single y value, with a few outliers due to noise. We aggregate the evidence that R is <i>locally </i>functional for a particular x value to assess whether R is <i>globally </i>functional for all x.</p><p>We refer to a set of extractions with the same relation R and argument x as a <i>contradiction set </i>R(x, •). Figure 1 shows three example contradic­tion sets. Each example illustrates a situation com­monly found in our data. Example A in Figure 1 shows strong evidence for a functional relation. 66 out of 70 TextRunner extractions for <i>was-born Jn (Mozart, PLACE) </i>have the same y value. An am­biguous x argument, however, can make a functional relation <i>appear </i>non-functional. Example B depicts a distribution of y values that appears less functional due to the fact that "John Adams" refers to multiple, distinct real-world individuals with that name. Finally, example C exhibits evidence for a non-functional relation.</p><p>A. was_bornin(Mozart, PLACE):</p><doubt alpha="60.0" length="35" tooSmall="False" monospace="0.0">Salzburg(66), Germany(3), Vienna(1)</doubt><p>B. was_born_in(John Adams, PLACE):</p><doubt alpha="61.5" length="39" tooSmall="False" monospace="0.0">Braintree(12), Quincy(10), Worcester(8)</doubt><p>C. lived_in(Mozart, PLACE):</p><doubt alpha="57.1" length="35" tooSmall="False" monospace="0.0">Vienna(20), Prague(13), Salzburg(5)</doubt><p>Figure 1: Functional relations such as example A have a different distribution of y values than non-functional rela­tions such as C. However, an ambiguous x argument as in B, can make a functional relation <i>appear </i>non-functional.</p><subsection number="3.1" title="Formal Model of Functions in Text"><p>To decide whether R is functional in x for all x, we first consider how to detect whether R is <i>lo­cally functional </i>for a particular value of x. The local functionality of R with respect to x is the probabil­ity that R is functional estimated solely on evidence from the distribution of y values in a contradiction set R(x, •).</p><p>To decide the probability that R is a function, we define global functionality as the average local func­tionality score for each x, weighted by the probabil­ity that x is unambiguous. Below, we outline an EM-style algorithm that alternately estimates the proba­bility that R is functional and the probability that x is ambiguous.</p><p>Let R* indicate the event that the relation R is locally functional for the argument x, and that x is locally unambiguous for R. Also, let D indicate the set of observed tuples, and define DR(x .) as the multi-set containing the frequencies for extractions of the form R(x, •). For example the distribution of extractions from Figure 1 for example A is <i>Dw</i><i>asJ&gt;ornJ,n</i><i>(M</i><i>ozaTt,-) = {66' </i><footnote anchor="3"/>' <footnote anchor="1"/>}.</p><p>Let #R be the probability that R(x, •) is locally functional for a random x, and let 6^ be the vector of these parameters across all relations R. Likewise, #u represents the probability that x is locally unam­biguous for random R, and 6u the vector for all x.</p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">13</doubt><page local="4" global="14"/><p>We wish to determine the <i>maximum a pos­teriori </i>(MAP) functionality and ambiguity pa­rameters given the observed data D, that is arg max<b>q/ q« </b>P(ef, eu|D). By Bayes Rule:</p><doubt alpha="45.5" length="11" tooSmall="False" monospace="0.0">P(9f,eu|D)=</doubt><doubt alpha="61.1" length="18" tooSmall="False" monospace="0.0">p (d|ef,eu)p(ef,eu</doubt><doubt alpha="40.0" length="5" tooSmall="False" monospace="0.0">p (d)</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(1)</doubt><p>We outline a generative model for the data, P(D|ef, <i>eu). </i>Let us assume that the event R* de­pends only on <i>6fR </i>and and further assume that given these two parameters, local ambiguity and lo­cal functionality are conditionally independent. We obtain the following expression for the probability of <i>R*x</i><i> </i>given the parameters:</p><doubt alpha="63.6" length="11" tooSmall="False" monospace="0.0">p (rX|ef,eu</doubt><p>We assume each set of data DR(x .<b>) </b>is gener­ated independently of all other data and parameters, given <i>R*x. </i>From this and the above we have:</p><doubt alpha="47.4" length="19" tooSmall="False" monospace="0.0">p(D|ef,eu) =J](p (d</doubt><doubt alpha="25.0" length="4" tooSmall="False" monospace="0.0">I7?*</doubt><doubt alpha="44.4" length="9" tooSmall="False" monospace="0.0">R(x,-)|Rx</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">R,x</doubt><doubt alpha="36.7" length="30" tooSmall="False" monospace="0.0">+p (Dr(x,.)|-rx)(i -efR0u))(2)</doubt><p>These independence assumptions allow us to ex­press P (D|ef, eu) in terms of distributions over DR(xr) given whether or not <i>R% </i>holds. We use the Urns model as described in (Downey et al., 2005) to estimate these probabilities based on binomial distributions. In the single-urn Urns model that we utilize, the extraction process is modeled as draws of labeled balls from an urn, where the labels are either correct extractions or errors, and different labels can be repeated on varying numbers of balls in the urn.</p><p>Let k = max Dr(x_.), and let n = <i>J2</i><i> </i>Dr^^; we will approximate the distribution over DR(xr) in terms of k and n. If <i>R(x,</i><i> </i>•) is locally func­tional and unambiguous, there is exactly one cor­rect extraction label in the urn (potentially repeated multiple times). Because the probability of correct­ness tends to increase with extraction frequency, we make the simplifying assumption that the most fre­quently extracted element is correct.<footnote anchor="3"/> In this case, k is the number of correct extractions, which by the</p><footnote label="3">As this assumption is invalid when there is not a unique maximal element, we default to the prior P (R X) in that case.</footnote><p>Urns model has a binomial distribution with pa­rameters n and p, where <i>p</i><i> </i>is the precision of the ex­traction process. If R(x, •) is <i>not </i>locally functional and unambiguous, then we expect k to typically take on smaller values. Empirically, the underlying fre­quency of the most frequent element in the -Rx case tends to follow a Beta distribution.</p><p>Under the model, the probability of the evidence given Rx is:</p><doubt alpha="50.0" length="36" tooSmall="False" monospace="0.0">P(Dr(x,.)|Rx)«P(k,n|Rx)=Qpfc(1-p)n_k</doubt><p>And the probability of the evidence given - Rx is:</p><doubt alpha="42.3" length="26" tooSmall="False" monospace="0.0">P(Dr(x,.)|-Rx) «P(k,n|-Rx)</doubt><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">© Jo</doubt><doubt alpha="41.7" length="24" tooSmall="False" monospace="0.0">1p'k+a/-i(1_p,)n+ß/-i-fc</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">dp'</doubt><doubt alpha="38.5" length="26" tooSmall="False" monospace="0.0">(n)r(n - k + ßf )r(af + k)</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(3)</doubt><doubt alpha="57.9" length="19" tooSmall="False" monospace="0.0">B(af ,ßf)r(af+ßf+n)</doubt><p>where n is the sum over DR(xr), r is the Gamma function and <i>B </i>is the Beta function. aff and ßf are the parameters of the Beta distribution for the - Rxcase. These parameters and the prior distributions are estimated empirically, based on a sample of the data set ofrelations described in Section 5.1.</p></subsection><subsection number="3.2" title="Estimating Functionality and Ambiguity"><p>Substituting Equation 3 into Equation 2 and apply­ing an appropriate prior gives the probability of pa­rameters ef and eu given the observed data D. However, Equation 2 contains a large product of sums—with two independent vectors of coefficients, ef and eu—making it difficult to optimize analyti­cally.</p><p>If we knew which arguments were ambiguous, we would ignore them in computing the function­ality of a relation. Likewise, if we knew which rela­tions were non-functional, we would ignore them in computing the ambiguity of an argument. Instead, we initialize the ef and eu arrays randomly, and then execute an algorithm similar to Expectation-Maximization (EM) (Dempster et al., 1977) to arrive at a high-probability setting of the parameters.</p><p>Note that if eu is fixed, we can compute the ex­pected fraction of locally unambiguous arguments x for which R is locally functional, using DR(x/r) and<page local="5" global="15"/></p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">14</doubt><p>Equation 3. Likewise, for fixed ef, for any given x we can compute the expected fraction of locally functional relations R that are locally unambiguous for x.</p><p>Specifically, we repeat until convergence:</p><p>1. Set öR = sR Ex P(Rx|Dr(x,.))#u for all R.</p><p>2. Set #u = sX Er P(Rx|Dr(x,.))^R for all x.</p><p>In both steps above, the sums are taken over only those x or R for which DR(xr) is non-empty. Also, the normalizer sR = Ex #u and likewise sx = Er öR.</p><p>As in standard EM, we iteratively update our pa­rameter values based on an expectation computed over the unknown variables. However, we alter­nately optimize two disjoint sets of parameters (the functionality and ambiguity parameters), rather than just a single set of parameters as in standard EM. Investigating the optimality guarantees and conver­gence properties of our algorithm is an item of future work.</p><p>By iteratively setting the parameters to the expec­tations in steps 1 and 2, we arrive at a good setting of the parameters. Section 5.2 reports on the perfor­mance of this algorithm in practice.</p></subsection></section><section number="4" title="System Overview"><p>AuContraire identifies phrases denoting func­tional relations and utilizes these to find contradic­tory assertions in a massive, open-domain corpus of text.</p><p>AuContraire begins by finding extractions of the form R(x,y), and identifies a set of relations R that have a high probability of being functional. Next, AuContraire identifies contradiction sets of the form R(x, •). In practice, most contradiction sets turned out to consist overwhelmingly of seem­ing contradictions—assertions that do not actually contradict each other for a variety of reasons that we enumerate in section 4.3. Thus, a major chal­lenge for AuContraire is to tease apart which pairs of assertions in R(x, •) represent genuine con­tradictions.</p><p>Here are the main components of AuContraire as illustrated in Figure 2:</p><p><b>Extractor: </b>Create a set of extracted assertions E from a large corpus of Web pages or other docu­ments. Each extraction R(x, y) has a probability p</p><p><b>Web Corpus</b></p><p><i>Functional </i><i>(R), </i><i>pf</i></p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">X</doubt><p>Gazetteer ■—» WordNet —►</p><p><b>Contradiction Detector</b></p><doubt alpha="63.6" length="33" tooSmall="False" monospace="0.0">Contradiction{R{x,y1),R{x,y2)),pc</doubt><p>Figure 2: AuContraire architecture of being correct.</p><p><b>Function Learner: </b>Discover a set of functional re­lations F from among the relations in E. Assign to each relation in F a probability pf that it is func­tional.</p><p><b>Contradiction Detector: </b>Query E for assertions with a relation R in F, and identify sets C of po­tentially contradictory assertions. Filter out seeming contradictions in C by reasoning about synonymy, meronymy, argument types, and argument ambigu­ity. Assign to each potential contradiction a proba­bility pc that it is a genuine contradiction.</p><subsection number="4.1" title="Extracting Factual Assertions"><p>AuContraire needs to explore a large set of factual assertions, since genuine contradictions are quite rare (see Section 5). We used a set of extrac­tions E from the Open Information Extraction sys­tem, TextRunner (Banko et al., 2007), which was run on a set of 117 million Web pages.</p><p>TextRunner does not require a pre-defined set of relations, but instead uses shallow linguistic anal­ysis and a domain-independent model to identify phrases from the text that serve as relations and phrases that serve as arguments to that relation. TextRunner creates a set of extractions in a sin­gle pass over the Web page collection and provides an index to query the vast set of extractions.</p><p>Although its extractions are noisy, TextRunner provides a probability that the extractions are correct, based in part on corroboration of facts from different Web pages (Downey et al., 2005).<page local="6" global="16"/></p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">15</doubt></subsection><subsection number="4.2" title="Finding Potential Contradictions"><p>The next step of AuContraire is to find contra­diction sets in E.</p><p>We used the methods described in Section 3 to estimate the functionality of the most frequent rela­tions in E. For each relation R that AuContraire has judged to be functional, we identify contradic­tion sets R(x, •), where a relation R and domain ar­gument x have multiple range arguments y.</p></subsection><subsection number="4.3" title="Handling Seeming Contradictions"><p>For a variety of reasons, a pair of extractions R(x,yi) and R(x,y2) may not be actually contra­dictory. The following is a list of the major sources of false positives—pairs of extractions that are not genuine contradictions, and how they are handled by AuContraire. The features indicative of each condition are combined using Logistic Regression, in order to estimate the probability that a given pair, {R(x, y1), R(x, y2)} is a genuine contradiction.</p><p><b>Synonyms: </b>The set of potential contradictions <i>died-from(Mozart</i><i>r)</i><i> </i>may contain assertions that Mozart died from <i>renal failure </i>and that he died from <i>kidney failure. </i>These are distinct values of y, but do not contradict each other, as the two terms are synonyms. AuContraire uses a variety of knowl­edge sources to handle synonyms. WordNet is a re­liable source of synonyms, particularly for common nouns, but has limited recall. AuContraire also utilizes synonyms generated by Resolver (Yates and Etzioni, 2007)— a system that identifies syn­onyms from TextRunner extractions. Addition­ally, AuContraire uses edit-distance and token-based string similarity (Cohen et al., 2003) between apparently contradictory values of y to identify syn­onyms.</p><p><b>Meronyms: </b>For some relations, there is no con­tradiction when yi and y2 share a meronym, <i>i.e. </i>"part of" relation. For example, in the set <i>borndn(Mozart^) </i>there is no contradiction be­tween the y values "Salzburg" and "Austria", but "Salzburg" conflicts with "Vienna". Although this is only true in cases where y occurs in an up­ward monotone context (MacCartney and Manning, 2007), in practice genuine contradictions between y-values sharing a meronym relationship are ex­tremely rare. We therefore simply assigned contra­dictions between meronyms a probability close to zero. We used the Tipster Gazetteer<footnote anchor="4"/> and WordNet to identify meronyms, both of which have high pre­cision but low coverage.</p><p><b>Argument Typing: </b>Two y values are not contra­dictory if they are of different argument types. For example, the relation <i>borndn </i>can take a date or a location for the y value. While a person can be born in only one year and in only one city, a per­son can be born in both a year and a city. To avoid such false positives, AuContraire uses a sim­ple named-entity tagger<footnote anchor="5"/> in combination with large dictionaries of person and location names to as­sign high-level types (person, location, date, other) to each argument. AuContraire filters out ex­tractions from a contradiction set that do not have matching argument types.</p><p><b>Ambiguity: </b>As pointed out in Section 3, false con­tradictions arise when a single x value refers to mul­tiple real-world entities. For example, if the con­tradiction set <i>bornJn(John Sutherland, </i><i>•)</i><i> </i>includes birth years of both 1827 and 1878, is one of these a mistake, or do we have a grandfather and grandson with the same name? AuContraire computes the probability that an x value is unambiguous as part of its Function Learner (see Section 3). An x value can be identified as ambiguous if its distribution of y values is non-functional for multiple functional re­lations.</p><p>If a pair of extractions, {R(x, y1), R(x, y2)}, does not fall into any of the above categories and R is functional, then it is likely that the sentences under­lying the extractions are indeed contradictory. We combined the various knowledge sources described above using Logistic Regression, and used 10-fold cross-validation to automatically tune the weights associated with each knowledge source. In addi­tion, the learning algorithm also utilizes the follow­ing features:</p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">f</doubt><p>• Global functionality of the relation, ÖR</p><p>• Global unambiguity of x,</p><footnote label="4">http://crl.nmsu.edu/cgi-bin/Tools/CLR/ clrcat</footnote><p>http://search.cpan.org/~simon/ Lingua-EN-NamedEntity-1.1/NamedEntity.pm<page local="7" global="17"/></p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">16</doubt><p>• Local functionality of R(x, •)</p><p>• String similarity (a combination oftoken-based similarity and edit-distance) between y1 and y2</p><p>• The argument types (person, location, date, or other)</p><p>The learned model is then used to estimate how likely a potential contradiction {R(x, y1), R(x, y2)} is to be genuine.</p></subsection></section><section number="5" title="Experimental Results"><p>We evaluated several aspects of AuContraire: its ability to detect functional relations and to de­tect ambiguous arguments (Section 5.2); its preci­sion and recall in contradiction detection (Section 5.3); and the contribution of AuContraire's key knowledge sources (Section 5.4).</p><subsection number="5.1" title="Data Set"><p>To evaluate AuContraire we used TextRun-NER's extractions from a corpus of 117 million Web pages. We restricted our data set to the 1,000 most frequent relations, in part to keep the experiments tractable and also to ensure sufficient statistical sup­port for identifying functional relations.</p><p>We labeled each relation as functional or not, and computed an estimate of the probability it is functional as described in section 3.2. Section 5.2 presents the results of the Function Learner on this set of relations. We took the top 2% (20 relations) as F, the set of functional relations in our exper­iments. Out of these, 75% are indeed functional. Some examples include: <i>was born in, died in, </i>and <i>was founded by.</i></p><p>There were 1.2 million extractions for all thou­sand relations, and about 20,000 extractions in 6,000 contradiction sets for all relations in F.</p><p>We hand-tagged 10% of the contradiction sets R(x, •) where R G F, discarding any sets with over 20 distinct y values since the x argument for that set is almost certainly ambiguous. This resulted in a data set of 567 contradiction sets containing a total of 2,564 extractions and 8,844 potentially contradic­tory pairs of extractions.</p><p>We labeled each of these 8,844 pairs as contradic­tory or not. In each case, we inspected the original sentences, and if the distinction was unclear, con­sulted the original source Web pages, Wikipedia ar­ticles, and Web search engine results.</p><p>In our data set, genuine contradictions over func­tional relations are surprisingly rare. We found only 110 genuine contradictions in the hand-tagged sam­ple, only 1.2% of the potential contradiction pairs.</p></subsection><subsection number="5.2" title="Detecting Functionality and Ambiguity"><p>We ran AuCONTRAIRE's EM algorithm on the thousand most frequent relations. Performance con­verged after 5 iterations resulting in estimates of the probability that each relation is functional and each x argument is unambiguous. We used these proba­bilities to generate the precision-recall curves shown in Figure 3.</p><p>The graph on the left shows results for function­ality, while the graph on the right shows precision at finding unambiguous arguments. The solid lines are results after 5 iterations of EM, and the dashed lines are from computing functionality or ambiguity with­out EM <i>(i.e. </i>assuming uniform values of ec when computing ef and vice versa). The EM algorithm improved results for both functionality and ambigu­ity, increasing area under curve (AUC) by 19% for functionality and by 31% for ambiguity.</p><p>Of course, the ultimate test of how well AuCon­traire can identify functional relations is how well the Contradiction Detector performs on automati­cally identified functional relations.</p></subsection><subsection number="5.3" title="Detecting Contradictions"><p>We conducted experiments to evaluate how well AuContraire distinguishes genuine contradic­tions from false positives.</p><p>The bold line in Figure 4 depicts AUCONTRAIRE performance on the distribution of contradictions and seeming contradictions found in actual Web data. The dashed line shows the performance of Au­Contraire on an artificially "balanced" data set that we constructed to contain 50% genuine contra­dictions and 50% seeming ones.</p><p>Previous research in CD presented results on manually selected data sets with a relatively bal­anced mix of positive and negative instances. As Figure 4 suggests, this is a much easier problem than CD "in the wild". The data gathered from the Web is badly skewed, containing only 1.2% genuine con­tradictions.</p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">17</doubt><page local="8" global="18"/><p><b>Functionality</b> <b>Ambiguity</b></p><doubt alpha="26.1" length="23" tooSmall="False" monospace="0.0">n-1-f-1-1-r n-1-f-1-1-r</doubt><doubt alpha="0.0" length="110" tooSmall="False" monospace="0.0">0.0        0.2        0.4        0.6        0.8        1.0 0.0        0.2        0.4        0.6        0.8 1.0</doubt><p>Recall Recall</p><p>Figure 3: After 5 iterations of EM, AuContraire achieves a 19% boost to area under the precision-recall curve (AUC) for functionality detection, and a 31% boost to AUC for ambiguity detection.</p><doubt alpha="33.3" length="6" tooSmall="True" monospace="0.0">u&gt; o 0</doubt><doubt alpha="75.0" length="4" tooSmall="False" monospace="0.0">CM Ö</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">O Ö</doubt><doubt alpha="10.3" length="58" tooSmall="False" monospace="0.0">0.0        0.2        0.4        0.6        0.8 1.0 Recall</doubt><p>Figure 4: Performance of AuContraire at distinguish­ing genuine contradictions from false positives. The bold line is results on the actual distribution of data from the Web. The dashed line is from a data set constructed to have 50% positive and 50% negative instances.</p></subsection><subsection number="5.4" title="Contribution of Knowledge Sources"><p>We carried out an ablation study to quantify how much each knowledge source contributes to Au-Contraire's performance. Since most of the knowledge sources do not apply to numeric argu­ment values, we excluded the extractions where y is a number in this study. As shown in Figure 5, performance of AuContraire degrades with no knowledge of synonyms (NS), with no knowledge of meronyms (NM), and especially without argu­ment typing (NT). Conversely, improvements to any of these three components would likely improve the performance of AUCONTRAIRE.</p><p>The relatively small drop in performance from no meronyms does not indicate that meronyms are not essential to our task, only that our knowledge sources for meronyms were not as useful as we hoped. The Tipster Gazetteer has surprisingly low coverage for our data set. It contains only 41% of the y values that are locations. Many of these are matches on a different location with the same name, which results in incorrect meronym information. We estimate that a gazetteer with complete coverage would <i>increase </i>area under the curve by approxi­mately 40% compared to a system with meronyms from the Tipster Gazetteer and WordNet.</p><p><b>Percentage AUC</b></p><p>Figure 5: Area under the precision-recall curve for the full AuContraire and for AuContraire with knowl­edge removed. NS has no synonym knowledge; NM has no meronym knowledge; NT has no argument typing.</p><p>To analyze the errors made by AuContraire, we hand-labeled all false-positives at the point of maximum F-score: 29% Recall and 48% Precision.</p><doubt alpha="100.0" length="2" tooSmall="True" monospace="0.0">CO</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">ö</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">o o</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">o</doubt><p>AuContraire<page local="9" global="19"/></p><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">NS</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">NM</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">NT</doubt><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">18</doubt><p>Figure 6 reveals the central importance of world knowledge for the CD task. About half of the errors (49%) are due to ambiguous x-arguments, which we found to be one of the most persistent obstacles to discovering genuine contradictions. A sizable por­tion is due to missing meronyms (34%) and missing synonyms (14%), suggesting that lexical resources with broader coverage than WordNet and the Tipster Gazetteer would substantially improve performance. Surprisingly, only 3% are due to errors in the extrac­tion process.</p><figure caption="Figure 6: Sources oferrors incontradictiondetection."></figure><p>All of our experimental results are based on the automatically discovered set of functions F. We would expect AuContraire's performance to im­prove substantially if it were given a large set of functional relations as input.</p></subsection></section><section number="6" title="Related Work"><p>Condoravdi <i>et al. </i>(2003) first proposed contradiction detection as an important NLP task, and Harabagiu <i>et al. </i>(2006) were the first to report results on con­tradiction detection using negation, although their evaluation corpus was a balanced data set built by manually negating entailments in a data set from the Recognizing Textual Entailment confer­ences (RTE) (Dagan et al., 2005). De Marneffe <i>et al. </i>(2008) reported experimental results on a contra­diction corpus created by annotating the RTE data sets.</p><p>RTE-3 included an optional task, requiring sys­tems to make a 3-way distinction: {entails, contra­dicts, neither} (Voorhees, 2008). The average per­formance for contradictions on the RTE-3 was preci­sion 0.11 at recall 0.12, and the best system had pre­cision 0.23 at recall 0.19. We did not run AuCon­traire on the RTE data sets because they contained relatively few of the "functional contradictions" that AuContraire tackles. On our Web-based data sets, we achieved a precision of 0.62 at recall 0.19, and precision 0.92 at recall 0.51 on the balanced data set. of course, comparisons across very different data sets are not meaningful, but merely serve to un­derscore the difficulty of the CD task.</p><p>In contrast to previous work, AuContraire is the first to do CD on data automatically extracted from the Web. This is a much harder problem than using an artificially balanced data set, as shown in Figure 4.</p><p>Automatic discovery of functional relations has been addressed in the database literature as <i>Func­tional Dependency Mining </i>(Huhtala et al., 1999; Yao and Hamilton, 2008). This focuses on dis­covering functional relationships between sets of at­tributes, and does not address the ambiguity inherent in natural language.</p></section><section number="7" title="Conclusions and Future Work"><p>We have described a case study of contradiction de­tection (CD) based on functional relations. In this context, we introduced and evaluated the AuCon­traire system and its novel EM-style algorithm for determining whether an arbitrary phrase is func­tional. We also created a unique "natural" data set of <i>seeming contradictions </i>based on sentences drawn from a Web corpus, which we make available to the research community.</p><p>We have drawn two key lessons from our case study. First, many seeming contradictions (approx­imately 99% in our experiments) are not genuine contradictions. Thus, the CD task may be much harder on natural data than on RTE data as sug­gested by Figure 4. Second, extensive background knowledge is necessary to tease apart seeming con­tradictions from genuine ones. We believe that these lessons are broadly applicable, but verification of this claim is a topic for future work.</p></section><section title="Acknowledgements"><p>This research was supported in part by NSF grants IIS-0535284 and IIS-0312988, ONR grant N00014-08-1-0431 as well as gifts from the Utilika Founda­tion and Google, and was carried out at the Univer­sity of Washington's Turing Center.</p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">19</doubt><page local="10" global="20"/></section><references><p>M. Banko and O. Etzioni. 2008. The tradeoffs between traditional and open relation extraction. In <i>Proceed­ings of ACL.</i></p><p>M. Banko, M. Cafarella, S. Soderland, M. Broadhead, and O. Etzioni. 2007. Open information extraction from the Web. In <i>Procs. ofIJCAI.</i></p><p>W.W. Cohen, P. Ravikumar, and S.E. Fienberg. 2003.</p><p>A comparison of string distance metrics for name-matching tasks. In <i>IIWeb.</i></p><p>Cleo Condoravdi, Dick Crouch, Valeria de Paiva, Rein­hard Stolle, and Daniel G. Bobrow. 2003. Entailment, intensionality and text understanding. In <i>Proceedings of the HLT-NAACL 2003 workshop on Text meaning, </i>pages 38—45, Morristown, NJ, USA. Association for Computational Linguistics.</p><p>I. Dagan, O. Glickman, and B. Magnini. 2005. The PASCAL Recognising Textual Entailment Challenge. <i>Proceedings ofthe PASCAL Challenges Workshop on Recognising Textual Entailment, </i>pages 1-8.</p><p>Marie-Catherine de Marneffe, Anna Rafferty, and Christopher D. Manning. 2008. Finding contradic­tions in text. In <i>ACL 2008.</i></p><p>A. P.Dempster, N.M. Laird, and D.B.Rubin. 1977. Max­imum likelihood from incomplete data via the EM al­gorithm. <i>Journal ofthe Royal Statistical Society Se­ries B, </i>39(1):1-38.</p><p>D. Downey, O. Etzioni, and S. Soderland. 2005. A Prob­abilistic Model of Redundancy in Information Extrac­tion. In <i>Procs. ofIJCAI.</i></p><p>Sanda Harabagiu, Andrew Hickl, and Finley Lacatusu. 2006. Negation, contrast and contradiction in text pro­cessing. In <i>AAAI.</i></p><p>Yka Huhtala, Juha Karkkainen, Pasi Porkka, and Hannu Toivonen. 1999. TANE: An efficient algorithm for discovering functional and approximate dependencies. <i>The Computer Journal, </i>42(2):100-111.</p><p>B. MacCartney and C.D. Manning. 2007. Natural Logic for Textual Inference. In <i>Workshop on Textual Entail-ment and Paraphrasing.</i></p><p>Ellen M. Voorhees. 2008. Contradictions and justifica­tions: Extensions to the textual entailment task. In <i>Proceedings ofACL-08: HLT, </i>pages 63-71, Colum­bus, Ohio, June. Association for Computational Lin­guistics.</p><p>Hong Yao and Howard J. Hamilton. 2008. Mining func­tional dependencies from data. <i>Data Min. Knowl. Dis-</i> <i>cov.</i><i>, </i>16(2):197-219.</p><p>A. Yates and O. Etzioni. 2007. Unsupervised resolution of objects and relations on the Web. In <i>Procs. ofHLT.</i></p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">20</doubt></references></body></article>