Realizing Expressions of Doubt in Collaborative Dialogue *
Leah Schroeder and Sandra Carberry
Computer and Information Sciences University of Delaware Newark, DE 19716 {sehroede, carberry}@cis.udel.edu
Participants in a collaborative natural language dialogue must develop mutual beliefs about what is said, what is meant, and the implications for the task at hand. We may think of each utterance as a proposed change to the agents' common ground (Clark, 1996). Since autonomous agents enter the dialogue with differing domain, world, and personal knowledge, it is inevitable that some beliefs conveyed by an utterance will not be accepted because they conflict with existing beliefs of the agent. However, it is also the case that these conflicting beliefs will not necessarily result in rejection of the proposed beliefs, but in subdialogues to negotiate a modification that is acceptable to both agents(Chu-Carroll and Carberry, 1995). One way to begin such a subdialogue is to express doubt at the beliefs proposed by an utterance. In the following example, the boldface utterance is expressing doubt at the previous utterance
H: and - there's no reason why you shouldn't have an ira for last yr J: well i thought they just started this yr
H: oh no. ira's were available as long as you are not a participant in an existing pension
An expression of doubt is an utterance that conveys uncertain disbelief in a proposition that was introduced in an earlier utterance. An expression of doubt signals that the speaker does not accept the utterance at which she is expressing doubt, but she is neither expressing a "neutral" attitude toward it nor rejecting it with certainty
A natural language system must be able to express doubt, particularly in cases where it has incomplete or uncertain knowledge. Examination of natural language corpora shows that expressions of doubt may be realized in a variety of forms. Furthermore, the form of the utterance conveys information about the nature of the doubt that is important for the subsequent resolution of the conflict. Thus a collaborative natural language system must be able to generate utterances that convey doubt naturally and effectively. This paper presents our work on realizing expressions of doubt appropriately in natural language dialogues.
In Chu-Carroll and Carberry (1998) the collaborative planning process is modeled Propose-Evaluate-Modify cycle, in which an agent is able to detect conflicts in belief and initiate collaborative negotiation subdialogues to attempt to resolve the conflicts. They use a modified version of Galliers belief revision mechanism(Galliers, 1992; Logan et al., 1994) to determine whether to accept a proposition and in determining which conflicting beliefs to use to refute an utterance that is not accepted. However, their work does not address how an expression of doubt should be realized in a natural language utterance.
Vander Linden and Di Eugenio (Vander Linden and Di Eugenio, 1996) studied negative imperatives in instructional texts.
We assume appropriate mechanisms for detecting conflict and determining when to engage in a sub-dialogue by expressing doubt(Chu-Carroll and Carberry, 1998), as well as an appropriate belief revision mechanism, and in this paper concentrate on how an expression of doubt should be realized as an utterance. A cooperative agent should be as informative as needed, without expressing too much irrelevant information(Grice, 1975). Thus, in formulating an expression of doubt, we must consider how much the doubted agent needs to know in order to collaborate in resolving the doubt and how much we can expect him to infer without being explicitly told. In addition, Clark (1996) argues that participants in discourse select utterances that express their communicative intent efficiently, often in sentence fragments. Since such efficiency of expression is the expected natural form of discourse, a hearer is likely to derive unintended implications from significantly-less economical realizations.
Expressions of doubt, by our definition, signal nonacceptance because of 'uncertain disbelief. In order for the doubted agent to attempt to collaborate in resolving the doubt, he needs to know several things. Most basically, he needs to recognize that there is doubt in a particular utterance. In the absence of an objection to an utterance, the speaker will assume an implicit acceptance (Lambert and Carberry, 1999). To efficiently negotiate an acceptable resolution to the belief conflict, ideally the doubted agent must know something about the beliefs of the doubting agent; in particular, which belief(s) are causing her nonacceptance, and the strength of these beliefs. If the doubted agent decides to retain his original beliefs, this information helps him to construct an argument that will be maximally effective and efficient in his attempt to convince the doubting agent(Chu-Carroll and Carberry, 1998).
To identify how expressions of doubt are realized in naturally occurring dialogue and how these realizations convey the requisite beliefs, we analyzed features of individual expressions of doubt extracted from natural corpora, and correlated the various forms of the utterances with the features of the underlying beliefs. However, as explained in Section 3.3, the use of machine learning techniques was not appropriate due to the nature of our corpus. Section 3.1 discusses features of underlying beliefs and Section 3.2 discusses the various forms that an expression of doubt can take. Section 3.3 then presents a set of rules that relate the two.
As noted above, beliefs play a prominent role in expressions of doubt, since a speaker will ideally convey enough information for the hearer to discern 1) that she is expressing doubt, 2) what she is doubting, 3) any support she has for the doubt, and 4) the strength of this support. In addition, speakers tend to differentiate new supporting information from that which is already part of the common ground and which should already have been considered. These beliefs are often not explicitly stated, but are assumed to be inferable by the doubted agent based on his knowledge of the previous dialogue, knowledge of the other agent's beliefs, a model of stereotypical beliefs, linguistic knowledge, and the particular realization of the doubting agent's utterance.
For example, consider the following assertion and two possible responses, each expressing doubt at the proposition Pdoubt that John Smith gets $800 a month from Social Security
S: John Smith gets $800 a month in Social Security.
In 1) U relies on mutual domain knowledge to express doubt at Pdoubt by contending some other proposition Pi that implies ^Pdoubt (Lambert and Carberry, 1999), namely that Smith is younger than 62. In the rest of this paper, Pdoubt refers to the doubted proposition and Pi to a proposition other than Pdoubt, if any, that is the reason for this doubt.
In addition, expectations also play a role in expressions of doubt. In the simplest case, the violated expectation is just that Pdoubt is false. In other situations, an agent may have an expectation that a proposition will be false if instantiated with some particular subset of its possible instantiations. Responses that conflict with these expectations may-provoke an expression of doubt, even though the doubting agent may have little or no support for the expectation. Such violated expectations are often signaled by elliptical fragments, such as response 2) above where U conveys not only that she doesn't accept Pdoubt, but also that her doubt stems from the instantiation of the amount term as $800.
We assume the propositions Pdoubt, Pe, and Pi, as well as the fact that we want to express doubt, as inputs to our generation process. Note that every expression of doubt will be associated with some Pdoubt and some Pe, since for every expression of doubt, there must be some doubted proposition and some inconsistency between the doubting agent's expectations and belief in Pdoubt- There may or may not be an associated Pi, depending on the doubting agent's beliefs supporting ^Pdoubt- Lack of a Pi indicates that the agent's belief in ^Pdoubt is unsupported
Based on the information that a speaker will ideally convey when expressing doubt (as discussed at the beginning of this section), we hypothesize that the following aspects of a speaker's beliefs are significant factors in how an expression of doubt is realized.
3.1.1 Features Associated with Pdoubt Endorsement of Pdoubt- Refers to the authority behind the asserted proposition, which impacts the strength of the hearer's doubt(Chu-Carroll and Carberry, 1998)
• Expert - The information is coming from a domain expert, or coming from someone with firsthand knowledge (including personal preferences).
• Reliable - The agent suggesting the proposition is not an expert, but is considered a generally knowledgeable source.
• Questionable - Information that doesn't come from an expert or reliable agent, or that is stated uncertainly by such an agent.
Pe feature: Pe refers to a violated expectation. In the following, we identify three kinds of expectations that may be violated by an assertion. For illustrative purposes, assume that S has made the following assertion:
S: The most you will get back on your taxes is $400.
• Term-value:
The doubting agent may fail to accept -Pdoubt with Jerm instantiated to -value, due to an expectation that -value is not one of the instantiations of Jerm that would make -Pdoubt true. For example, the hearer of the above assertion by S may have expected a much larger value than $400, with little or no support for this expectation.
• Constraint:
Pe = False(-Pdoubt, -term, -value, -constraint) The doubting agent may fail to accept -Pdoubt due to an expectation that -Pdoubt will be false when Jerm is instantiated with -value, in situations in which -constraint holds. This constraint is not a term in -Pdoubt, but the doubting agent believes that the speaker of -Pdoubt intends that the constraint hold. For example, the hearer of the above assertion by S may believe that S means $400 for the whole year, but may have expected a larger amount unless S was referring to, say, quarterly taxes.
• General:
Pe = False(-Pdoubt)
The doubting agent may fail to accept Pdoubt in its entirety without having a specific objection to any-particular term in the proposition.
3.1.3 Features Associated with Pi
• Old - A prior conflicting belief is already part of the explicit common ground of the dialogue.
• New - The doubting agent doesn't believe that her conflicting belief is already part of the common ground established by the preceding dialogue.
Endorsement of Pi refers to the strength of evidence supporting the belief Pi that is in conflict with the doubted belief. The endorsements are listed here from strongest to weakest.
• First-hand - Belief is a personal preference or something directly experienced.
• Expert - Belief supported by expert testimony, or thought to be common knowledge among experts in this domain.
• Reliable - Belief communicated from someone who, while not an expert, is generally considered a knowledgeable source of information.
• Default - Belief believed to be common knowledge, in the sense that the speaker strongly believes it and strongly believes that others who belong to a certain community (namely one which she has in common with the other dialogue agent) believe it as well.
• Derived - Belief is derived from other beliefs in such a way that it is considered strong.
• Hypothesis - The belief is derived from other beliefs in such a way that it is considered weak. This category includes beliefs derived from analogy with another belief in a similar proposition.
• None - The belief is unsupported.
Endorsement of Implication refers to the strength of evidence supporting the belief that Pi being true implies that Pdoubt is not true. The endorsements are listed from strongest to weakest. We assume the same definitions as the category above and that the two lists lie on the same strength scale. That is, an implication endorsed as reliable is the same strength Pi endorsed as reliable and stronger than a Pi endorsed as hypothesis. The only-addition to this list is the Logical endorsement to account for instances in which Pdoubt can be logically-deduced from Pi.
• Logical - ^Pdoubt directly inferred from Pi.
• First-hand
• Expert
• Reliable
• Default
• Derived
• Hypothesis
Expressions of doubt occur in a variety of forms. We distinguish them according to the surface form of the utterance, the presence of two clue words, and the specificity of the information conveyed.
Surface Form
• Surface Negative Question - "Isn't that only-worth what someone will pay for it?" This category also includes negative tag questions.
• Simple interrogative - "Can I join the IRA when I am 65?"
• Statement as Question - "I must file a return?" This category also includes elliptical fragments such as "$400?"
• Simple declaration - "I calculated 10."
• Proposition within a belief clause - "I thought they only started this year."
Clue word Specificity - General forms of the expression can be more or less specific in the amount of information conveyed.• Even though
• Generic: Sentence that is a general question of the previous utterance.
h. you still -you have to file a state income tax return as well
• Repetition: Repetition of a phrase from previous utterance.
h. OK, what I'd like you to do is lend him the 20 thousand.
• Repetition+ : Repetition of phrase from previous utterance plus new information
h. right, the maximum amount of credit that you will be able to get will be $400 on their tax return
• Contradict: Presentation of a belief that implies the negation of Pdoubt
h. and - there's no reason why you shouldn't have an ira for last yr j. well i thought they just started this yr
• Contradict+Source: Presentation of a contradictory belief and the source of that belief.
j. mm. harry another thing, i have the internal uh revenue uh ask you about that 6256 $ uh since i have the fund he said no! so what do i do now?
• Explicit+Contradict: Explicit statement of disbelief, followed by a contradictory belief.
b. well ah he uh ... he belongs to a money market fund now and uh they will do that [invest it in govt securities as part of their individual retirement account] for him
h. i'm not so sure of it.. they may move it ah into a into a govt securities fund, but i'm not so sure that they can move it into individual securities — check that out
Many of the expressions of doubt in our corpus are non-ideal, because they were not recognized as doubt or because information that was not included in the utterance, but could have been, was ultimately-needed to resolve the doubt. Thus it was not appropriate to use the corpus as training data for machine learning. Consequently, the following rules are based on our examination of naturally occurring dialogues, as well as our judgments of which of these naturally-occurring forms was more or less successful in the dialogue.
The majority of expressions of doubt occurring in the corpus were of the general form of one agent expressing a belief Pi that implies a contradiction to the proposition being doubted (though a clue word may be necessary - see rules CW1 and CW2 on the next page). These contradictions all require that the doubted agent be able to recognize the implied negation of Pdoubt from the proposition in the doubting agent's utterance.
I'll: Specificity = Contradict, Surface Form = surface negative question with Pi if Endorsement of Implication at least as strong as Endorsement of Pi and Endorsement of Pi not greater than default
Surface negative questions convey uncertain belief in a proposition and a belief that the doubted agent will be able to infer the implication. This form is therefore appropriate when Pi is considered more questionable than the implication or if Pi and the implication are endorsed equally strongly, in which case it is reasonable to address Pi before the implication.
PI2: Specificity = Contradict, Surface Form = Pi embedded in a belief clause if Endorsement of Pi = Reliable or Expert and Endorsement of Implication stronger than default and Endorsement of Pi stronger than Endorsement Of Pdoubt (except those instances covered by rule PI6 below) Propositions embedded in belief clauses appear to convey stronger beliefs than those expressed as a surface negative question. In an informal survey, graduate students were given dialogues which ended with several alternative forms of expressions of doubt. They were asked to rate the strength of belief in the underlying proposition conveyed by each form. For the majority of instances, expressions of the form Pi embedded in a belief clause were judged to convey-stronger beliefs than those in surface negative form.
PI3: Specificity = Contradict, Surface Form = simple declaration of Pi if Endorsement of Pi = first-hand and Endorsement of Implication stronger than Default Doubt expressed by asserting the truth of a proposition reflects beliefs that are more strongly held than those embedded in a belief clause. The doubted agent is not "invited" to address the doubt by providing evidence against the held belief, but to instead weaken belief in the implication if that is possible (or concede that the other agent is correct).
PI4: Specificity = Contradict, Surface Form = simple interrogative about whether Pdoubt is possible if Pi if Endorsement of Implication weaker than Endorsement of Pi and no stronger than default and Endorsement of Pi is stronger than None If the doubting agent is uncertain about Pi, but has more uncertainty about whether Pi —¥ ^Pdoubt than about whether Pt is true, then she should call attention to the implication so that the doubted agent knows that refuting the implication may be the easiest way to resolve the doubt. In the corpus agents often use a question (like "Can I join the ira when I'm 65?") to emphasize the implication, while the other forms emphasize Pi.
PI5: Specificity = Contradict+Source, Surface Form = simple declaration of Pi if Endorsement of Pi = Reliable or Expert and Endorsement of Implication stronger than default and Endorsement of Pdoubt = Endorsement of Fi
This situation corresponds to the case where two conflicting beliefs are very strong and considered equally plausible. Since many belief revision systems would prefer to keep the old belief, revising the least amount possible (Logan et al., 1994), the existing beliefs will be relatively resistant to revision, and will only be displaced by an inconsistent belief of higher endorsement. That is, the doubted agent can persuade the doubting agent by convincing her that he is more expert than the old source. Therefore, this extra source information is important for resolving the doubt.
PI6: Specificity = Explicit + Contradict, Surface Form = simple declaration of disbelief followed by simple declaration of Pi if Endorsement of /', = expert and Endorsement of implication = logical or expert and Endorsement of Pdoubt = questionable The Explicit+Contradict method of expressing doubt conveys a strong resistance to changing the previously held belief, but unlike a certain rejection, invites the doubted agent to present more evidence in his next turn. This form is therefore appropriate when an agent feels support for her belief is greatly-superior to evidence for an opposing belief, but is willing to negotiate further.
If Pi exists, then clue words are often used in expressions of doubt. Rules CW1 and CW2 add a clue word.
CW1: Clue word = Even though if Endorsement of Implication weaker than Endorsement of Pi and Endorsement of Pi is stronger than Hypothesis and Commonality = Old "Even though" directly signals a potential contradiction and implies that the agent has some strong belief in a fact that may or may not conflict with the doubted utterance and which is already a part of the common ground of the participants.
CW2: Clue word = But if Endorsement of implication = first-hand, hypothesis, or none and Commonality = New "But" signals an objection. Since the doubted agent has to infer that Pi implies the negation of the doubted proposition, the doubting agent should give a clue that she is expressing doubt when she expects the implication to be non-obvious. That is, if the implication relies on information that is known only to her or information which she has little evidence for (and therefore little cause to believe that the other agent believes the same), she should use a clue word.
Rules PE1 - PE3 are applicable when there is no Pi.
PE1: Specificity = generic, Surface Form = statement as question if Pe feature = general
This represents the situation in which the system has a weak belief in ^Pdoubt that is unsupported by-other evidence. The simple questioning of Pdoubt usually has the effect of prompting the other agent for more information or a reassertion of the proposition's truth.
PE2: Specificity = Repetition (of .value), Surface Form = statement as question if Pe feature = term-value
When doubt is expressed simple repetition, the agent is calling attention to a specific portion of the previous utterance. By questioning the value of a term in .Pdoubt, especially in situations where there is no reason to believe it has been misunderstood, the agent is calling attention to this instantiation as the reason for the doubt.
PE3: Specificity = Repetition+ (repetition of .value plus -constraint), Surface Form = statement as question if Pe feature = constraint
Partial repetitions containing extra information are much the same as repetitions described above, except that if the doubted agent has a more specific reason for not accepting Pdoubt, she should express it so that the doubted agent may address it. The new information in the utterance gives the constraint under which the agent doubts that the term is instantiated appropriately. So a response of "$400 for the whole year?" to the assertion by S at the beginning of section 3.1.2 conveys the implication that $400 might be a reasonable amount for part of the year and thus that the agent expected the amount to be larger than 400 for the entire year.
Our corpus consists of spontaneous utterances that are imperfect and thus cannot be considered a "gold standard" by which to judge our realizations of expressions of doubt. Therefore, we used human subjects to perform a preliminary evaluation of our generation methodology.
In section 3 we contend that a speaker must form her utterance so that the hearer can determine: that she is expressing doubt, what she is doubting, the strength of her doubt, and the support for this doubt. In order to determine how well the forms we generate accomplish this, we had six subjects judge dialogue segments which ended with a response that was, in some cases, an expression of doubt. The expressions of doubt in the questionnaire corresponded to the forms that our rules would generate.
The subjects were asked to read the dialogue segments, and to answer questions about a particular highlighted utterance. The subjects were first asked whether the utterance was expressing doubt. If so they were asked to specify at which utterance, and to rate the strength of the doubt. Each dialogue was also followed by a list of propositions which included those we judged to be the relevant /' . /',. and implication beliefs that would have caused the generation and seemed appropriate from the dialogue. The subjects were asked to choose and rate the strength of those that were communicated by the doubting agent in order to see if these correlated with the endorsements from which they would have been generated.
There were ten responses which were intended to be expressions of doubt; all were correctly judged as such by all subjects. There were four responses given to the subjects which were not expressions of doubt, of which all except one was judged correctly by all subjects
We had asked the subjects to rate the strength of the doubt. Subjects almost always rated instances in which doubt was expressed directly at Pdoubt as conveying weaker doubt than forms expressing doubt by-contending Pi. Although strength of doubt doesn't correspond directly to one of our belief features, this is consistent with the idea that supporting evidence may not be attributed if it is not expressed.
We next asked the subjects to pick from a list of propositions those which contributed to the doubt in order to see if they attributed to the agent the /' . /',.
and implication beliefs that would have caused the form of expression of doubt to be generated. We also encouraged subjects to write in beliefs which were not included, but none did. Out of the 60 instances (ten expressions of doubt times six subjects), the subjects five times chose beliefs that we did not represent as contributing to the doubt and three times failed to recognize a belief that did contribute.
The subjects also rated the beliefs according to strength. We evaluated these ratings to see if the communicated strengths were correlated with the endorsements of beliefs that would have generated this form. Since subjects varied in the ranges that they used in rating the strengths of the beliefs, we looked at the scores relative to each subjects ratings of the other beliefs.
Most of the strength ratings were consistent with the rules. The most frequent inconsistency was the case in which we would have generated a form based on slightly different endorsements for Pt and the implication, but our subjects rated them equivalent strengths. While it may be the case that people don't actually perceive a difference, it may also be the case that numerical ratings don't fully capture the same information that the notion of endorsements do.
The only significant inconsistency with our rules was one utterance in which doubt was expressed by "I thought that, but my husband, he wasn't sure if that just meant ss pension." We had represented the husband reliable source and thus generated information about the source of the conflicting proposition. In this instance, the doubt was not judged very strong by our subjects, and the agent's belief in her husband's expertise as relatively weak. In future work, we will further explore expressions of doubt for which it is important to communicate the source of information.
We consider this a preliminary evaluation to show that the rules we have formulated thus far are reasonable. Further evaluation will be needed to provide evidence that subjects really do draw different inferences based on the different forms of expression and that our rules accurately capture these differences.
This paper has presented rules that could be used by a natural language system to realize expressions of doubt. We have identified several forms that are used to express doubt in naturally occurring dialogues. Our rules correlate these forms to beliefs of the doubting agent, taking into consideration the beliefs that must be conveyed for the utterance to be a successful expression of doubt.
Preliminary evaluation shows that the belief feature values in our rules correspond to human subjects' intuitions about the strength of the doubt. In addition, the beliefs that would generate each form are consistent with the beliefs that the subjects attribute to the doubting agent when that form was used.
Future work will concentrate on refining the features and exploring more explicit reasoning about the beliefs of the other agent. We also plan to explore the role of intonation in realizing expressions of doubt.
Jennifer Chu-Carroll and Sandra Carberry. 1995. Communication for conflict resolution in multi-agent collaborative planning. In Proceedings of the International Conference on Multi-Agent Systems, pages 49-56.
Jennifer Chu-Carroll and Sandra Carberry. 1998. Collaborative response generation in planning dialogues. Computational Linguistics, 24(3):355-400.
Herbert Clark. 1996. Using Language. Cambridge University Press.
Julia Rose Galliers. 1992. Autonomous belief revision and communication. In P. Gardenfors, editor, Belief Revision, Cambridge tracts in theoretical computer science. Cambridge University Press, Cambridge, England.
H. Paul Grice. 1975. Logic and Conversation. In P. Cole and J. L. Morgan, editors, Syntax and Semantics III: Speech Acts, pages 41-58, N.Y. Academic Press.
Lynn Lambert and Sandra Carberry. 1999. A process model for recognizing communicative acts and modeling negotiation subdialogues. Computational Linguistics, 25:1-54.
Brian Logan, Steven Reece, Allison Cawsey, Julia Galliers, and Karen Sparck Jones. 1994. Belief revision and dialogue management in information retrieval. Technical report, University of Cambridge Computer Laboratory.
Harry Gross Transcripts. 1982. Transcripts derived from tapes of the radio talk show harry gross: Speaking of your money. Provided by the Dept. of Computer Science at the University of Pennsylvania.
Keith Vander Linden and Barbara Di Eugenio. 1996. A corpus study of negative imperatives in natural language instructions. In Proceedings of the 16th. International Conference on Computational Lin-gusitics (COLING-96), Copenhagen.