<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="233"/><title>Combining Confidence Scores with Contextual Features for Robust Multi-Device Dialogue</title><pubinfo>Proceedings of the Australasian Language Technology Workshop 2005,pages 233-240, Sydney, Australia, December 2005.</pubinfo><author surname="Cavedon" givenname="Lawrence"><org  name="Stanford University" country="USA" city="Stanford"/></author><author surname="Purver" givenname="Matthew"><org  name="Stanford University" country="USA" city="Stanford"/></author><author surname="Ratiu" givenname="Florin"><org  name="Stanford University" country="USA" city="Stanford"/></author></firstpageheader><frontmatter><p><b>Combining Confidence Scores with Contextual Features for Robust Multi-Device Dialogue*</b></p><p><b>Lawrence Cavedon Matthew Purver, Florin Ratiu</b></p><p>National ICT Australia, Victoria Research Lab    CSLI, Stanford University</p><p>and CS&amp;IT, RMIT University      Cordura Hall, 210 Panama St. Stanford Melbourne VIC, Australia CA 94305, USA</p><p>lawrence.cavedon@nicta.com.au       {mpurver,fratiu}@stanford.edu</p></frontmatter><abstract>We present an approach to multi-device dia­logue that evaluates and selects amongst candi­date dialogue moves based on features at mul­tiple levels. Multiple sources of information can be combined, multiple speech recognition and parsing hypotheses tested, and multiple de­vices and moves considered to choose the high­est scoring hypothesis overall. The approach has the added benefit of potentially re-ordering n-best lists of inputs, effectively correcting er­rors in speech recognition or parsing. A current application includes conversational interaction with a collection of in-car devices. </abstract></header><body><section number="1" title="Introduction"><p>In this paper, we describe recent enhancements to the <i>CSLI Dialogue Manager </i>(CDM) infras­tructure to increase robustness, in particular in (but not exclusive to) multi-device settings. Dialogue contributions may be processed using multiple information sources (e.g. deep syn­tactic parsing and shallow topic classification), scored at multiple levels (e.g. acoustic, seman­tic and context-based), and bid for by multiple agents, with the overall highest-confidence bid chosen.</p><p>The CDM provides a multi-device infrastruc­ture, with customization to new applications and addition of plug-and-play devices eased by a declarative dialogue-move scripting language (Mirkovic and Cavedon, 2005). However, decid­ing which device an utterance is directed at is not always straightforward. One of our current application areas is a conversational interface to in-car devices, including entertainment, restau­rant recommendation, navigation and telematic systems (Weng et al., 2004); in such an envi­ronment, a request such as <i>"Play X" </i>might be</p><p>* This work was performed while all the authors were employed at CSLI, Stanford University, and was par­tially supported by the US government's NIST Advanced Technology Program.</p><p>directed at an MP3 player or a DVD player. Eye-gaze (useful in multi-human dialogue) is not available, and we cannot rely on explicit device naming. One option is to use the reso­lution of NP arguments as disambiguating in­formation (in our <i>"Play X" </i>example, whether X is a song or a movie). However, the NP-resolution process itself is often device-specific (see below), preventing NPs from being prop­erly resolved until device has been determined.</p><p>Our proposed solution, inspired by ap­proaches to multi-agent task allocation such as Contract Net (Smith, 1980), is to allow all devices to perform shallow processing of the incoming utterance, each producing multi­ple possible candidate dialogue moves. Poten­tial device-move combinations are then scored against a number of features, including speech-recognition and parse confidence, discourse con­text, current <i>device-under-discussion</i>, and NP argument analysis. The device associated with the highest-scoring dialogue move is given first option to process the utterance. A disambigua­tion question may be generated if no device is a clear winner, or a confirmation question if the winning bid is not scored high enough.</p><p>Device choice, move choice, and selection of best ASR/parser hypothesis are thereby made simultaneously, rather than being treated as in­dependent processes. As well as allowing for principled device identification, this has the benefit of scoring hypotheses on the basis of multiple information sources, including context. The highest scoring result overall may not cor­respond to the highest-confidence result from the ASR or parser n-best list alone, but n-best lists are effectively re-ordered based on device and dialogue context, allowing parsing errors such as incorrect PP-attachment to be automat­ically corrected. Confirmation and clarification behaviour can also be governed not only by ASR or parse confidence, but by the overall score.</p><page local="2" global="234"/><p><b>Related Approaches </b>Rayner et al. (1994) combine speech recognition confidence scores with various intra-utterance linguistic features to re-order n-best hypotheses; Chotimongkol and Rudnicky (2001) also include move bigram statistics. Walker et al. (2000) use similar fea­ture combination to identify misrecognised ut­terances. More recently, Gabsdil and Lemon (2004) also include pragmatic information such as NP resolution, and simultaneously choose from an n-best list while identifying misrecogni-tion. They also divide misrecognised utterances into two overall confidence ranges, one for out­right rejection and one for confirmation/clarifi­cation. Similarly Gabsdil and Bos (2003) com­bine acoustic confidences with semantic infor­mation, and Schlangen (2004) with bridging ref­erence resolution, in order to allow clarification on an integrated basis. All of these approaches assume a single-device setting and hence no ambiguity of move type once the correct word string or parse has been identified. Here we extend these approaches to allow a principled choice of move/device pairing.</p></section><section number="2" title="Background"><subsection number="2.1" title="Dialogue Manager Architecture"><p>Our focus is on <i>activity-oriented dialogue, </i>dis­cussing tasks or activities that are jointly per­formed by a human and one or more intelligent devices or agents. By "joint activity", we mean that the human participates in specifying the activity, clarifying requests, interpreting obser­vations, and otherwise supporting the agent in the performance of the activity. Systems en­gaging in such dialogue characteristically re­quire deep knowledge about the task domain and the devices/agents they provide access to, in order to know what information is critical to the tasks, and know what information about task performance is appropriate to provide to the user. CSLI has been developing activity-oriented dialogue systems for a number of years, for applications such as multimodal control of robotic devices (Lemon et al., 2002), speech-enabled tutoring systems (Fry et al., 2001), and conversational interaction with in-car devices (Weng et al., 2004).</p><p>The dialogue system architecture (Figure 1) centers around the CSLI Dialogue Manager, which can be used with various different exter­nal components: speech-recognizer, NL parser, NL generation, speech-synthesizer, as well as connections to external application-specific components such as ontologies or knowledge bases, and the dialogue-enabled devices them­selves. Clean interfaces and representation-neutral processes enable the CDM to be used relatively seamlessly with different NL compo­nents, while interaction with external devices is mediated by <i>Activity Models, </i>declarative spec­ifications of device capabilities and their rela­tionships to linguistic processes.</p><p>The CDM uses the <i>information-state update </i>(ISU) approach to dialogue management (Lars-son and Traum, 2000). The ISU approach ex­tends the more traditional finite-state-based ap­proaches used for simple dialogues (in which di­alogue context is represented as one of a finite number of states, and each dialogue move re­sults in a state transition), maintaining a richer representation of information-state. This in­cludes the dialogue context as well as e.g. de­vice and activity status, together with a set of update rules defining the effect of dialogue moves on the state (e.g. adding new infor­mation and referents for anaphora resolution, and triggering new tasks, activities and system responses). This approach allows more com­plex dialogue types with advanced strategies for context-dependent utterance interpretation (in­cluding fragments and revisions), NP resolution, issue tracking and improved speech-recognizer performance (Lemon and Gruenstein, 2004).</p></subsection><subsection number="2.2" title="The CSLI Dialogue Manager"><p>Generic ISU toolkits (e.g. TrindiKit (Traum et al., 1999), DIPPER (Bos et al., 2003)) pro­vide general data structures for representing state and a language for specifying update rules, but the specific state and rules used are left to the individual application. The CDM is a specific implementation of an ISU dialogue-management system, providing data structures and processes for update specifically designed as suitable to activity-oriented dialogue, but adaptible to different applications and domains.</p><p>The two central components of the CDM in­formation state are the <i>Dialogue Move Tree</i> (DMT) and the <i>Activity Tree.</i><i> </i>The DMT rep­resents the dialogue context and history, with each dialogue move represented as a node in the tree, and incoming moves interpreted in context by attachment to an appropriate open parent node (for example, <b>WhAnswer </b>moves at­tach to their corresponding <b>WhQuestion </b>nodes). This tree structure specifically supports <i>multi­threaded, multi-topic </i>conversations (Lemon et al., 2002), with branches representing topics or threads:<page local="3" global="235"/> a dialogue move that cannot attach itself to the most recent active node may in­stead attach to another open branch (corre­sponding to a resumed conversation) or open a new branch (a new conversation thread) by attaching itself to the root node. The DMT also serves as context for interpreting fragments, multi-utterance constructs, and revisions, and provides discourse structure for tasks such as NP-resolution. In tandem, the Activity Tree manages the underlying activities, fully instan­tiating new activities via their Activity Models (e.g. resolving NP referents or spawning sub­dialogues to fill missing arguments), editing ex­isting ones as a result of revisions or corrections, and monitoring their execution (possibly gen­erating system moves notifying completion or failure).</p><figure caption="Figure 1: Dialogue System Architecture"></figure><p>Other data structures that are part of the in­formation state include: the <i>salience list </i>(NPs and their referents for anaphora resolution); <i>multimodal input buffers </i>(semantic interpreta­tions of GUI events); and the <i>system agenda </i>(potential system outputs scheduled by the di­alogue manager). See (Lemon et al., 2002) for details.</p></subsection><subsection number="2.3" title="Dialogue Move Scripting"><p>In early versions of the CDM, dialogue moves were coded completely programmatically (in Java). While libraries of general-purpose di­alogue moves (e.g.     <b>Command, WhQuestion,</b> etc.) were re-used wherever possible, cus­tomization to new domains generally required significant programming effort in defining both new dialogue moves and their effects, and pro­cesses such as reference resolution. More re­cently, Mirkovic and Cavedon (2005) describe a dialogue-move scripting language designed to expedite customization to new domains. Each script serves a number of purposes:</p><p>1. hierarchical definition of dialogue moves, allowing inheritance and re-use of existing moves, while allowing customization to a specific domain;</p><p>2. mapping of utterance representations to appropriate dialogue moves, including ar­gument values for devices' activity models;</p><p>3. definition of attachment rules for information-state update;</p><p>4. dialogue move-specific specification of out­put to be generated, for disambiguation or requests for required information.</p><p>Listing 1 shows the skeleton of a sample dialogue-move script for a <b>play Command </b>move for an MP3 player. The specific syntax of the <i>Input </i>and <i>Output </i>fields can be ignored for now: they simply match the interfaces of the parser and generator respectively. Variables in the di­alogue move script correspond to variables in the Activity Model (AM) for the corresponding device. The AM for the MP3 device contains<page local="4" global="236"/></p><p><b>User  Command:play  {     </b><b>//</b><b> </b><i>inherits  from  generic   Command   dialogue move </i><b>Description   "play something"</b> <b>Input  { </b><i>//</i><i>  templates  for  matching parser output</i> <i>//</i><i>  topi c   cl assi fi er  match  wi th argument</i> <b>Producing  { </b><i>//</i><i>  templates  for  system   output:</i><i> questions</i> <b>System  WHQuestion:</b><b> disambiguate</b> <b>System  WHQuestion :</b><b>fill : play :_playable - object {</b> <b>} </b><i>//</i><i>  templates  for  system   output:</i><i> reports</i> <b>CloseOn  System  Report:</b><b> play: playing {</b></p><doubt alpha="60.0" length="40" tooSmall="False" monospace="0.0">//full  parse  match:   ''play/start X''</doubt><doubt alpha="59.3" length="81" tooSmall="False" monospace="0.0">1.0       SYN{  s(   features(mood(imperative)),   predicate(#play/vb|#start/vb),</doubt><doubt alpha="60.6" length="94" tooSmall="False" monospace="0.0">?arglist(obj :_playable-object ,?sbj :*)) }//full  parse  match:   ''I want   to play/hear X''</doubt><doubt alpha="57.7" length="78" tooSmall="False" monospace="0.0">1.0       SYN{  s(   features(mood(indirect)),   predicate(#play/vb|#hear/vb),</doubt><doubt alpha="58.8" length="102" tooSmall="False" monospace="0.0">?arglist(obj :_playable-object ,?sbj :*)) }//topi c   cl assi fi er match0.1       TOPIC{  play_item }</doubt><doubt alpha="46.7" length="92" tooSmall="False" monospace="0.0">0.25    AND {  TOPIC{  play_item  },       SYN {  arglist(obj :_playable-object , *)   } } }</doubt><doubt alpha="46.5" length="71" tooSmall="False" monospace="0.0">Output  {avs   (e1  /  play   : question   (q1  /  what)   : agent I) }</doubt><doubt alpha="51.9" length="81" tooSmall="False" monospace="0.0">Output  {avs   ( e1  /  play   : patient   ( p1  / [song])   : aspect continuous)</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">}</doubt><p>Listing 1: Sample dialogue move script for a <b>play Command </b>for an MP3 device a <b>play </b>operation with a (required) <i>playable-object </i>argument. When an incoming utterance matches an <i>Input </i>template from Listing 1, the <i>playable-object </i>variable is filled by unification, and resolved to an object from the device's do­main which then fills the corresponding slot in the activity. For details, see (Mirkovic and Cavedon, 2005).</p></subsection><subsection number="2.4" title="Multi-Device Dialogue"><p>The CDM has also been extended to multi­device dialogue, with the scripting approach allowing easy dynamic plug-and-play specifica­tion of new "dialogue-enabled" devices. Note that this does not constitute multi-party dia­logue: interaction is still mediated by a single dialogue manager, between a user and a <i>De­vice Manager </i>with which devices register them­selves. However, the plug-and-play requirement (necessitated by the in-car application (Weng et al., 2004)) has resulted in important extensions to the dialogue management infrastructure.</p><p>Mirkovic and Cavedon (2005) describe a framework for encapsulating devices with in­formation required to "dialogue-enable" them. Each device has associated with it the following components:</p></subsection></section><section number="1." title="a set of dialogue-move scripts;"><p>2. an Activity Model describing any device functionality accessible by dialogue;</p></section><section number="3." title="a device-specific ontology and knowledge base (KB);"><p>4. rules for device-specific NP-resolution.</p><p>Any significantly different forms of interac­tion requiring device-specific dialogue manage­ment processes must still be specified as new Java classes (referred to as <i>DM process exten­sions </i>in Figure 1), but in general the above four components contain the device-specific informa­tion required for dialogue-enabling new devices.</p><p>Note that NP resolution rules are included in the device definition; while pronoun resolu­tion tends to be domain-independent, resolving definite descriptions and demonstratives is of­ten device-dependent, and resolving named ref­erents often requires constructing appropriate queries to a device-specific knowledge-base.</p><p>Devices can now be added dynamically to the DMT, registering themselves with the Device Manager and becoming associated with their own nodes to which new conversation threads can attach; "current device" becomes part of the information-state and interpreting incoming utterances is performed in this context.</p><p>In this context, device selection—determining which device an utterance is associated with— becomes a further complication: an utterance may (on the surface) be potentially applicable to multiple devices: e.g. "Play X" could be applicable to either an MP3 player or a DVD player.<page local="5" global="237"/> Our original proposal was to create a dialogue move consistent with each such device and then score its applicability based on other factors, e.g. ability to resolve the object ref­erence (the MP3 player would resolve a song­name, the DVD player a movie name). The rest of the paper generalises this approach to a wider range of possible disambiguities, involv­ing a greater number of scoring features, and re­sults in more interesting behaviours than simple device-disambiguation.</p></section><section number="3" title="Multiple Interpretation Methods"><p>The first new extension to the CDM described here is the use of multiple information sources in parallel to classify dialogue move type and produce an activity-specific representation. In most systems (and previous incarnations of the CDM) a single interpretation mechanism is cho­sen which is best suited to the application at hand, be it e.g. an open-domain statisti­cal parser, a domain-specific constraint-based grammar, or keyword-spotting techniques. We extend this approach here to allow arbitrary multiple interpretation mechanisms, each pro­ducing its own (independent) interpretation hy­pothesis and associated confidence. In the cur­rent application, we use both a statistical parser producing relatively deep dependency struc­tures, and a shallow maximum-entropy-based topic classifier.</p><p>Dialogue move scripts, such as the one sketched in Listing 1, are used to construct in­stantiations of candidate dialogue moves for a device, based on incoming user utterances (and planned system outputs, although we focus on the former here). This is governed by the <i>Input </i>field for each move type, which specifies a set of patterns: when an utterance representation matches an <i>Input </i>pattern, a candidate node of the appropriate type can be created. As List­ing 1 shows, patterns can now be defined in terms of interpretation method as well as the interpreted form itself: <b>SYN </b>patterns match the output of the statistical parser, <b>TOPIC </b>patterns match the output of the topic classifier, while <b>AND </b>patterns match combinations of the two. Further general pattern types are available (e.g. <b>LF </b>for semantic logical forms, <b>STRING </b>for sur­face string keyword-matching) but are not used in the current application.</p><p>Each pattern is associated with a weight, used in the overall move scoring function described in Section 4 below.   This allows moves created from matches against deep structure to be scored highly (e.g. <b>SYN </b>patterns in which pred­icate and arguments are specified and matched against), shallow matches to be scored low (e.g. simple <b>TOPIC </b>matches), and combined matches to have intermediate scores (e.g. a combina­tion of an appropriate <b>TOPIC </b>classification with a <b>SYN </b>parser output containing a suitable NP ar­gument pattern). Depending on other elements of the scoring function (e.g. the ASR confi­dence associated with the hypothesised string being tested) and on competing move hypothe­ses, low scores may lead to clarification being re­quired (and therefore clarification will be more likely when only low-scoring (shallow) patterns are matched). Behaviour can therefore be made more robust: when deep parsing fails, a shallow hypothesis can be used instead (clarifying/con­firming this specific hypothesis as necessary de­pending on its confidence) rather than resorting to a rejection or general clarification. Scores are currently set manually and determined by test­ing on sample dialogues; future work will exam­ine learning them from data.</p></section><section number="4" title="Dialogue Move Selection"><p>In the general case, multiple possible candidate dialogue moves will be produced for a given user utterance, for a number of reasons:</p></section><section number="1." title="multiple hypotheses from ASR/parser out­put;"><p>2. multiple interpretation methods (deep parsing vs. shallow classification);</p><p>3. multiple possible move types for a candi­date interpretation;</p><p>4. multiple antecedent nodes (active dialogue threads), including multiple devices, for a particular move type.</p><p>These are not independent: it is important to consider all factors simultaneously, to allow an integrated scoring function for each candidate and thus consider the best overall. The skele­ton algorithm for instantiating and selecting a dialogue move is therefore as follows:<footnote anchor="1"/><page local="6" global="238"/></p><footnote label="1">Note that we will not create O x N x M candidates: only a subset of script entries (if any) will match for each node and n-best entry.</footnote><p><b>foreach  open node O</b> <b>foreach n-best   list   entry N</b></p><p><b>foreach  matching   script   entry M create   candidate move score   all candidates if   (score(top)   </b><b>&gt;&gt;</b><b>  s core(second)) then</b></p><p><b>select  top candidate else</b></p><p><b>generate  question  to disambiguate if   ( score( selected-node)   &lt; threshold) generate  question  to confirm</b></p><p>The interesting aspect of the above process is the scoring function. Dialogue-move candi­dates are scored using a number of weighted features, ranging from speech-recognizer confi­dence, through to pragmatic features such as the "device in focus" and recency of the DMT node the candidate would attach to. The full list of features currently considered is shown in Table 1. Note the inclusion of features at many levels, from acoustic recognition confi­dences through syntactic parse confidence to se­mantic and pragmatic features.</p><subsection number="4.1" title="Reordering n-best candidates"><p>This integrated scoring mechanism therefore al­lows n-best list input to be re-ordered: dialogue-move candidates are potentially instantiated for each n-best list entry and the highest-scoring candidate chosen. While the n-best list rank and confidence are factors in the overall score, other features may outweigh them, resulting in an initially lower-ranked n-best entry becoming the highest-scoring dialogue move.</p><p>Evaluation so far has been limited to ini­tial testing on a manually constructed set of test inputs, using only a subset of the features: those shown italicised in Table 1 are not cur­rently available due to either implementational issues (for full domain referent resolution and KB queries) or lack of domain data (for move bigram frequencies). Our test set includes 400 sentences, of which 300 have been used in train­ing the statistical parser and 100 are unseen variations; it currently covers only utterances related to a single device (a restaurant recom­mendation system) and does not include speech recognition hypotheses (we are therefore testing parse n-best reordering only). We are currently working towards evaluation on a full set of fea­tures, with user-generated multi-device speech input.</p><p>However, even with the restricted set of fea­tures, preliminary testing on this set shows encouraging results: the percentage of sen­tences for which the correct parse is chosen increases from 90% to 94%, a 41% reduction in error with several common parse errors be­ing corrected. One example is incorrect PP-attachment (a notoriously difficult challenge for statistical parsers). The example below (from a restaurant recommendation scenario), shows the top two n-best list entries for a sentence as produced by our statistical parser:</p></subsection></section><section number="1." title="how  about   [a restaurant"><doubt alpha="60.7" length="28" tooSmall="False" monospace="0.0">[in  Grant]]   [on Mayfield]</doubt></section><section number="2." title="how  about   [a restaurant"><doubt alpha="58.6" length="29" tooSmall="False" monospace="0.0">[ in Grant]   [ on Mayfield]]</doubt><p>Here, the second is lower-ranked but correct, taking both PPs as modifying <i>restaurant, </i>while the first treats only one as modifying <i>restau­rant, </i>one as a sentential modifier. As the sec­ond allows two database-query constraints to be filled (city and street name), and the first just one, this boosts its overall score enough to overcome its lower parse confidence, and it is selected and used in DMT attachment. Similar improvements are gained with nominal modi­fiers:</p><doubt alpha="45.0" length="20" tooSmall="False" monospace="0.0">1 . how  about   [ a</doubt><p><b>[[cheap]   chinese] restaurant]</b></p><doubt alpha="47.4" length="19" tooSmall="False" monospace="0.0">2. how  about   [ a</doubt><doubt alpha="64.7" length="34" tooSmall="False" monospace="0.0">[ cheap]   [ chinese ] restaurant]</doubt></section><section number="3." title="how  about [a"><p><b>[ cheap   chinese ] restaurant]</b></p><p>Here the second is correct, treating <i>cheap </i>and <i>chinese </i>as both independently modifying <i>restaurant </i>; the first takes <i>cheap </i>as modifying <i>chinese, </i>and the third takes <i>cheap chinese </i>as a single multi-word unit. Again, as the second fills two database-query constraints (price level and cuisine type), its overall score becomes highest. Evaluation of the improvement achieved is cur­rently in progress.</p><subsection number="4.2" title="Move type comparison"><p>The scoring function for feature combination is currently manually defined. When comparing between candidate moves of the same type, this is relatively straightforward, although hardly trivial and inherently done to a high extent by subjective expertise. However, it becomes much less straightforward when comparing can­didates of different types, as some move types and some DMT attachment contexts will allow only a subset of the features to have meaning­ful values. However, comparison between move types is essential, as two ASR hypotheses with<page local="7" global="239"/></p><p>Recognition features:   recognition and parse probabilities;</p><p>recognition and parse n-best ranks; Semantic features:       topic classification for the parse (with score); for dialogue moves spawning activities:</p><p>- number of slots filled by input pattern; - <i>number of resolved/unresolved slots after NP resolution;</i></p><p>- <i>number of ambiguously resolved slots after NP resolution; </i>for queries about database objects:</p><p>- set of constraints sent to the knowledge base;</p><p>- <i>cardinality of the set of knowledge base query results; </i>Contextual features:    current most active node; current activity; position and recency of the parent node in the active node list; <i>bi-gram frequencies of the dialogue moves:</i><i></i></p><p>- <i>DMT attachments - pairs of child and parent node types;</i> - <i>pairs of chronologically consecutive user nodes.</i><i></i></p><table caption="Table 1: Move"></table><p>similar recognition scores may have very differ­ent possible move types:</p></subsection></section><section number="1." title='Command: "Play a rock song by Cher"'></section><section number="2." title='Query: "What rock songs are there?"'><p>We are therefore currently investigating the use of machine learning techniques to improve on our current manual definitions. With an­notated data the optimal weights of a scoring function that combines all the features can be automatically learned (see (Gabsdil and Lemon, 2004)).</p><subsection number="4.3" title="Dialogue-move disambiguation"><p>In order for a winning bid to be unambigu­ously accepted, its score must exceed the next highest score by more than a predefined thresh­old. If not, we take the choice of winning bid to be within our margin of error, and the dia­logue manager asks a disambiguating clarifica­tion question. For example, if the pair of sen­tences in the previous section result in hypothe­sis dialogue moves with scores within the margin of error, then the dialogue manager generates a question of the form:</p><p>"Did you want to play a rock song by Cher or did you ask about rock songs?"</p><p>Alternatively, in some cases there may be a clear highest-scoring bid (i.e. one of high <i>rel­ative </i>confidence) which is itself of low <i>absolute </i>confidence. In such cases, rather than act on the move unconditionally we ask the user for clarifi­cation. If the score is below a certain confidence</p><p>Scoring Features threshold T we treat the highest bid as a rea­sonable hypothesis, but ask for confirmation of the intended move; following the previous ex­ample, this would result in a question such as:</p><p>"I'm not sure I understood that. Did you want to play a rock song by Cher?"</p><p>If the score is below a second critical minimum threshold T2 we take this as a failure in interpre­tation, and prompt for general clarification. As even the best hypothesised move is likely to be incorrect in this case (being of such low confi­dence), asking for specific confirmation is likely to be counter-productive or annoying (see e.g.</p><p>(San-Segundo et al., 2001)).</p><p>Threshold values are currently specified as part of dialogue-move definitions; a future di­rection is to automatically learn optimal values for the thresholds.</p></subsection></section><section number="5" title="Discussion and Conclusions"><p>We have described a number of strategies im­plemented in the CSLI Dialogue Manager to more robustly handle ambiguous or misunder­stood utterances, and low-confidence interpre­tations. Features from multiple sources of ev­idence are combined to rate the possible di­alogue move candidates as interpretations of a user utterance. Features include confidence scores from ASR and parser, as well as seman­tic and pragmatic criteria, and measures related to the dialogue context itself. As well as se­lecting dialogue move, in our multi-device set­ting the approach has the benefit of selecting the device being addressed. Although we have not yet performed a full evaluation of the efficacy of this approach, we have observed sev­eral examples of the n-best list of inputs be­ing (correctly) re-ordered—i.e. after misclas-sification by the statistical parser, the candi­date dialogue-move corresponding to the correct (though lower-confidence) parse can still be se­lected.<page local="8" global="240"/> We are currently gathering data in order to provide a concrete evaluation.</p><p>Confidence thresholds (upper and lower bounds) set by the dialogue designer specify the levels at which a candidate move is rejected, re­quires explicit confirmation by the user, or sim­ply accepted. Future work includes automati­cally learning optimal values for these thresh­olds and optimal weights on the features for scoring candidate dialogue-moves, applying the techniques of e.g. Gabsdil and Lemon (2004) to our multi-device setting.</p></section><references><p>J. Bos, E. Klein, O. Lemon, and T. Oka. 2003. DIPPER: Description and formalization of an information-state update dialogue system ar­chitecture. In <i>Proceedings of the 4th SIGdial Workshop on Discourse and Dialogue.</i></p><p>A. Chotimongkol and A. Rudnicky. 2001. N-best speech hypotheses reordering using lin­ear regression. In <i>Proceedings of the 7th Eu­ropean Conference on Speech Communication and Technology (EUROSPEECH).</i></p><p>J. Fry, M. Ginzton, S. Peters, B. Clark, and H. Pon-Barry. 2001. Automated tutoring di­alogues for training in shipboard damage con­trol. In <i>Proc. 2nd SIGdial Workshop on Dis­course and Dialogue.</i></p><p>M. Gabsdil and J. Bos. 2003. Combining acous­tic confidence scores with deep semantic anal­ysis for clarification dialogues. In <i>Proc. 5th International Workshop on Computational Semantics (IWCS-5).</i></p><p>M. Gabsdil and O. Lemon. 2004. Combining acoustic and pragmatic features to predict recognition performance in spoken dialogue systems. In <i>Proc. 42nd Annual Meeting </i><i>ofthe</i><i> ACL.</i></p><p>S. Larsson and D. Traum. 2000. Informa­tion state and dialogue management in the TRINDI dialogue move engine toolkit. <i>Natu­ral Language Engineering, </i>6.</p><p>O. Lemon and A. Gruenstein. 2004. Multi­threaded context for robust conversational in­terfaces: Context-sensitive speech recognition and interpretation of corrective fragments.</p><p><i>ACM Transactions on Computer-Human In­teraction, </i>11(3). O. Lemon, A. Gruenstein, and S. Peters. 2002. Collaborative activities and multi-tasking in dialogue systems. <i>Traitement Automatique des Langues, </i>43(2). D. Mirkovic and L. Cavedon. 2005. Practical plug-and-play dialogue management. In <i>Pro­ceedings of the Annual Meeting of the Pa­cific Association of Computational Linguis­tics (PACLING).</i></p><p>M. Rayner, D. Carter, V. Digalakis, and P. Price. 1994. Combining knowledge sources to reorder n-best speech hypothesis lists. In <i>Proceedings of the ARPA Human Language Technology Workshop. </i>R. San-Segundo, J. M. Montero, J. Ferreiros, R. Cordoba, and J. M. Pardo. 2001. Design­ing confirmation mechanisms and error re­cover techniques in a railway information sys­tem for spanish. In <i>Proc. 2nd SIGdial Work­shop on Discourse and Dialogue. </i>D. Schlangen. 2004. Causes and strategies for requesting clarification in dialogue. In <i>Proc. 5th SIGdial Workshop on Discourse and Di­alogue.</i></p><p>R. G. Smith. 1980. The contract net protocol: High level communication and control in a distributed problem solver. <i>IEEE Transac­tions on Computers, </i>C-29(12):1104-1113. D. Traum, J. Bos, R. Cooper, S. Larsson, I. Lewin, C. Matheson, and M. Poesio. 1999. A model of dialogue moves and information state revision. In <i>Task Oriented Instructional Dialogue (TRINDI): Deliverable 2.1. </i>Univer­sity of Gothenburg. M. Walker, J. Wright, and I. Langkilde. 2000. Using natural language processing and dis­course features to identify understanding er­rors in a spoken dialogue system. In <i>Proceed­ings ofthe 17th International Conference on Machine Learning. </i>F.   Weng,   L.   Cavedon,   B. Raghunathan,</p><doubt alpha="65.9" length="44" tooSmall="False" monospace="0.0">D. Mirkovic, H. Cheng, H. Schmidt, H. Bratt,</doubt><doubt alpha="62.5" length="40" tooSmall="False" monospace="0.0">R. Mishra, S. Peters, L. Zhao, S. Upson,</doubt><p>E. Shriberg, and C. Bergmann. 2004. A conversational dialogue system for cognitively overloaded users. In <i>Proc. 8th International Conference on Spoken Language Processing</i> <i>(INTERSPEECH).</i><i></i></p></references></body></article>