<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="241"/><title>Automatic Utterance Segmentation in Instant Messaging Dialogue</title><pubinfo>Proceedings of the Australasian Language Technology Workshop 2005,pages 241-249, Sydney, Australia, December 2005.</pubinfo><author surname="Ivanovic" givenname="Edward"><org  name="University of Melbourne" country="Australia" city="Melbourne"/></author></firstpageheader><frontmatter><p><b>Automatic Utterance Segmentation in Instant Messaging Dialogue</b></p><p><b>Edward Ivanovic</b></p><p>Department of Computer Science and Software Engineering University of Melbourne edwardi@csse.unimelb.edu.au</p></frontmatter><abstract>Instant Messaging (IM) chat sessions are real-time, text-based conversations which can be analyzed using dialogue-act mod­els. Dialogue acts represent the seman­tic information of an utterance, however, messages must be segmented into utter­ances before classification can take place. We describe and compare two statistical methods for automatic utterance segmen­tation and dialogue-act classification in task-based IM dialogue. It is shown that IM messages can be automatically seg­mented and classified to a very high ac­curacy using statistical machine learning. </abstract></header><body><section number="1" title="Introduction"><p>Dialogue acts are a useful first level of analysis for describing discourse structure as they represent the illocutionary force of utterances such as assertions and declarations. Early work on speech act theory by Austin (1962) and Searle (1979) has been ex­tended in dialogue acts to model the conversational functions that utterances can perform. Table 1 shows an example dialogue with utterance segments and dialogue acts.</p><p>As illustrated in Table 1, some messages contain multiple utterances and thus require segmentation before each utterance can be classified as a dialogue act. Once utterances are classified, the dialogue-acts may then be used for subsequent tasks such as machine translation (Tanaka and Yokoo, 1999), dia­logue game detection (Levin et al., 1999), and, in the case of spoken dialogue, speech recognition (Stol-cke et al., 2000).</p><p>Speaker Message</p><doubt alpha="100.0" length="7" tooSmall="False" monospace="0.0">Sanders</doubt><p>Customer [Hello Customer]CONVENTIONAL-OPENING, [thank you for contacting MSN Shopping]THANKING. [This is Sanders and I look forward to assist­ing you today]STATEMENT [How are you doing today?]OPEN-QUESTION [good]STATEMENT, [thanks]THANKING [How may I help you today?]OPEN-QUESTION</p><p>Table 1: An example of the beginning of a dia­logue in our corpus showing utterance boundaries and dialogue-act tags in superscript.</p><p>Instant Messaging (IM) consists of two or more people typing messages to each other in real time on a line-by-line basis. Although IM dialogue can take place with a group of people simultaneously writing to each other, for the purposes of this study we assume only two-party dialogue. As described in Ivanovic (2005), sequences of words are grouped into three levels: the first level is a <i>Turn, </i>consisting of at least one <i>Message, </i>which consists of at least one <i>Utterance, </i>defined as follows: <b>Turn: </b>A dialogue participant normally writes one or more message then waits for the other participant to respond, hence taking <i>turns </i>in writing messages. <b>Message: </b>A message is defined as a group of words that are sent from one dialogue participant to the other participant as a single unit. This is usually achieved by typing a message and pressing the En­ter key or a 'Send' button on the client program. A single turn can span multiple messages. <b>Utterance: </b>An utterance can be thought of as one complete semantic unit with respect to dialogue acts. This can be a complete sentence or as short as an emoticon (e.g. ":-)" to smile). Messages contain one or more utterances.<page local="2" global="242"/> Although it is possible to send a message mid-utterance, resulting in utterances span­ning messages, so such instances occur in our cor­pus, which our model assumes. Example utterances, enclosed within brackets, are shown in Table 1.</p><p>Most utterance segmentation research to date has focussed on transcribed speech. The aim of speech segmentation, however, is different to that re­quired by dialogue act classification. That is, large-vocabulary speech recognisers segment speech into <i>acoustic </i>segments for more efficient processing, us­ing criteria such as non-speech intervals and turn boundaries in dialogue. These methods are not ap­propriate for IM utterance segmentation because the acoustic segmentation methods rely on the recorded waveform of speech, which does not exist in IM di­alogue.</p><p>We show that utterance segmentation for dialogue act classification requires very different criteria to transcribed speech segmentation. Our methods for dialogue act utterance segmentation are based on linguistically and statistically motivated approaches.</p><p>The rest of this paper is organised as follows. The data collection and dialogue act tag set are described in Section 2. The methods and language models used in our experiment are explained in Section 3. Evaluation techniques we use are in Section 4. Our experimental results and discussion are in Section 5, with the conclusions and future work in Section 6.</p></section><section number="2" title="Data and Dialogue Act Tag Set"><p>Our data was collected in previous work (Ivanovic, 2005) from an online IM-based support service and consisted of nine chat sessions, totalling 550 utter­ances, 6,500 words, with a mean message length of 10 words. The chat sessions were manually seg­mented into utterances by one person and used as a gold standard. These utterances were then annotated by three people</p><p>Table 2 shows the dialogue act tag set we use, which was also taken from previous work as de­scribed in Ivanovic (2005). The tag set was cho­sen by manually labelling our corpus using tags that seemed appropriate from the 42 tags used by Stol-cke et al. (2000), which in turn were based on the Dialog Act Markup in Several Layers (DAMSL) tag set (Core and Allen, 1997).</p><p>Table 2: The 12 dialogue act labels with examples and frequencies given as percentages of the total number of utterances in our corpus.</p><p>A Kappa analysis used to compare inter-annotator agreement normalised for chance (Siegel and Castel­lan, 1988), resulted in a value of 0.87 with 89% agreement (ivanovic, 2005). A Kappa statistic of 0.8 and above is considered a satisfactory indication that our corpus can be labelled reliability using our tag set (Carletta, 1996).</p><p>A complete list of the 12 dialogue acts we used is shown in Table 2 along with examples and the fre­quency of each dialogue act in our corpus.</p></section><section number="3" title="Methods"><p>Our first goal was to determine which features ob­tained from iM transcripts would be useful in detect­ing utterance segments within messages. The data available from IM chat transcripts are the <i>speaker, message text, </i>and <i>time stamp </i>of each message. Un­like regular written prose, iM chats are often very informal—omitting usual punctuation such as com­mas, periods, question marks, and initial capital let­ters for proper names and new sentences. Spelling mistakes, acronyms for common phrases, and un-grammatical messages are also quite common.</p><p>The observation that utterances in our data do not cross message boundaries allows us to focus on seg­menting one message at a time. we use two ap-<page local="3" global="243"/></p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Tag</p></td><td class="cell"><p>Example</p></td><td class="cell"><p>%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Statement</p></td><td class="cell"><p>I am sending you the page now</p></td><td class="cell"><p>36.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Thanking</p></td><td class="cell"><p>Thank you for contacting us</p></td><td class="cell"><p>14.7</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Yes-No-Question</p></td><td class="cell"><p>Did you receive the page?</p></td><td class="cell"><p>13.9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Response-Ack</p></td><td class="cell"><p>Sure</p></td><td class="cell"><p>7.2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Request</p></td><td class="cell"><p>Please let me know how I can assist</p></td><td class="cell"><p>5.9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Open-Question</p></td><td class="cell"><p>how do I use the international version?</p></td><td class="cell"><p>5.3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Yes-Answer</p></td><td class="cell"><p>yes, yeah</p></td><td class="cell"><p>5.1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Conventional-Closing</p></td><td class="cell"><p>Bye Bye</p></td><td class="cell"><p>2.9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>No-Answer</p></td><td class="cell"><p>no, nope</p></td><td class="cell"><p>2.5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Conventional-Opening</p></td><td class="cell"><p>Hello Customer</p></td><td class="cell"><p>2.3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Expressive</p></td><td class="cell"><p>haha, :-), grr</p></td><td class="cell"><p>2.3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Downplayer</p></td><td class="cell"><p>my pleasure</p></td><td class="cell"><p>1.9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>A: <i>[int</i><i>J </i>hello <i>uh</i>] <i>[np </i>customer <i>nn</i>] , <i>o </i><i>&lt;s&gt; </i>[vp thank <i>vb</i>] <i>[np </i>you prp] [pp for in] [vp contact vbg] [np Msn <i>nnp </i>Shopping <i>nnp</i>] . <i>o </i><i>&lt;s&gt; </i><i>[np </i>this <i>dt</i>] [vp be <i>vbz</i>] [np Sanders <i>nnp</i>] [o and <i>cc</i>] [np I prp] [vp look <i>vbp</i>] <i>[advp </i>forward <i>rb</i>] [pp to <i>to</i>] [vp assist <i>vbg</i>] [np you prp] [np today <i>NN</i>] . <i>o</i></p><p>A: [advp how <i>wrb</i>] [o be vbp] [np you <i>prp</i>] [vp do vbg] [np today <i>nn</i>] ? <i>o </i>B: <i>[adjp </i>good jj] , o &lt;s&gt; [np thanks <i>nns</i>] proaches to segment the messages: Hidden Markov Models (HMMs) and a probabilistic model based on parse trees. We discuss each of these in turn.</p><figure caption="Figure 1: Sample tagged and chunked data."></figure><subsection number="3.1" title="HMM Method"><p>In the absence of reliable punctuation cues, we looked at approaches based on the available lexi­cal information. One such method was to use an HMM to find the most likely segment boundaries. We experimented with three versions of the HMM approach, based on sequences of: (i) lemmas, (ii) part of speech tags, and (iii) head words of chunks.</p><p>The rationale behind using chunks is that the number of possible segments is reduced since ut­terance boundaries do not lie within chunks. The data was assigned POS tags and segmented into chunks via the FNTBL Toolkit (Ngai and Flo­rian, 2001), which is an efficient implementation of Eric Brill's Transformation-based learning algo­rithm (Brill, 1995). Lemmatisation on our corpus was performed using the morphological tools de­scribed in Minnen et al. (2001).</p><p>Figure 1 illustrates some characteristics of the data. Utterance boundaries are marked by &lt;s&gt; tags, chunk boundaries are enclosed within brackets, and words' POS tags are shown in subscript after the word. The actual chunks in the data use IOB tags similar to that described in Ramshaw and Marcus (1995).</p><p>We first trained an n-gram statistical language model with add-one smoothing and Katz backoff (Katz, 1987) to hypothesize the most probable lo­cations of utterance boundaries for each individual message. The resulting segmentations were then evaluated using the WindowDiff metric as described in Section 4.</p><p>Elements used to represent the segments were lemmas, POS tags, and chunks. Segment beginnings in our training data were marked with a &lt;s&gt; tag. This allowed each element to be in one oftwo states: S or NO-S depending on whether it had a &lt;s&gt; tag before it. We build two probability distributions <i>PS</i><i></i>and <i>PNo-S </i>representing the probability that token <i>tk </i>is at the beginning of a segment or not, respec­tively. Using this state information permits us to use an HMM with the following forward computation for the likelihoods of the states at each position <i>k </i>as described by Stolcke and Shriberg (1996):</p><doubt alpha="50.0" length="36" tooSmall="False" monospace="0.0">Pno-s(ti...tk)   =Pno-s(ti...tk-i) x</doubt><doubt alpha="57.1" length="14" tooSmall="False" monospace="0.0">p(tk\tk-2tk-l)</doubt><doubt alpha="50.0" length="16" tooSmall="False" monospace="0.0">+Ps(ti...tk-i) x</doubt><doubt alpha="53.8" length="13" tooSmall="False" monospace="0.0">p(tk\&lt;s&gt;tk-l)</doubt><doubt alpha="48.5" length="33" tooSmall="False" monospace="0.0">Ps(ti...tk)   =Pno-s(ti...tk-i) x</doubt><doubt alpha="45.8" length="24" tooSmall="False" monospace="0.0">p(&lt;s&gt;\tk-2tk-l)p(tk\&lt;s&gt;)</doubt><doubt alpha="43.9" length="41" tooSmall="False" monospace="0.0">+Ps(ti...tk-i) x p(&lt;s&gt;\&lt;s&gt;tk-i )p(tk\&lt;s&gt;)</doubt><p>where <i>t </i>is a lemma, POS or chunk token. A corre­sponding Viterbi algorithm is then used to find the most likely sequence of S and NO-S states given the lemmatised words. Note that this model treats seg­ment marks, &lt;s&gt;, as tokens.</p></subsection><subsection number="3.2" title="Parse Tree Method"><p>Parse trees generally contain nodes of clauses as illustrated in Figure 2. We assume that utterance boundaries only occur at major syntactic boundaries. This is similar in principle to the use of chunks as described in Section 3.1, where we hypothesise that a segment boundary exists before each token. The notion of a token, however, changes from represent­ing chunks to sub-trees within a parse tree. Since a token in this context represents multiple words, and utterance segments may only occur in between tokens, this method significantly reduces the possi­bility of obtaining false-positive segment boundaries<page local="4" global="244"/></p><doubt alpha="66.7" length="3" tooSmall="True" monospace="0.0">NN2</doubt><doubt alpha="100.0" length="1" tooSmall="True" monospace="0.0">I</doubt><doubt alpha="83.3" length="6" tooSmall="True" monospace="0.0">DDI VP</doubt><doubt alpha="66.7" length="12" tooSmall="True" monospace="0.0">thatVB^^^^NP</doubt><doubt alpha="25.0" length="4" tooSmall="True" monospace="0.0">I /\</doubt><doubt alpha="100.0" length="3" tooSmall="True" monospace="0.0">RRQ</doubt><doubt alpha="33.3" length="3" tooSmall="True" monospace="0.0">I ^</doubt><doubt alpha="83.3" length="6" tooSmall="True" monospace="0.0">howVD0</doubt><doubt alpha="100.0" length="1" tooSmall="True" monospace="0.0">S</doubt><doubt alpha="70.0" length="10" tooSmall="True" monospace="0.0">be+sJJ NN1</doubt><doubt alpha="80.0" length="10" tooSmall="True" monospace="0.0">doPPIS1 VP</doubt><doubt alpha="75.0" length="4" tooSmall="True" monospace="0.0">IVV0</doubt><doubt alpha="100.0" length="5" tooSmall="True" monospace="0.0">useAT</doubt><doubt alpha="100.0" length="2" tooSmall="True" monospace="0.0">NP</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">T</doubt><doubt alpha="100.0" length="2" tooSmall="True" monospace="0.0">JJ</doubt><doubt alpha="66.7" length="3" tooSmall="True" monospace="0.0">NN1</doubt><doubt alpha="79.3" length="29" tooSmall="True" monospace="0.0">the     international versior</doubt><p>Figure 2: Parse tree of a message showing utter­ances separated into sub-trees as generated by RASP (Briscoe and Carroll, 2002).</p><p>when compared with using word or chunk tokens as­suming correct parse trees. If the parse trees are not correct, however, this technique will have the oppo­site effect. This is discussed in more detail in Sec­tion 5.3.</p><p>To produce the parse trees, we use the RASP (Ro­bust Accurate Statistical Parsing) parser described in Briscoe and Carroll (2002). RASP is designed to be domain-independent in order to handle text from different genres. Given that our data comes from instant messaging, which exhibits less pre­dictable prose than that typically found in newspa­pers, we chose RASP over other parsers such as Collins (1999) and Charniak (2000) that are opti­mised on the Wall Street Journal treebank.</p><p>Utterance segments in our data always occur within a maximum depth of 2 nodes from the root of the parse tree. Using this depth limit, we first build a table of possible "cuts" through the tree. These cuts, or proper analyses as described in Chomsky (1965), contain every combination of sub-trees, as illustrated in Figure 3, resulting in a sequence <i>C </i>of nodes:</p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">C</doubt><doubt alpha="63.6" length="11" tooSmall="False" monospace="0.0">ti;t2;tS;th</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(1)</doubt><p>where each combination <b><i>ti </i></b>is a sequence of tree nodes such that:</p><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">ti</doubt><doubt alpha="41.2" length="17" tooSmall="False" monospace="0.0">ti,t2)tS;■ ■ ■;tn</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(2)</doubt><p>where the leaves of each tree node <i>ti </i>represent a pos­sible utterance.</p><p>We then calculate the most likely dialogue act for the leaves (words) within each node, independently in the combination table. The result and its corre­sponding dialogue-act are stored with the node ti. Next, we calculate the probability of a correct se­quence of utterances based on the product of the dialogue-act classification probabilities, using the following formulae:</p><doubt alpha="56.7" length="30" tooSmall="False" monospace="0.0">(t*, d*)=arg maxTTP(di\U,di-i)</doubt><doubt alpha="57.6" length="33" tooSmall="False" monospace="0.0">p(di\ti)di-i)= p(di\di-i)np(v\di)</doubt><p>vGleaves(ti)</p><p>where t* is the bestnode combination(orsegments), <i>C </i>is the set of proper analyses, <i>P (di\ti) </i>is the prob­ability of node <i>ti </i><i>G </i>t being dialogue-act <i>d </i>based on its leaves (words), <i>di-i </i>is the previously assigned dialogue-act (using bigrams), and <i>v </i>is a word in node <i>t</i><i>i </i>.</p><p>Using this method has the effect of evaluating the classification and segmentation tasks at the same time, taking the most probable combination. Al­gorithm 1 shows the process used to find the best proper analysis in <i>C</i>. The classify method re­turns the highest probability of all dialogue acts given the words in node <i>n </i>using the naive Bayes method. It also returns the corresponding dialogue act which is then stored with the respective node <i>n.</i></p><p>Importantly, the naive Bayes algorithm uses a bag-of-words as its features, taking the product of each word's probability of being in any given dia­logue act. This allows the product in line 6 of Al­gorithm 1 to be used as a ranking score amongst the proper analyses even though the number of nodes <i>n </i>in t may vary within <i>C</i>. If a different classifica­tion algorithm were used, then line 6 may have to be modified to preserve mathematical tractability.</p><doubt alpha="100.0" length="3" tooSmall="True" monospace="0.0">TOP</doubt><table caption="Figure 3: Proper analyses, Csfrom a parse tree." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>A</p><p><i>/\</i></p></td><td class="cell"><p></p></td><td class="cell"><p><i>C</i></p></td><td class="cell"><p><i>tj</i><b><i>,i</i></b></p></td><td class="cell"><p><i>tj</i><b><i>,2</i></b></p></td><td class="cell"><p><i>tj</i><b><i>,s</i></b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>ti</p></td><td class="cell"><p>Ai,i</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>B C</p></td><td class="cell"><p><i>==</i></p></td><td class="cell"><p></p></td><td class="cell"><p>B2,1</p></td><td class="cell"><p>C2,2</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>A</p></td><td class="cell"><p></p></td><td class="cell"><p>ts</p></td><td class="cell"><p>Bs,i</p></td><td class="cell"><p>Ds,2</p></td><td class="cell"><p>Es,s</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>D E</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="5" global="245"/><p><b>Algorithm 1 </b>Find best utterance segmentations <b><i>ti </i></b><i>G </i><i>C</i>. The classify method also returns the best di­alogue acts and probabilities which are stored with <u>their nodes </u><u>n.</u>_</p><p>1: <i>maxp </i><i>— </i>0 {stores the best probability} 2: <i>maxt </i><i>— </i>None {best tree node} 3: <b>for all </b>t in <i>C </i><b>do</b> 5: <b>for all </b><i>n </i>in t <b>do</b> 6: <i>p </i><i>— </i><i>p</i><i>x </i>classify(leaves(n)) 8: <b>if </b><i>p &gt; maxp </i><b>then</b></p><doubt alpha="20.0" length="5" tooSmall="False" monospace="0.0">4:p—1</doubt><doubt alpha="66.7" length="9" tooSmall="False" monospace="0.0">7:end for</doubt><doubt alpha="62.5" length="8" tooSmall="False" monospace="0.0">9:maxp—p</doubt><doubt alpha="55.6" length="9" tooSmall="False" monospace="0.0">10:maxt—t</doubt><doubt alpha="55.6" length="9" tooSmall="False" monospace="0.0">11:end if</doubt><doubt alpha="60.0" length="10" tooSmall="False" monospace="0.0">12:end for</doubt></subsection></section><section number="4" title="Evaluation"><p>The segmentation algorithms described in Section 3 were evaluated via 9-fold cross-validation where eight of the chat sessions in our corpus were used for training and one for testing. This process is repeated for all dialogues and the mean result is presented.</p><p>In this section, we first discuss why the standard information retrieval evaluation metrics ofrecall and precision are not appropriate for this type of seg­mentation, and then discuss the WindowDiff metric, which is used instead.</p><subsection number="4.1" title="Using the Recall and Precision Metrics for Segmentation"><p>The standard information retrieval metrics of recall and precision are not well-suited to evaluating seg­mentation tasks. Recall is the ratio of correctly hy­pothesised segment boundaries to the total number of actual boundaries. Precision is the ratio of correct boundaries detected to all hypothesised boundaries.</p><p>There are two main problems with using these metrics for segmentation tasks: the first is re­lated to the inherently subjective nature of seg­mentation. An example is with the message "ok - that's great, thanks" in which "ok - that's great" could be segmented and tagged as a sin­gle Acknowledgement or as the two utterances: "[ok]AcKNOWLEDGEMENT - [that's great]STATEMENT". Deciding which segmentation should be considered correct depends largely on how the utterances will be used, that is, the downstream task. The traditional recall and precision metrics will regard the alterna­tive segmentation as an error.</p><p>Similarly, if our corpus has a message that is manually segmented into two or more adjacent ut­terances with the same dialogue-act, the system should not necessarily be penalised for regard­ing the span of text as one segment. For ex­ample, "[Goodbye]CoNVENTIONAL-CLOSING and [take care]CoNVENTIONAL-CLOSING" could just be marked as one utterance.</p><p>The second problem with using recall and preci­sion to evaluate segmentation tasks is the question of how to handle near-boundary misses, that is, a false-positive that occurs near a true boundary. Using re­call and precision in the way described will penalise a system equally regardless of whether a hypothe­sised segment boundary is off by one or ten words.</p></subsection><subsection number="4.2" title="The WindowDiff Metric"><p>The manually segmented data is used as a gold stan­dard with which to compare hypothesised segmen­tations using the WindowDiff metric. The Window-Diff metric, proposed by Pevzner and Hearst (2002), aims to improve segmentation evaluation by reward­ing near-misses.</p><p>WindowDiff works by choosing a window size <i>k </i>that is typically equal to half of the average seg­ment length in a corpus. This k-sized window then slides over the hypothesised segmentation data and compares segment and non-segment marks with the reference data. If the number of hypothesised and reference segments within the window size differ, a counter is incremented and the window continues to the next position. The final score is then divided by the number of scans performed. A perfect system would therefore receive a zero score.</p><p>In most segmentation tasks, segment lengths are uniformly distributed, so using a fixed value for <i>k </i>is appropriate. However, because utterance lengths in our data vary considerably, as shown in Figure 4, we evaluate for different values of <i>k.</i><i> </i>We adjust <i>k</i><i> </i>from 1 to 20 for each message, taking the mean result for each value of k. The maximum allowable value of <i>k </i>is the message length on a per-message basis. This technique provides a fair evaluation given the varied utterance lengths.</p><p>Another question for our experiment is whether allowing any deviation from our reference seg­mented data is acceptable, such as inserting a bound­ary somewhere near an actual boundary.<page local="6" global="246"/> Depending on where a boundary is inserted, this may result in two incomplete utterances as in example (1) below:</p><doubt alpha="57.4" length="47" tooSmall="False" monospace="0.0">(1) a.    Ref: [thanks] [you've been very help-</doubt><doubt alpha="75.0" length="4" tooSmall="False" monospace="0.0">ful]</doubt><doubt alpha="63.8" length="47" tooSmall="False" monospace="0.0">b.    Hyp: [thanks you've] [been very help­ful]</doubt><p>The segments in (1) differ only by one word, but the resulting utterances in (1-b) are confusing, espe­cially when taken in isolation. In this case, we would not want to allow any deviation from the reference data. However, there are cases where a near-miss is acceptable, such as in (2):</p><doubt alpha="54.3" length="46" tooSmall="False" monospace="0.0">(2) a.    Ref:   [Thank you for waiting], Cus-</doubt><p>tomer. [I have found a page that lists a wide variety of Rock Climbing Shoes] b.    Hyp:   [Thank you for waiting, Cus­tomer]. [I have found a page that lists a wide variety of Rock Climbing Shoes]</p><p>Here, the hypothesised segmentation (2-b) is just as acceptable as the reference (2-a).</p><p>The difference between examples (1) and (2) is that (2-b) has maintained the clause boundaries. The word <i>Customer </i>in (2-a) is not part of either segment, so including it in the utterance does not affect the adjacent utterance. Since an utterance is a complete phrase, this is the only way a near-miss may still be considered correct. Some other exceptions exist involving single-word utterances which will not be considered here.</p></subsection></section><section number="5" title="Experimental Results and Discussion"><p>The WindowDiff results for the various models and window sizes are shown in Figure 5 along with the baseline WindowDiff scores. A lower score indi­cates higher accuracy. The best result was achieved by the parse tree method. The worst result was given by the HMM POS tag model, but it still exceeded the baseline.</p><p>The relative difference between the models varies little as the window size changes. The Window-Diff score begins to taper off as <i>k </i>increases past 20 words, which is at approximately the 90th percentile</p><doubt alpha="0.0" length="6" tooSmall="True" monospace="0.0">25 00%</doubt><doubt alpha="0.0" length="6" tooSmall="True" monospace="0.0">20.00%</doubt><doubt alpha="0.0" length="6" tooSmall="True" monospace="0.0">15.00%</doubt><doubt alpha="0.0" length="6" tooSmall="True" monospace="0.0">10.00%</doubt><doubt alpha="0.0" length="5" tooSmall="True" monospace="0.0">5 00%</doubt><doubt alpha="0.0" length="5" tooSmall="True" monospace="0.0">0.00%</doubt><p>Figure 4: Frequency distribution of utterance length in words. The mean length is 7.6 words and the me­dian is 6 words.</p><p>of utterance lengths in our corpus. This plateau is due to window lengths having no effect on shorter messages as a result of the adjustment we make to <i>k </i>when <i>k </i>is greater than the message length.</p><p>The better evaluation scores for small values of <i>k </i>are simply due to the way the WindowDiff al­gorithm compares segments within a window. An equal penalty is applied regardless of whether there are five or two segments within a window that should only contain one. Therefore, a window length span­ning the entire message will at most return only one penalty if the hypothesised segments differ at all from the reference segments. Since the window spans the entire message, only one comparison is performed which results in the equivalent of a 100% error rate. Conversely, when <i>k </i>is small, the num­ber of unequal windows between the reference and hypothesised segmentations will also be small since we have so few false positives. At the same time, the number of comparisons will be high, leading to a low WindowDiff score.</p><p>A perfect score of 0 is never achieved since there are always some misaligned segments. We never see a score of 1 since many of the single-utterance mes­sages are accurately detected, as discussed in Sec­tion 5.1 below. Likewise, none of the models ap­proach the baseline as the window size increases, which indicates that some of the multi-utterance messages are also accurately detected.</p><p>Although no individual value of <i>k </i>can be used to judge performance because of the varying segment (utterance) lengths in our corpus, we can confidently gauge the performance of each method relative to each other method since their respective rankings re­main constant for all values of <i>k.</i><page local="7" global="247"/><i></i></p><subsection number="5.1" title="Baseline"><p>An analysis of our data revealed that messages con­tain up to three utterances. Of these messages, 60% contain only one utterance, 20% contain two utter­ances, and the remaining 20% consist of three utter­ances.</p><p>The baseline is calculated by assuming that each message contains only one utterance since this is the majority class.</p></subsection><subsection number="5.2" title="HMM Results"><p>We used three types of features with the HMM: lem­mas, POS tags, and the head word of chunked data. The POS tag model performs the worst, whereas the lemma model is the best of the HMM models. This indicates that cue words play a major part in deter­mining utterance segment boundaries. Replacing the words with their respective POS tags loses this infor­mation.</p><p>Using POS tags can sometimes help overcome data sparseness problems as it has the effect of generalising words. However, in this case it overgeneralises, resulting in poorer performance.</p><p>The rationale behind using chunks is that it re­duces the number of possible boundaries as we hy­pothesise boundaries between chunks rather than words. Since utterance boundaries do not lie within chunks, this may have increased the probability of correct segment boundary detection. However, the results show that the HMM benefits from using all words rather than only the chunks' head words.</p><p>The main types of errors produced by the HMM are false positives based on words that commonly occur at the start of an utterance, such as "what", occurring mid-sentences as in (3):</p><doubt alpha="63.2" length="38" tooSmall="False" monospace="0.0">(3)     butI'mnot sure what to get her</doubt><p>The reference data has this as one utterance, but the HMM detects a false positive starting at "what".</p></subsection><subsection number="5.3" title="Parse Tree Results"><p>The parse tree method gives the best results. A qual­itative evaluation of the dialogue act classifications assigned to detected utterances gave an accuracy of 84%. The baseline for the dialogue act classification task was 36%, which was the majority class being Statement.</p><p>The most common type of error the parse tree method makes is to separate words near the root of a parse tree away from a deeper right node.<page local="8" global="248"/> Figure 6 shows a parse tree produced by RASP for (4) below:</p><doubt alpha="100.0" length="1" tooSmall="True" monospace="0.0">I</doubt><doubt alpha="100.0" length="1" tooSmall="True" monospace="0.0">S</doubt><doubt alpha="100.0" length="2" tooSmall="True" monospace="0.0">NP</doubt><doubt alpha="100.0" length="3" tooSmall="True" monospace="0.0">TOP</doubt><doubt alpha="100.0" length="2" tooSmall="True" monospace="0.0">UH</doubt><doubt alpha="50.0" length="2" tooSmall="False" monospace="0.0">J*</doubt><doubt alpha="100.0" length="3" tooSmall="True" monospace="0.0">PPY</doubt><doubt alpha="100.0" length="5" tooSmall="True" monospace="0.0">youIF</doubt><doubt alpha="100.0" length="3" tooSmall="True" monospace="0.0">for</doubt><doubt alpha="80.0" length="5" tooSmall="True" monospace="0.0">VM RR</doubt><doubt alpha="72.2" length="18" tooSmall="True" monospace="0.0">would    surelyVV0</doubt><doubt alpha="50.0" length="4" tooSmall="True" monospace="0.0">JA ,</doubt><doubt alpha="61.5" length="13" tooSmall="True" monospace="0.0">right    ,CCB</doubt><doubt alpha="87.5" length="8" tooSmall="True" monospace="0.0">butPPIS1</doubt><doubt alpha="72.7" length="11" tooSmall="True" monospace="0.0">VP XX JJ JJ</doubt><doubt alpha="25.0" length="12" tooSmall="False" monospace="0.0">/\  I    I I</doubt><doubt alpha="62.9" length="35" tooSmall="True" monospace="0.0">CC    VD0not+     speak/read french</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">I I</doubt><doubt alpha="83.3" length="6" tooSmall="True" monospace="0.0">and do</doubt><doubt alpha="66.7" length="9" tooSmall="True" monospace="0.0">NP1 PPIS1</doubt><doubt alpha="56.2" length="16" tooSmall="True" monospace="0.0">toVV0     PPY RT</doubt><doubt alpha="37.5" length="8" tooSmall="True" monospace="0.0">I    I I</doubt><doubt alpha="66.7" length="18" tooSmall="True" monospace="0.0">help     you today</doubt><doubt alpha="77.8" length="9" tooSmall="True" monospace="0.0">not+VP DD</doubt><doubt alpha="25.0" length="4" tooSmall="True" monospace="0.0">/\ I</doubt><doubt alpha="53.8" length="13" tooSmall="True" monospace="0.0">VV0     IOany</doubt><doubt alpha="85.7" length="7" tooSmall="True" monospace="0.0">know of</doubt><doubt alpha="100.0" length="2" tooSmall="True" monospace="0.0">VP</doubt><doubt alpha="100.0" length="2" tooSmall="True" monospace="0.0">PP</doubt><doubt alpha="75.0" length="4" tooSmall="True" monospace="0.0">IVD0</doubt><doubt alpha="100.0" length="3" tooSmall="True" monospace="0.0">VVG</doubt><doubt alpha="100.0" length="5" tooSmall="True" monospace="0.0">tryTO</doubt><doubt alpha="80.0" length="5" tooSmall="True" monospace="0.0">do XX</doubt><doubt alpha="91.7" length="12" tooSmall="True" monospace="0.0">approach+inç</doubt><figure caption="Figure 6: Erroneous parse tree of sentence (4) as produced by RASP."></figure><p>(4) Thank you for approaching us.   I would surely try to help you today</p><p>The parse tree for (4) is problematic. The first word, "thank", is detached from the S node that contains the rest of the sentence. Our model treats (4) as a sequence of word to­kens <i>W = w1,w2</i><i>,w3,...,w13 </i>and finds that P(Thanking|wi) <i>x </i>P(Statement|W2<footnote anchor="13"/>) <i>&gt; P(d\W</i>), where <i>d </i>is any dialogue act. In this in­stance, RASP failed to segment the two sentences in this message, which prevented our model from evaluating the correct utterances. This illustrates the high dependency our model has on the quality ofthe generated parse trees.</p><p>Another type of error is that the model does not detect any segmentations within a message where there ought to be. An instance of this is in (5) be­low:</p><p>(5) right, but I do not know of any and do not speak/read french</p><p>The reference data has the word "right" segmented and tagged as Response-Ack and the rest of the message as one Statement. However, our model does not evaluate that possibility as the correspond­ing parse tree in Figure 7 does not combine the words as would be expected.</p><figure caption="Figure 7: Parse tree of sentence (5) as produced by RASP."></figure></subsection></section><section number="6" title="Conclusion and Future Work"><p>Finding utterance boundaries in IM dialogue is a critical step for aiding utterance classification and downstream language processing modules such as dialogue response planning. We have shown that the parse trees model obtains the best results. Of the HMM models, the HMM over lemmas in messages performs better than using chunked data and POS­tags, which lose too much information and impede accuracy.</p><p>The parse tree method performed best overall and has the advantage of combining both segmentation and classification tasks in one step to give the op­timal combined result. It is based on the linguistic intuition that utterances are complete constituents, which are modelled well by parse trees. However, this heavy reliance on the quality of the parse trees is also a weakness. Most of the errors obtained us­ing the parse tree method may be attributed to poor, or at least unexpected, parse trees being produced. That notwithstanding, the preliminary results using the RASP parser are very encouraging.</p><p>In future work, we intend to focus more on pars­ing IM messages, taking into account some of its distinct characteristics. Some obvious steps to pro­duce better parse trees are to perform spelling cor­rections and expand acronyms, such as "idk" for "I don't know". Existing parsers will thus be able to produce more accurate parse trees, which will in turn result in higher segmentation accuracy.</p><p>We will also investigate the subjectiveness of ut­terance segmentation by performing Kappa (Siegel and Castellan, 1988) analysis on our segmentation boundaries.<page local="9" global="249"/> The Kappa analysis will give an indica­tion as to the meaningful upper bounds of the per­formance of our system.</p></section><section title="Acknowledgments"><p>Thanks to Steven Bird, Timothy Baldwin, and the Language Technology Group at Melbourne Univer­sity for their constructive comments. Thanks also to Trevor Cohn, Phil Blunsom, and the anonymous reviewers for their very helpful feedback. The data used in this study was POS tagged, chunked, and parsed by Timothy Baldwin using the tools de­scribed with some modifications.</p></section><references><p>John L. Austin. 1962. <i>How to do Things with Words. </i>Clarendon Press, Oxford.</p><p>Eric Brill.   1995.   Transformation-based error-driven learning and natural language processing: a case study in part-of-speech tagging. <i>Computational Linguistics, </i>21(4):543-565.</p><p>Ted Briscoe and John Carroll. 2002. Robust accurate sta­tistical annotation of general text. In <i>Proceedings of the Third International Conference on Language Re­sources and Evaluation, </i>pages 1499-1504, Las Pal­mas, Gran Canaria.</p><p>Jean Carletta. 1996. Assessing agreement on classifica­tion tasks: the kappa statistic. <i>Computational Linguis­tics, </i>22(2):249-254.</p><p>Eugene Charniak. 2000. A maximum-entropy-inspired parser. In <i>Proceedings of the first conference on North American chapter of the Association for Computa­tional Linguistics, </i>pages 132-139, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.</p><p>Noam Chomsky. 1965. <i>Aspects of the Theory of Syntax. </i>MIT Press, Cambridge, MA.</p><p>Michael John Collins. 1999. <i>Head-driven statistical models for natural language parsing. </i>Ph.D. thesis, University ofPennsylvania, Philadelphia. Supervisor­Mitchell P. Marcus.</p><p>Mark Core and James Allen. 1997. Coding dialogs with the DAMSL annotation scheme. <i>Working Notes of the AAAI Fall Symposium on Communicative Action in Humans and Machines, </i>pages 28-35.</p><p>Edward Ivanovic. 2005. Dialogue act tagging for instant messaging chat sessions. In <i>Proceedings ofthe ACL Student Research Workshop, </i>pages 79-84, Ann Arbor,</p><p>Michigan, June. Association for Computational Lin­guistics.</p><p>Slava M. Katz. 1987. Estimation of probabilities from sparse data for the language model component of a speech recognizer. <i>IEEE Transactions on Acoustics, Speech, and Signal Processing, </i>35(3):400-401.</p><p>Lori Levin, Klaus Ries, Ann Thyme-Gobbel, and Alon Lavie. 1999. Tagging of speech acts and dialogue games in Spanish call home. <i>Towards Standards and Tools for Discourse Tagging (Proceedings ofthe ACL Workshop at ACL '99), </i>pages 42-47.</p><p>Guido Minnen, John Carroll, and Darren Pearce. 2001. Applied morphological processing ofenglish. <i>Natural Language Engineering, </i>7(3):207-223.</p><p>Grace Ngai and Radu Florian. 2001. Transformation-based learning in the fast lane. In <i>Proceedings of NAACL-2001, </i>pages 40-47.</p><p>Lev Pevzner and Marti A. Hearst. 2002. A critique and improvement of an evaluation metric for text segmen­tation. <i>Computational Linguistics, </i>28(1):19-36.</p><p>Lance Ramshaw and Mitch Marcus. 1995. Text chunk­ing using transformation-based learning. In David Yarovsky and Kenneth Church, editors, <i>Proceedings of the Third Workshop on Very Large Corpora, </i>pages 82-94, Somerset, New Jersey. Association for Compu­tational Linguistics.</p><p>JohnR. Searle. 1979. <i>Expression and Meaning: Studies in the Theory ofSpeech Acts. </i>Cambridge University Press, Cambridge, UK.</p><p>Sidney Siegel and N. JohnCastellan, Jr. 1988. <i>Nonpara-metric statistics for the behavioral sciences. </i>McGraw-Hill, second edition.</p><p>Andreas Stolcke and Elizabeth Shriberg. 1996. Au­tomatic linguistic segmentation of conversational speech. In <i>Proceedings, ICSLP 96. Fourth Interna­tional Conference on Spoken Language, </i>volume 2, pages 1005-1008, Philadelphia, PA, Oct. ICSLP.</p><p>Andreas Stolcke, Noah Coccaro, Rebecca Bates, Paul Taylor, Carol Van Ess-Dykema, Klaus Ries, Eliza­beth Shriberg, Daniel Jurafsky, Rachel Martin, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. <i>Computational Linguistics, </i>26(3):339-373.</p><p>Hideki Tanaka and Akio Yokoo. 1999. An efficient statistical speech act type tagging system for speech translation systems. In <i>Proceedings of the 37th con­ference on Association for Computational Linguistics, </i>pages 381-388. Association for Computational Lin­guistics.</p></references></body></article>