<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="264"/><title>HotSpots: Visualizing Edits to a Text</title><author surname="Bangalore" givenname="Srinivas"><org  name="AT&amp;T Labs - Research AT&amp;T Labs - Research" country="USA"/></author><author surname="Smith" givenname="David"><org  name="Park Ave Florham Park" country="USA"/></author></firstpageheader><frontmatter><p><b>HotSpots: Visualizing Edits to a Text</b></p><p><b>Srinivas Bangalore David Smith</b></p><p>AT&amp;T Labs - Research AT&amp;T Labs - Research</p><p>180 Park Ave 180 Park Ave</p><p>Florham Park, NJ 07932 Florham Park, NJ 07932</p><p>srini@research.att.com dsmith@research.att.com</p></frontmatter><abstract>Compared to the telephone, email based cus­tomer care is increasingly becoming the pre­ferred channel of communication for corpora­tions and customers. Most email-based cus­tomer care management systems provide a method to include template texts in order to re­duce the handling time for a customer's email. The text in a template is suitably modified into a response by a customer care agent. In this paper, we present two techniques to im­prove the effectiveness of a template by pro­viding tools for the template authors. First, we present a tool to track and visualize the ed­its made by agents to a template which serves as a vital feedback to the template authors. Second, we present a novel method that au­tomatically extracts potential templates from responses authored by agents. These meth­ods are investigated in the context of an email customer care analysis tool that handles over a million emails a year. </abstract></header><body><section number="1" title="Introduction"><p>Email based customer care is increasingly becom­ing the preferred channel of communication for cor­porations and customers compared to the conven­tional telephone-based customer care. For cus­tomers, email channel offers several advantages -there are no tedious menus to navigate, there is no waiting time to reach an operator, the request can be formulated at the customer's pace and additional material supporting the case can be attached to the email. There is also a record of the service re­quest for the customer unlike the telephone-based customer care. However, there are also limitations of the email channel. The most significant one is that the customer-agent interaction could be drawn out over successive emails spanning over several days as opposed to being resolved in one or two telephone calls. For corporations, the asynchronous nature of email-based customer care offers signifi­cant opportunities to reduce operations cost by ef­fective load balancing compared to telephone-based customer care. It is quite common for an email cus­tomer care agent to work on several cases simulta­neously over a period of a few hours. Email chan­nel also offers higher bandwidth for corporations to send additional information in the form of web links, images and video or audio instructions.</p><p>The effectiveness of customer care in the email channel is measured using two competing metrics: Average Handling Time (AHT) and Customer Ex­perience Evaluation (CEE). AHT measures the time taken from when a customer email is opened to the time when the response is sent out. This time is typ­ically averaged over a period of a week or a month for reporting purposes. CEE measures customer sat­isfaction through a survey of a random subset of cus­tomers who have interacted with the email customer care center. These surveys typically involve qual­itative and quantitative questions and measure the quality of the interactions along a number of differ­ent dimensions. As is the case in many surveys the population responding to such questionnaires is typ­ically small and very often quite biased. We do not use the CEE metric for the work we report in this paper.</p><p>As is evident from the definitions of AHT and CEE, it is in the interest of a corporation to minimize AHT while maximizing CEE. In order to reduce AHT, most email customer care systems (Kana, 2008; Genesys, 2008) provide a mechanism for an agent to respond to a customer's email by selecting a predefined template text that can be quickly cus­tomized to serve as the response. The template text is usually associated with a problem category it is in­tended to address and might even be suggested to the agent automatically using classification techniques</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">264</doubt><p><i>Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, </i>pages 264—273, Honolulu, October 2008. ©2008 Association for Computational Linguistics<page local="2" global="265"/></p><p>applied to the customer's email. Once the template is selected, the agent edits the template text to per­sonalize as well as add case specific details as part of composing a response. Each of the text edits con­tributes to the handling time of the email. Hence, it is in the interest of the template designer to mini­mize the number of edits of the template in order to lower AHT.</p><p>Although most email management systems pro­vide a mechanism to author the template text, there is typically no mechanism to monitor and track how these templates are modified by the agents when they compose a response. This information is vital to the template authors when creating new versions of the templates that reduce the number of edits and consequently reduce AHT.</p><p>In this paper, we present two methods for improv­ing the templates in a principled manner. After de­scribing the related work in Section 2, we present a brief description of the email tracking tool we have developed in Section 3. In Section 4, we present a tool called HotSpots that helps visualize the edits being made by the customer care agents to the tem­plates. This tool provides a visual feedback to the template authors and suggests means of improving the template text based on the edits made by agents. In Section 5, we present a new approach to automat­ically identify emerging templates - texts that are repeatedly created by agents and are similar to each other but distinct from the current template text. We use AHT as the metric to minimize for automatic identification of emerging templates. We discuss some of the issues concerning this work in Section 6 and conclude in Section 7.</p></section><section number="2" title="Related Work"><p>There are few threads of research that are relevant to the work presented in this paper. First, the topic of email response generation in the context of customer care has been investigated by (Coch, 1996; Lapalme and Kosseim, 2003; Zukerman and Marom, 2007). In (Coch, 1996), the authors model multi-sentence generation of response letters to customer com­plaints in French. The generation model is carefully crafted for the domain using domain-specific rules for conceptual planning, rhetorical relations and sur­face word order operators.  They show that their approach performs better than predefined templates and slightly worse than human generated responses. In (Lapalme and Kosseim, 2003), the authors ex­plore three different approaches based on classifica­tion, case-based reasoning and question-answering to compose responses to queries in an email cus­tomer care application for the telecommunication in­dustry. The case-based reasoning approach is the most similar to the template approach we follow. In (Zukerman and Marom, 2007), the authors investi­gate an approach to assembling a response by first predicting the clusters of sentences to be included in the response text and then applying multi-document summarization techniques to collate the representa­tive sentences into a single response. In contrast, in this paper, due to constraints from the deployment environment, we rely on a template-based approach to response generation. We focus on providing tools for investigating how the templates are modified and suggest techniques for evolving more effective tem­plates based on quantitative criteria.</p><p>Another thread of relevant research are methods for visualizing texts. There are several methods that have been proposed to provide a visual map of a set of text documents with the focus of illustrating the relatedness of these texts (Card et al., 1999). Us­ing a metric for comparing texts (e.g. n-gram over­lap) , the texts are clustered and the resulting clus­ters are visualized as two or three dimensional color maps. These approaches are useful to depict similar­ities in a static repository of documents or the return results of a search query. These maps are primar­ily designed for exploration and navigation through the document space. While the underlying algorithm we use to illustrate the text edits is similar to the one used in text map visualizations, our focus in this pa­per is to provide a mechanism for template designers to quickly identify the variants of a template sen­tence created by the agents.</p><p>A third thread is in the context of human­assisted machine translation, where a human trans­lator post-edits the output of a machine translation system (Foster et al., 1997; Foster et al., 2002; Och et al., 2003). In order to improve the efficiency of a human translator, the k-best output of a translation system could be displayed as word or phrase choices which are color coded based on the confidence value assigned by the translation model. While the approach we follow is partly motivated by the post­editing paradigm, there are significant differences in the context we apply this approach.<page local="3" global="266"/> In the context of this paper, the template designer is presented a summary of the set of variants created by each agent for each sentence of the template. The task of the template designer is to use this tool to select (or con­struct) a new variant for the template sentence with the aim of minimizing the need for editing that sen­tence in future uses of the template.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">265</doubt></section><section number="3" title="Email Customer Care"><p>Typically, a large email customer care management center receives over 100,000 emails a month. These centers typically use a customer care management system that offer not only logging and tracking of emails but also tools for improving the efficiency of agents responding to emails. Usually, an incom­ing customer email is categorized into a set of few topics/issues. The categorization might be done au­tomatically based on regular expressions involving keywords in the email or using weighted classifiers that are trained on data. In order for an agent to re­spond to an incoming email, these systems provide a text box which allows the agent to author a response from scratch. However, most email customer care systems offer the ability to store a prefabricated re­sponse (also called <i>templates), </i>instead of agents hav­ing to author a response from scratch. These tem­plates are typically associated with a problem cate­gory or an issue that they are intended to address.</p><p>A template helps an agent compose a well-formed response quickly. It contains hints for information that the agent should enter as well as indications of where that information should be entered in the tem­plate. The template might also contain helpful infor­mation to the customer in addition to legal verbiage that the customer needs to be aware of.</p><p>An agent receives a customer email and after comprehending the issues and consulting the cus­tomer records in the database, selects one of the pre­defined set of templates that best addresses the is­sues raised in the email. Less frequently, she might even select more than one template to compose the response. She then proceeds to edit and personal­ize the chosen templates to better suit the customer's email. An example of a 'generic' template - not associated with a specific problem category is shown in Figure 1.</p><p>Greetings Contact.FirstName,</p><p>Thank you for your email in regard to</p><p>XXXXXXXX.</p><p>I will be happy to assist you with your in­quiry.</p><p>XXX BODY XXX If I can be of any further assistance, please reply directly to this email. Thank you for using our company. We appreciate your business and contin­ued loyalty. Regards, Agent.FirstName</p><figure caption="Figure 1: An example of a generic template"></figure><p>The process of selecting an appropriate template that addresses the customer's inquiries could be quite tedious when there are hundreds of templates. Email management systems offer tools that suggest appropriate template to use based on the content of the customer's email. These tools are trained using classification techniques on previous email interac­tions.</p><p>As mentioned earlier, there are two metrics that are typically used to measure the effectiveness and efficiency of email responses. Customers are sur­veyed after their email interaction to assess their level of satisfaction for the service they received. This is usually called the Customer Experience Evaluation (CEE) and includes an evaluation of the customer's total interaction experience with the cor­poration, not just the last email interaction. A small subset of customers who had an interaction with the email center is randomly chosen (typically in the or­der of about 10% of customers) and are invited to take part in the follow-up survey. Typically, only a small percent (about 10%) of the customers who receive these invitations respond to the survey; ef­fectively about 1% of the total emails have customer survey scores.</p><p>A second metric that is also used to measure the efficiency of an operation is called the average han­dling time (AHT) which measures the average of times taken by agents to respond to emails.<page local="4" global="267"/> The handling time includes the time to comprehend the email, the time for database lookup and the time for response composition. It is in the interest of the email customer care operation to minimize AHT and maximize CEE scores.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">266</doubt><subsection number="3.1" title="Email Customer Care Analysis Tool"><p>We have designed and developed an Email Customer Care Analysis Tool (ECAT) to help analyze the op­erations of the email care center. It provides an end-to-end view from the activities involved in answer­ing emails to the results of subsequent customer care surveys. In addition, ECAT also provides insights into how the agents are editing templates as well as guides template authors in designing more effective templates.</p><p>ECAT is a web-based tool and offers a birds-eye summary of the operations aggregated by region, the template used, and the customer satisfaction sur­vey results. Using this tool, analysts can drill down through a series of views until they are eventually presented with the results of a single survey or a sin­gle email interaction.</p><p>One of the most useful functions of the tool is that it shows the extent to which agents edit the tem­plates in the process of creating responses to cus­tomer emails. The degree to which a template is edited is based on Levenshtein string edit distance metric (Levenshtein, 1966). This metric measures the number of edits (substitution, deletion and inser­tions) of words that are needed to transform a tem­plate into a response. The number of edits is normal­ized by the number of words in the template. These morphing scores can be viewed for a single email or averaged per agent or per template used. The scores range from 100 to 0, with 100 representing a tem­plate which hadn't been edited at all.</p><p>The tool also allows the morphing score to be viewed alongside the handling time for an email, in other words the amount of time that the agent spends gathering data and actually composing a response. Handling time is an important metric since it is di­rectly related to cost of operating the email customer care center. The more editing an agent does, the more time they take to respond to a customer. So, the number of templates, their precise wording and the ease with which agents can distinguish them obviously have significant influences on overall han­dling time.</p><p>Beyond the confines of the email centers them­selves, the CEE is the most important elements in gauging the effectiveness of the agent. The survey asks customers to rate their overall satisfaction with the email reply to their question. Five is the high­est score which equates with 'Extremely Satisfied' while a one equals 'Extremely Dissatisfied.' Cus­tomers are also asked to rank the email in terms of it's content, clarity, professionalism and the length of time it took to receive a reply. The customer is also allowed to enter some free text so that they can say how satisfied they were, or not, with how an in­quiry or problem was dealt with. Customers can also say whether they called the company using a tele­phone channel before turning to the email channel.</p><p>The survey files, all of which can be accessed in their entirety from within the ECAT tool, also con­tain information on what templates were used when replying to the customer. They also tell the analyst who the replying agent was and whether this was the first or a subsequent email in communications between the customer and the company.</p><p>The ECAT tool juxtaposes this CEE score with the template morphing score to show correlations between customer satisfaction and the degree to which the template had been edited. This data is graphed so that the analyst can immediately see if heavy editing of a template is leading to higher CEE. Heavy editing with a low customer rating could mean that the template is not helping the agent to respond correctly to the customer.</p></subsection></section><section number="4" title="HotSpots"><p>We designed the HotSpots tool that provides in­sights to the template authors on how templates are being edited by the agents when creating responses. It suggests methods for improving the next version of the template so as to reduce edits by agents and hence reduce the handling time for an email. In this section, we discuss the algorithm and the visualiza­tion of the information that aids template authors in improving the efficacy of the templates.</p><p>The HotSpots algorithm proceeds in two steps as shown in Algorithm 1. The first step creates an alignment between the template string and the re<b>Algorithm 1 </b>Compute HotSpots for a template <i>T </i>given a response set R<page local="5" global="268"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">267</doubt><doubt alpha="40.0" length="10" tooSmall="False" monospace="0.0">1:EdEv =4&gt;</doubt><doubt alpha="31.2" length="16" tooSmall="False" monospace="0.0">2: T = S1S2...Sn</doubt><doubt alpha="43.8" length="16" tooSmall="False" monospace="0.0">3:Ts= [si\l&lt;i&lt;n}</doubt><doubt alpha="48.7" length="39" tooSmall="False" monospace="0.0">4:Rs= [rj\Rj GR,Rj=r{rj2...rLj, 1&lt;i&lt;mj}</doubt><doubt alpha="64.7" length="17" tooSmall="False" monospace="0.0">5:Index:[TsURs}—I</doubt><doubt alpha="57.9" length="38" tooSmall="False" monospace="0.0">6:Tin= Index(s1)Index(s2)... Index(sn)</doubt><p>7: <b>for all </b><i>R </i>G R <b>do</b></p><doubt alpha="42.9" length="14" tooSmall="False" monospace="0.0">8:R =nr2...rnR</doubt><p>9:    Rin = <i>Index(r1)Index(r2)... Index(rnR) </i>// compute distance with sentences as tokens and return the alignment and score 10:    <i>(alignment, score) = IndDist(Tin, Rin) </i>// for each of the sentences in T, update its map 11:    <b>for all </b><i>si</i><i> </i>G <i>T</i><i> </i><b>do</b> 13:       <b>if </b>(s^ <i>e) </i><i>G </i><i>alignment </i><b>then</b></p><doubt alpha="52.9" length="17" tooSmall="False" monospace="0.0">12:in = Index(si)</doubt><doubt alpha="61.3" length="31" tooSmall="False" monospace="0.0">14:EdEv[in].map = EdEv[in].mapU</doubt><doubt alpha="63.2" length="68" tooSmall="False" monospace="0.0">[*delete*}15:else//(si,rj) Galignment16:EdEv[in].map = EdEv[in].mapU</doubt><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">[rj}</doubt><doubt alpha="58.6" length="29" tooSmall="False" monospace="0.0">17:end if18:end for19:end for</doubt><p>// Cluster the response sentences aligned for each template sentence 20: <b>for all </b><i>s</i><i>i G </i><i>T </i><b>do</b> <b>Algorithm 2 </b>KmedianCl: Compute <i>k </i>centroids for a set of strings <i>S </i>using k-median clustering algorithm</p><doubt alpha="52.9" length="17" tooSmall="False" monospace="0.0">21:in = Index(si)</doubt><doubt alpha="64.4" length="45" tooSmall="False" monospace="0.0">22:Cl = KmedianCl(EdEv[in}.map,ncl)23:end for</doubt><p>1: <i>cs </i><i>= </i><i>4&gt;</i><i> </i>// centroid of string s's cluster 2: cej <i>= </i><i>4&gt;</i><i> </i>// centroid of cluster <i>i</i> 3: <i>numcl </i><i>= 0 </i>// number of cluster created so far 6:     <b>if </b><i>numcl </i><i>= 0 </i><b>then</b></p><doubt alpha="66.7" length="30" tooSmall="False" monospace="0.0">4:Clj=4&gt;// members of clusteri</doubt><doubt alpha="66.7" length="30" tooSmall="False" monospace="0.0">5:while(numcl&lt;k)A(numcl&lt;\S\)do</doubt><doubt alpha="65.2" length="23" tooSmall="False" monospace="0.0">7:ceo=argmin^ Dist(c,s)</doubt><doubt alpha="85.7" length="7" tooSmall="False" monospace="0.0">cES sGS</doubt><p>8:    <b>else </b>// select the string <i>(s) </i>that is farthest from its centroid <i>(c</i><i>s </i>) 9: <i>ce</i><i>numci </i><i>= </i><i>argmax Dist(c</i><i>s,</i><i> </i><i>s)</i></p><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">sES</doubt><doubt alpha="55.6" length="9" tooSmall="False" monospace="0.0">10:end if</doubt><p>// Move strings to the closest cluster and compute centroids until the set of centroids don't change</p><doubt alpha="66.7" length="9" tooSmall="False" monospace="0.0">11:repeat</doubt><p>12:        <b>for all </b><i>s </i><i>G </i><i>S </i><b>do</b></p><doubt alpha="62.5" length="24" tooSmall="False" monospace="0.0">13:i*=argmin Dist(cei,s)</doubt><doubt alpha="50.0" length="10" tooSmall="False" monospace="0.0">14:cs=cei*</doubt><doubt alpha="43.8" length="16" tooSmall="False" monospace="0.0">15:Cl,*=Cli*U[s}</doubt><p>// Computed the closest cluster centroid <i>cei</i> to s.</p><doubt alpha="60.0" length="10" tooSmall="False" monospace="0.0">16:end for</doubt><p>// Recompute the cluster centroids <i>ce</i><i>i </i>17: <b>for all </b><i>i </i>such that <i>0 &lt; </i><i>i </i><i>&lt; </i><i>numcl </i><b>do </b>18: <i>cei </i><i>= </i><i>argmin  ^ Dist(c,s)</i></p><p><i>c </i><i>E </i><i>Cl</i><i>i </i><i>s</i><i>E</i><i>Cl</i><i>i</i></p><doubt alpha="60.0" length="10" tooSmall="False" monospace="0.0">19:end for</doubt><p>20:     <b>until </b>set of centroids does not change 21:     <i>numcl </i><i>= </i><i>numcl </i><i>+ 1 </i>// new cluster added 22: <b>end while</b><page local="6" global="269"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">268</doubt><p>sponse string. For the purposes of this paper, we consider the alignments between the template text and the response text with sentences as tokens in­stead of a word-based alignment. The rationale for this tokenization is that for template develop­ers the visualization of the edits is expected to be more meaningful when aggregated at the sentence level rather than at the word level. In the second step, using the sentence-level alignment we compute the edit events (insertion, deletion and substitution) of the template sentences in order to create the re­sponse. All the edits events associated with a tem­plate sentence are then clustered into <i>k </i>clusters and the centroids of the <i>k </i>clusters are displayed as the potential changes to that sentence. We next describe these two steps in detail as illustrated in Algorithm 1 and Algorithm 2.</p><p>Given a set of responses <i>R </i>that agents create us­ing a template <i>T, </i>Algorithm 1 proceeds as follows. Each of the sentences in the template and the set of responses are mapped into an integer index (Line 1). The template <i>T </i>and each of the responses in R are split into sentences and mapped into index sequences (Line 6 and Line 9). The alignment be­tween the two index strings is computed in Line 10. This is a dynamic programming algorithm sim­ilar to computing Levenshtein distance between two strings, except the cost function used to compute the match between tokens is as shown below.</p><p>From the alignment that maps <i>s</i><i>i </i>to <i>r</i><i>j, </i>we collect the set of response sentences associated with each template sentence (Line 13-16). These sentences are then clustered using k-median clustering method (il­lustrated in Algorithm 2) in Line 22.</p><p>In Algorithm 2, we illustrate the method of clus­tering we use to summarize the set of sentences we have collected for each template sentence after the alignment step. The algorithm is similar to the k-means algorithm (Duda et al., 2001), however, given that we are clustering strings instead of real num­bers (as is typical in applications of k-means), we re­strict the centroid of a cluster to be one of the mem­bers of the set being clustered, hence the name k-median algorithm (Martnez-Hinarejos et al., 2003). The distance function to measure the closeness of two strings is instantiated to be an <i>n</i>-gram overlap between the two strings.<footnote anchor="1"/></p><p>The algorithm iterates over three steps until the data is partitioned into <i>k </i>clusters (Line 5). The first step (Lines 6-10) is the initialization of a centroid for a new cluster. Initially when the data is not parti­tioned into any cluster, the median string of the data set is used as the initial centroid. For subsequent iterations, the farthest point from all the centroids computed thus far is used as the centroid for the new cluster. In the second step (Lines 11-16), each mem­ber of the data set is assigned to the nearest clus­ter based on its distance to that cluster's centroid. Finally, in the third step (Lines 17-20), the cluster centroids are recomputed based on the new cluster memberships. Steps two and three are repeated until there are no changes in the cluster memberships and cluster centroids. This completes the introduction of a new cluster for the data.</p><p>For the purposes of our task, we use up to a four-gram overlap to measure distance between two strings and use <i>k </i><i>= 5 </i>for clustering the data.</p><subsection number="4.1" title="Visualizing the HotSpots"><p>The HotSpots page was created within the ECAT tool to surgically dissect the way in which templates were being morphed. For a given template, as shown in Figure 2, the analyst is presented with a copy of the texts from the current and previous versions of that template. Each sentence in the two versions of the template are color coded to show how fre­quently the agents have changed that sentence. This involved running the HotSpots algorithm against ap­proximately 1,000 emails per template version. A sentence that is colored red is one that was changed in over 50% of the emails that were responded to using that template. An orange sentence is one that was edited in between 30% and 50%, green is be­tween 10% to 30% and blue is between 0% and 10%. The more often a sentence is edited the 'hotter' the color.</p><p>The analyst can see the typical substitutions for a sentence by hovering the mouse over that sentence. The typical sentences computed as the centroids of the clusters created using Algorithm 2 are them­selves color coded using the same identification sys-<page local="7" global="270"/></p><footnote label="1">We have also experimented with a symmetric version of Levenshtein distance, but we prefer the n-gram overlap score due to its linear run time complexity.</footnote><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">269</doubt><p>Figure 2: Example of two versions of a template and the edit score (Avg. Morph. Score) and centroids associated with each sentence of the template.</p><p>tem. A typical sentence that occurred in over 50% of the emails is colored red. A typical sentence that occurred in 30% to 50% of the emails was orange and so on.</p><p>In seeing the two versions side by side, the an­alyst can visually inspect the agents' edits on the current version of a template relative to the previ­ous version. If the previous version of the template is a 'hotter' document (with more red sentences), it means that the changes made to the template by the author had led to less editing by agents thus speeding up the process of creating a customer response. If the current template looks hotter, it suggests that the changes made to the template were increasing the agents' edits and probably the email handling time.</p></subsection></section><section number="5" title="Automatic Extraction of Potential Templates"><p>The goal of the template author is to minimize the number of edits done to a template and thus in­directly lowering the handling time for an email. In the preceding section, we discussed a tool that aids the template authors to identify sentences where changes are most often made by agents to a tem­plate. This information could be used by the template authors to create a new version of the template that achieve the goal.</p><p>In this section, we investigate a technique that au­tomatically identifies a possible template with the potential of directly minimizing the average han­dling time for an email. We use the set of responses created by the agents using a given template and se­lect one of the responses to be generalized and stored as a new template. The response to be converted into a template is chosen so as to directly minimize the average handling time. In essence, we seek to partition the set of responses <i>R </i>generated from tem­plate <i>T</i><i> </i>into two clusters <i>R1</i><i> </i>and R2. These clus­ters have centroids <i>T </i>(current template) and <i>T' </i>(new template) such that constraint shown in 1 holds.</p><doubt alpha="55.2" length="29" tooSmall="False" monospace="0.0">(VreRlAHT(T, r) &lt; AHT(T',r))A</doubt><doubt alpha="48.3" length="29" tooSmall="False" monospace="0.0">(Vreß2AHT(T',r)&lt; AHT(T,r))(1)</doubt><p>Now, the quantity <i>AHT(T, </i><i>r)</i><i> </i>is logged as part of the email management system and corresponds to the time taken to respond to a customer's email.<footnote anchor="2"/><page local="8" global="271"/></p><footnote label="2">Although typically this time includes the time to look up a</footnote><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">270</doubt><p>However, we do not have access to <i>AHT(T', r) </i>for any <i>T' = T. </i>We propose a model to estimate this in the next section.</p><subsection number="5.1" title="Modeling Average Handling Time"><p>We model AHT as a linear combination of sev­eral factors which we believe would influence the handling time for an email. These factors in­clude the length in words of the customer's input email <i>(inplen), </i>the length in words of the template <i>(templatelen), </i>the length in words of the response <i>(resplen), </i>the total number of edits between the template and the response <i>(edit), </i>the normalized edit score <i>(nedit), </i>the number of individual events of the edit distance - substitution <i>(sub), </i>insertion (ins), deletion <i>(del) </i>and identity <i>(id), </i>the number of block (contiguous) substitution <i>(blksub), </i>block insertion <i>(blkins) </i>and block deletion <i>(blkdel). </i>Using these in­dependent variables, we fit a linear regression model using the AHT values for 6175 responses created from one particular template (say G). The result of the regression fit is shown in Equation 2 and the data and error statistics are shown in Table 2. It must be noted that the coefficients for the variables are not necessarily reflective of the importance of the vari­ables, since they compensate for the different ranges in variable values. We have also tried several differ-</p><p>the customer's account etc., we assume that time is quite similar for all responses created from the same template.</p><p>ent regression fits with fewer variables, but find that this fit gives us the best correlation with the data.</p><doubt alpha="31.6" length="196" tooSmall="False" monospace="0.0">AHT  =  0.5314 * inplen - 2.7648 * templatelen +1.9982 * resplen - 0.5822 * edit +2900.5242 * nedit +4.7499 * id - 1.6647 * del -1.6021 * ins + 26.6704 * blksub -15.239 * blkins + 24.3931 * blkdel</doubt><doubt alpha="0.0" length="12" tooSmall="False" monospace="0.0">-261.6627(2)</doubt><p>Table2: Datastatistics andthegoodness oftheregression model for 6175 AHT data points.</p><p>Based on the goodness statistics of the regression fit, it is clear the AHT model could be improved further. However, we acknowledge that AHT does not depend solely on the editing of a template to a response but involves several other components in­cluding the user interface, the complexity of cus­tomer's email, the database retrieval to access the customer's account and so forth.<page local="9" global="272"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">271</doubt><table caption="Table 1: Result of clustering responses using the AHT model as the distance metric." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Cluster</p></td><td class="cell"><p>Number of members</p></td><td class="cell"><p>Centroid (Template/Response)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p>1799</p></td><td class="cell"><p>GREETINGSPHR, Thank you for your recent email. On behalf of the company, I would like to extend my sincere apology for the problems you encountered when (XXX over key with appropriate response XXX). It is our goal to provide excellent customer service, and I am sorry that we did not meet that objective. Your input is very valuable, and we will take your feedback into consideration. Regards, Agent.FirstName</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2</p></td><td class="cell"><p>206</p></td><td class="cell"><p>GREETINGSPHR, Thank you for letting me know that you've been unable to send an online order to upgrade your NAMEDENTITY service. Please accept my apologies for any problems this issue may have caused you. You're a highly valued customer. I understand your concerns and I'll be happy to address them. I am investigating this issue. I have already made a personal commitment to email you tomorrow, with the resolution. Thank you for your patience and for choosing the company. We appreciate your business and continued loyalty. Sincerely, Agent.FirstName</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Mean AHT</p></td><td class="cell"><p>675.74 seconds</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Median AHT</p></td><td class="cell"><p>543 seconds</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Mode AHT</p></td><td class="cell"><p>366 seconds</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Standard Deviation</p></td><td class="cell"><p>487.72 seconds</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Correlation coefficient</p></td><td class="cell"><p>0.3822</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Mean absolute error</p></td><td class="cell"><p>320.2 seconds</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Root mean squared error</p></td><td class="cell"><p>450.64 seconds</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Total Number of Instances</p></td><td class="cell"><p>6175</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Nevertheless, we use this model to cluster a new set of 2005 responses originating from the same template <i>(G), </i>as shown in Equation 1. Using the <i>k</i>-median clustering as described earlier, we parti­tion the responses into two clusters. We restrict the first cluster centroid to be the template and search for the best centroid for the second cluster. The re­sults are shown in Table 1. The centroid for clus­ter 1 with 1799 members is the template itself while the centroid for cluster 2 with 206 members is a re­sponse that could be suitably generalized to serve as a template. The overall AHT for the 2005 responses using the template was 989.2 seconds, while the av­erage AHT for the members of cluster 1 and 2 was 971.9 seconds and 1140 seconds, indicating that the template had to be edited considerably to create the members of cluster 2.</p></subsection></section><section number="6" title="Discussion"><p>For the purposes of this paper, it is assumed that AHT is the same as or correlates well with the time to compose a response for an email. However, in most cases the email care agent might have to per­form several verification, validation, and problem resolution phases by consulting the specifics of a customer account before formulating and compos­ing a response. The time taken for each of these phases typically varies depending on the customer's account and the problem category. Nevertheless, we assume that the times for these phases is mostly a constant for a given problem category, and hence the results presented in this paper need to be interpreted on a per problem category basis.</p><p>A second limitation of the approach presented in this paper is that the metric used to measure the sim­ilarity between strings <i>(n</i>-gram overlap) is only a crude approximation of an ideal semantic similarity metric. There are however other similarity metrics (e.g. BLEU (Papineni et al., 2002)) which could be used equally well. The purpose of this paper is to il­lustrate the possibility of analysis of responses using one particular instantiation of the similarity metric.</p><p>In spite of the several directions that this work can be improved, the system and algorithms described in this paper have been deployed in an operational customer care center. The qualitative feedback we have received are extremely positive and analysts have greatly improved the efficiency of the opera­tion using this tool.</p></section><section number="7" title="Conclusions"><p>In this paper, we have presented two approaches that help template authors in designing effective tem­plates for email customer care agents. In the first ap­proach, we have presented details of a graphical tool that provides vital feedback to the template authors on how their templates are being modified by agents when creating responses. The template authors can accommodate this information when designing the next version of the template. We also presented a novel technique for identifying responses that can potentially serve as templates and reduce AHT. To­wards this end, we discussed a method to model AHT based on the characteristics of the customer's email, the template text and the response text.</p></section><section number="8" title="Acknowledgments"><p>We would like to thanks Mazin Gilbert, Junlan Feng, Narendra Gupta and Wenling Hsu for the discus­sions during the course of this work. We also thank the members who generously offered to their sup­port to provide us with data used in this study with­out which this work would not have been possible. We thank the anonymous reviewers for their useful suggestions in improving the quality of this paper.</p></section><references><p>S.K. Card, J. Mackinlay, and B. Shneiderman. 1999. <i>Readings in Information Visualization: Using Vision to Think. </i>Morgan Kaufmann.</p><p>J. Coch. 1996. Evaluating and comparing three text-production techniques. In <i>Proceedings of Coling-96, </i>pages 249-254, Copenhagen, Denmark.</p><p>R.O. Duda, P.E. Hart, and D.G. Stork. 2001. <i>Pattern Classification. </i>Wiley, New York.</p><p>G. Foster, P. Isabelle, and P. Plamondon. 1997. Tar­get text mediated interactive machine translation. <i>Ma­chine Translation, </i>12(1):175-194.</p><p>G. Foster, P. Langlais, and G. Lampalme. 2002. User-friendly text prediction for translators. In <i>EMNLP-02, </i>pages 46-51, Philadelphia, USA.</p><p>Genesys. 2008. http://www.genesys.com. Genesys Cor­poration.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">272</doubt><page local="10" global="273"/><p>Kana. 2008. http://www.kana.com. Kana Corporation.</p><p>G. Lapalme and L. Kosseim. 2003. Mercure: Towards an automatic e-mail follow-up system. <i>IEEE Compu­tational Intelligence Bulletin, </i>2(1):14-18.</p><p>V.I. Levenshtein. 1966. Binary codes capable of correct­ing deletions, insertion and reversals. <i>Soviet Physics Doklady, </i>10:707-710.</p><p>C.D. Martnez-Hinarejos, A. Juan, and F. Casacuberta. 2003. Generalized k-medians clustering for strings. <i>Lecture Notes in Computer Science, </i>2652/2003:502­509.</p><p>F.J. Och, R. Zens, and H. Ney. 2003. Efficient search for interactive statistical machine translation. In <i>EACL-</i>03.</p><p>K. Papineni, S. Roukos, T. Ward, and W.J. Zhu. 2002.</p><p>Bleu: A method for automatic evaluation of machine translation. In <i>Proceedings of 40th Annual Meeting ofthe Association ofComputational Linguistics, </i>pages 313-318, Philadelphia, PA, July. I. Zukerman and Y. Marom. 2007. Evaluation of a large-scale email response system. In <i>Proceedings </i><i>ofIJ-</i><i>CAI07, </i>Hyderabad, India.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">273</doubt></references></body></article>