<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="87"/><title>Handling Noisy Queries in Cross Language FAQ Retrieval</title><pubinfo>Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing,pages 87-96, MIT, Massachusetts, USA, 9-11 October 2010. ©2010 Association for Computational Linguistics</pubinfo><author surname="Contractor" givenname="Danish"><org  name="IBM Research India" country="India" city="New Delhi"/></author><author surname="Kothari" givenname="Govind"><org  name="IBM Research India" country="India" city="New Delhi"/></author><author surname="Faruquie" givenname="Tanveer"><org  name="IBM Research India" country="India" city="New Delhi"/></author><author surname="Subramaniam" givenname="L V"><org  name="IBM Research India" country="India" city="New Delhi"/></author><author surname="Negi" givenname="Sumit"><org  name="IBM Research India" country="India" city="New Delhi"/></author></firstpageheader><frontmatter><p><b>Handling Noisy Queries In Cross Language FAQ Retrieval</b></p><p><b>Danish Contractor   Govind Kothari   Tanveer A. Faruquie L. Venkata Subramaniam   Sumit Negi</b></p><p>IBM Research India Vasant Kunj, Institutional Area New Delhi, India {dcontrac,govkotha,ftanveer,lvsubram,sumitneg}@in.ibm.com</p></frontmatter><abstract>Recent times have seen a tremendous growth in mobile based data services that allow peo­ple to use Short Message Service (SMS) to access these data services. In a multilin­gual society it is essential that data services that were developed for a specific language be made accessible through other local lan­guages also. In this paper, we present a ser­vice that allows a user to query a Frequently-Asked-Questions (FAQ) database built in a lo­cal language (Hindi) using Noisy SMS En­glish queries. The inherent noise in the SMS queries, along with the language mismatch makes this a challenging problem. We handle these two problems by formulating the query similarity over FAQ questions as a combina­torial search problem where the search space consists of combinations of dictionary varia­tions of the noisy query and its top-N transla­tions. We demonstrate the effectiveness of our approach on a real-life dataset. </abstract></header><body><section number="1" title="Introduction"><p>There has been a tremendous growth in the number of new mobile subscribers in the recent past. Most of these new subscribers are from developing coun­tries where mobile is the primary information de­vice. Even for users familiar with computers and the internet, the mobile provides unmatched portability. This has encouraged the proliferation of informa­tion services built around SMS technology. Several applications, traditionally available on Internet, are now being made available on mobile devices using SMS. Examples include SMS short code services.</p><p>Short codes are numbers where a short message in a predesignated format can be sent to get specific information. For example, to get the closing stock price of a particular share, the user has to send a message IBMSTOCKPR. Other examples are search (Schusteritsch et al., 2005), access to Yellow Page services (Kopparapu et al., 2007), Email<footnote anchor="1"/>, Blog <footnote anchor="2"/> , FAQ retrieval <footnote anchor="3"/> etc. The SMS-based FAQ retrieval services use human experts to answer SMS ques­tions.</p><p>Recent studies have shown that instant messag­ing is emerging as the preferred mode of commu­nication after speech and email.<footnote anchor="4"/> Millions of users of instant messaging (IM) services and short mes­sage service (SMS) generate electronic content in a dialect that does not adhere to conventional gram­mar, punctuation and spelling standards. Words are intentionally compressed by non-standard spellings, abbreviations and phonetic transliteration are used. Typical question answering systems are built for use with languages which are free from such errors. It is difficult to build an automated question answer­ing system around SMS technology. This is true even for questions whose answers are well docu­mented like in a Frequently-Asked-Questions (FAQ) database. Unlike other automatic question answer­ing systems that focus on searching answers from a given text collection, Q&amp;A archive (Xue et al., 2008) or the Web (Jijkoun et al., 2005), in a FAQ database the questions and answers are already pro-<page local="2" global="88"/></p><footnote label="1"> http://www.sms2email.com/  2 http://www.letmeparty.com/  3 http://www.chacha.com/  4 http://www.whyconverge.com/</footnote><p>hwz typhoid fvr z sprd ivht du u mean bi srvce tndstry is rgstra at PKRO offce ncsry?</p><p>Figure 1: Sample SMS queries with Hindi FAQs vided by an expert. The main task is then to iden­tify the best matching question to retrieve the rel­evant answer (Sneiders, 1999) (Song et al., 2007). The high level of noise in SMS queries makes this a difficult problem (Kothari et al., 2009). In a multi­lingual setting this problem is even more formidable. Natural language FAQ services built for users in one language cannot be accessed in another language. In this paper we present a FAQ-based question an­swering system over a SMS interface that solves this problem for two languages. We allow the FAQ to be in one language and the SMS query to be in another.</p><p>Multi-lingual question answering and information retrieval has been studied in the past (Sekine and Grishman, 2003)(Cimiano et al., 2009). Such sys­tems resort to machine translation so that the search can be performed over a single language space. In the two language setting, it involves building a ma­chine translation system engine and using it such that the question answering system built for a sin­gle language can be used.</p><p>Typical statistical machine translation systems use large parallel corpora to learn the translation probabilities (Brown et al., 2007). Traditionally such corpora have consisted of news articles and other well written articles. Since the translation sys­tems are not trained on SMS language they perform very poorly when translating noisy SMS language. Parallel corpora comprising noisy sentences in one language and clean sentences in another language are not available and it would be hard to build such large parallel corpora to train a machine translation system. There exists some work to remove noise 2008) (Aw et al., 2006) (Neef et al., 2007) (Kobus et al., 2008). However, all of these techniques re­quire an aligned corpus of SMS and conventional language for training. Such data is extremely hard to create. Unsupervised techniques require huge amounts of SMS data to learn mappings of non­standard words to their corresponding conventional form (Acharyya et al., 2009).</p><doubt alpha="59.6" length="47" tooSmall="False" monospace="0.0">from SMS (Choudhury et al., 2007) (Byun et al.,</doubt><p>Removal of noise from SMS without the use of parallel data has been studied but the methods used are highly dependent on the language model and the degree of noise present in the SMS (Contractor et al., 2010). These systems are not very effective if the SMSes contain grammatical errors (or the sys­tem would require large amounts of training data in the language model to be able to deal with all pos­sible types of noise) in addition to misspellings etc. Thus, the translation of a cleaned SMS, into a second language, will not be very accurate and it would not give good results if such a translated SMS is used to query an FAQ collection.</p><p>Token based noise-correction techniques (such as those using edit-distance, LCS etc) cannot be di­rectly applied to handle the noise present in the SMS query. These noise-correction methods return a list of candidate terms for a given noisy token (E.g. 'gud' — &gt; 'god','good','guide' ) . Considering all these candidate terms and their corresponding trans­lations drastically increase the search space for any multi-lingual IR system. Also , naively replacing the noisy token in the SMS query with the top matching candidate term gives poor performance as shown by our experiments. Our algorithm handles these and related issues in an efficient manner.</p><p>In this paper we address the challenges arising when building a cross language FAQ-based ques­tion answering system over an SMS interface. Our method handles noisy representation of questions in a source language to retrieve answers across target languages. The proposed method does not require hand corrected data or an aligned corpus for explicit SMS normalization to mitigate the effects of noise. It also works well with grammatical noise. To the best of our knowledge we are the first to address issues in noisy SMS based cross-language FAQ re­trieval. We propose an efficient algorithm that can handle noise in the form of lexical and semantic cor­ruptions in the source language.</p></section><section number="2" title="Problem formulation"><p>Consider an input SMS Se in a source language e. We view Se as a sequence of n tokens Se = <i>s1, </i>s2,..., sn. As explained in the introduction, the input is bound to have misspellings and other lexical and semantic distortions. Also let <i>Qh </i>denote the set of questions in the FAQ corpus of a target language <i>h.</i><page local="3" global="89"/><i> </i>Each question <i>Qh </i><i>G Qh </i>is also viewed as a se­quence of tokens. We want to find the question Qhfrom the corpus <i>Qh </i>that best matches the SMS <i>Se.</i></p><p>The matching is assisted by a source dictionary <i>Ve </i>consisting of clean terms in e constructed from a general English dictionary and a domain dictio­nary of target language Vh built from all the terms appearing in Qh. For a token <i>si</i><i> </i>in the SMS in­put, term <i>te </i>in dictionary Ve and term <i>th </i>in dictio­nary Vh we define a <i>cross-lingual</i><i> </i><i>similarity</i><i> </i><i>mea­</i><i></i><i>sure</i><i> a(th, te,si) </i>that measures the extent to which term si matches <i>th </i>using the clean term <i>te. </i>We con­sider <i>th </i>a <i>cross</i><i> </i><i>lingual</i><i> </i><i>variant</i><i> </i>of si if for any <i>te </i>the cross language similarity measure <i>a(th, te, </i>si) &gt; e. We denote this as <i>th</i><i> </i>~ si.</p><p>We define a weight function <i>uo(th, te, </i>si) using the <i>cross</i><i> </i><i>lingual</i><i> </i><i>similarity</i><i> </i><i>measure</i><i> </i>and the <i>inverse</i><i> </i><i>doc­</i><i></i><i>ument</i><i> </i><i>frequency</i><i> </i>(idf) of <i>th</i><i> </i>in the target language FAQ corpus. We also define a scoring function to as­sign a score to each question in the corpus <i>Qh </i>using the weight function. Consider a question Qh G Qh. For each token si, the scoring function chooses the term from Qh having the maximum weight using possible clean representations of si; then the weight of the <i>n </i>chosen terms are summed up to get the score. The score measures how closely the question in FAQ matches the noisy SMS string Se using the composite weights of individual tokens.</p><doubt alpha="66.7" length="27" tooSmall="False" monospace="0.0">Score(Qh) = maxuj(th,te,si)</doubt><p>Our goal is to efficiently find the question Qh having the maximum score.</p></section><section number="3" title="Noise removal from queries"><p>In order to process the noisy SMS input we first have to map noisy tokens in Se to the possible correct lex­ical representations. We use a similarity measure to map the noisy tokens to their clean lexical represen­tations.</p><subsection number="3.1" title="Similarity Measure"><p>For a term <i>te</i><i> </i>G Ve and token si of the SMS input Se, the similarity measure <i>7(te, </i>si) between them is</p><p><i>LCSRatio(te,si) EditDistance </i><i>SMS </i>(te,si) <i>if te and s</i><i>i </i><i>share same starting character *</i> <i>otherwise</i></p><doubt alpha="25.0" length="12" tooSmall="False" monospace="0.0">7(*e,si) = &lt;</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(1)</doubt><p>Where <i>LCSRatio(t£,si) = </i>'"^igfhfc)<i>)) </i>and <i>LCS(te,si) </i>is the <i>Longest common subsequence </i>between <i>te </i>and si. <i>* The intuition behind this measure is that people typically type the first few characters of a word in an SMS correctly. This way we limit the possible variants for a particular noisy token</i></p><p>The <i>Longest</i><i> </i><i>Common</i><i> </i><i>Subsequence</i><i> </i><i>Ratio</i><i> </i>(LC-SRatio) (Melamed et al., 1999) of two strings is the ratio of the length of their LCS and the length of the longer string. Since in the SMS scenario, the dictio­nary term will always be longer than the SMS token, the denominator of LCSRatio is taken as the length of the dictionary term.</p><p>The <i>EditDistance</i><i>SMS</i><i> </i>(Figure 2) compares the Consonant Skeletons (Prochasson et al., 2007) of the dictionary term and the SMS token. If the Leven-shtein distance between consonant skeletons is small then 7<i>(te, s</i><i>i)</i><i> </i>will be high. The intuition behind us­ing <i>EditDistanceSMS </i>can be explained through an example. Consider an SMS token "gud" whose most likely correct form is "good". The longest common subsequence for "good" and "guided" with "gud" is "gd". Hence the two dictionary terms "good" and "guided" have the same LCSRatio of 0.5 w.r.t "gud", but the <i>EditDistanceSMS </i>of "good" is 1 which is less than that of "guided", which has <i>EditDistanceSMS </i>of 2 w.r.t "gud". As a result the similarity measure between "gud" and "good" will be higher than that of "gud" and "guided". Higher the <i>LCSRatio</i><i> </i>and lower the <i>EditDistanceSMS</i>, higher will be the similarity measure. Hence, for a given SMS token "byk", the similarity measure of word "bike" is higher than that of "break".</p></subsection></section><section number="4" title="Cross lingual similarity"><p>Once we have potential candidates which are the likely disambiguated representations of the noisy <b>Procedure </b><i>EditDistance</i><i>SMS</i><i>(te, s</i><i>i)</i><i> </i><b>Begin</b><page local="4" global="90"/></p><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">0</doubt><p><b>return </b><i>LevenshteinDistance(CS(si), CS(te)) + 1 </i><b>End</b> <b>Procedure </b><i>CS (t):</i><i> //Consonant Skeleton Generation </i><b>Begin</b></p><p>Step 1. remove consecutive repeated characters in t</p><p>// <i>(fall — fal) </i>Step 2. remove all vowels in t</p><p><i>//(painting — pntng, threat — thrt) </i><b>return </b><i>t</i></p><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">End</doubt><p>Figure 2: <i>EditDistanceSMS</i> term, we map these candidates to appropriate terms in the target language. We use a statistical dictionary to achieve this cross lingual mapping.</p><subsection number="4.1" title="Statistical Dictionary"><p>In order to build a statistical dictionary we use the statistical translation model proposed in (Brown et al., 2007). Under IBM model 2 the transla­tion probability of source language sentence ë = <i>{t\,... </i><i>,t?e,...</i><i> ,t</i><i>f<footnote anchor="1"/>}</i><i> </i>and a target language sentence h = <i>{t</i><i>h</i><i>,...,t</i><i>h</i><i>,...,t</i><i>le} </i>is given by <i>Pr(h</i>|ë) = 0(l|m) II E <i>t</i><i>(t</i><i>h</i><i>K</i><i>)a(j</i>|i, <i>m, l).</i><i></i></p><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">l m</doubt><doubt alpha="50.0" length="6" tooSmall="False" monospace="0.0">i=lj=0</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(2)</doubt><p>Here the word translation model <i>t</i><i>(th\te) </i>gives the probability of translating the source term to target term and the alignment model <i>a(j\i, m, l) </i>gives the probability of translating the source term at position <i>i </i>to a target position <i>j. </i>This model is learnt using an aligned parallel corpus.</p><p>Given a clean term <i>t\</i><i> </i>in source language we get all the corresponding terms <i>T</i><i> </i><i>=</i><i> </i><i>{t</i><i>h,</i><i>... ,t</i><i>h,</i><i>...</i><i>}</i><i> </i>from the target language such that word translation probability <i>t</i><i>(t</i><i>hl</i><i>t</i><i>l)</i><i> </i><i>&gt; e. </i>We rank these terms ac­cording to the probability given by the word trans­lation model <i>t</i><i>(th\te) </i>and consider only those tar­get terms that are part of domain dictionary i.e.</p><p><i>tkh </i>G Vh.</p></subsection><subsection number="4.2" title="Cross lingual similarity measure"><p>For each term <i>si</i><i> </i>in SMS input query, we find all the clean terms <i>t</i><i>e </i>in source dictionary Ve for which similarity measure <i>7(te, s</i><i>i)</i><i> </i><i>&gt; </i><i>&lt;\</i><i>). </i>For each of these term te, we find the cross lingual similar terms Tteusing the word translation model. We compute the cross lingual similarity measure between these terms as</p><doubt alpha="50.0" length="32" tooSmall="False" monospace="0.0">a(si,te,th)=7(te,si).T(th,te)(3)</doubt><p>The measure selects those terms in target lan­guage that have high probability of being translated from a noisy term through one or more valid clean terms.</p></subsection><subsection number="4.3" title="Cross lingual similarity weight"><p>We combine the idf and the cross lingual similarity measure to define the cross lingual weight function <i>L</i><i>ü</i><i>(t</i><i>h</i><i>,t</i><i>e</i><i>,s</i><i>i)</i><i> </i>as</p><doubt alpha="54.3" length="35" tooSmall="False" monospace="0.0">U(th,te,si)=a(th,te,si).idf (th)(4)</doubt><p>By using idf we give preference to terms that are highly discriminative. This is necessary because queries are distinguished from each other using in­formative words. For example for a given noisy token "bck" if a word translation model produces a translation output "wapas" (as in came back) or "peet" or "qamar" (as in back pain) then idf will weigh "peet" more as it is relatively more discrim­inative compared to "wapas" which is used fre­quently.</p></subsection></section><section number="5" title="Pruning and matching"><p>In this section we describe our search algorithm and the preprocessing needed to find the best question <i>Q</i><i>h </i>for a given SMS query.</p><subsection number="5.1" title="Indexing"><p>Our algorithm operates at a token level and its corre­sponding cross lingual variants. It is therefore nec­essary to be able to retrieve all questions <i>Qthh</i><i> </i>that contain a given target language term <i>t</i><i>h.</i><i> </i>To do this efficiently we index the questions in FAQ corpus us­ing <i>Lucene<footnote anchor="5"/>.</i><i> </i>Each question in FAQ is treated as a document. It is tokenized using whitespace as de­limiter before indexing.</p><footnote label="5">http://lucene.apache.org/java/docs/</footnote><page local="5" global="91"/><p>The cross lingual similarity weight calculation re­quires the <i>idf</i>for a given term <i>t</i><i>h.</i><i> </i>We query on this index to determine the number of documents <i>f </i>that contain <i>t</i><i>h.</i><i> </i>The <i>idf</i>of each term in Vh is precom-puted and stored in a hashtable with <i>t</i><i>h </i>as the key. The cross lingual similarity measure calculation re­quires the word translation probability for a given term <i>t</i><i>e.</i><i> </i>For every <i>t</i><i>e</i><i> </i>in dictionary <i>De, </i>we store Tte in a hashmap that contains a list of terms in the target language along with their statistically deter­mined translation probability <i>t</i><i>(th\te) &gt; e, </i>where <i>t</i><i>h </i>G Vh.</p><p>Since the query and the FAQs use terms from dif­ferent languages, the computation of IDF becomes a challenge (Pirkola, 1998) (Oard et al., 2007). Prior work uses a bilingual dictionary for translations for calculating the IDF. We on the other hand rely on a statistical dictionary that has translation probabil­ities. Applying the method suggested in the prior work on a statistical dictionary leads to errors as the translations may themselves be inaccurate.</p><p>We therefore calculate IDFs for target language term (translation) and use it in the weight measure calculation. The method suggested by Oard et al (Oard et al., 2007) is more useful in retrieval tasks for multiple documents, while in our case we need to retrieve a specific document (FAQ).</p></subsection><subsection number="5.2" title="List Creation"><p>Given an SMS input string <i>Se, </i>we tokenize it on white space and replace any occurrence of digits to their string based form (e.g. 4get, 2day) to get a se­ries of <i>n </i>tokens <i>s\,s2,... ,sn. </i>A list <i>Le </i>is created for each token <i>s</i><i>i </i>using terms in the monolingual dic­tionary <i>De. </i>The list for a single character SMS to­ken is set to <i>null</i><i> </i>as it is most likely to be a stop word. A term <i>te </i>from <i>De </i>is included in <i>Le </i>if it satisfies the threshold condition</p><doubt alpha="30.8" length="13" tooSmall="False" monospace="0.0">7(te,si)&gt;&lt;(5)</doubt><p>The threshold value <i>&lt; </i>is determined experimen­tally. For every <i>te </i><i>G</i><i> </i><i>Le </i>we retrieve <i>Tte</i><i> </i>and then retrieve the <i>idf</i><i> </i>scores for every <i>th </i><i>G</i><i> </i><i>Tte</i>. Using the word translation probabilities and the idf score we compute the cross lingual similarity weight to create a new list <i>Lih.</i><i> </i>A term <i>t</i><i>h </i>is included in the list only</p><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">if</doubt><doubt alpha="37.5" length="16" tooSmall="False" monospace="0.0">T(thlte)&gt;0.1 (6)</doubt><p>This probability cut-off is used to prevent poor quality translations from being included in the list.</p><p>If more than one term <i>t</i><i>e </i>has the same transla­tion <i>t</i><i>h,</i><i> </i>then <i>t</i><i>h</i><i> </i>can occur more than once in a given list. If this happens, then we remove repetitive oc­currences of <i>t</i><i>h </i>and assign it a weight equal to the maximum weight amongst all occurrences in the list, multiplied by the number of times it occurs. The terms <i>t</i><i>h </i>in <i>Lih</i><i> </i>are sorted in decreasing order of their similarity weights. Henceforth, the term "list" im­plies a sorted list.</p><p>For example given a SMS query "hw mch ds it cst to stdy in india" as shown in Fig. 3, for each token we create a list of possible correct dictionary words by dictionary look up. Thus for token "cst" we get dictionary words lik "cost, cast, case, close". For each dictionary word we get a set of possible words in Hindi by looking at statistical translation table. Finally we merged the list obtained to get single list of Hindi words. The final list is ranked according to their similarity weights.</p></subsection><subsection number="5.3" title="Search algorithm"><p>Given <i>Se </i>containing <i>n </i>tokens, we create <i>n </i>sorted lists <i>Lh, Lh,..., L</i><i>n </i>containing terms from the do­main dictionary and sorted according to their cross lingual weights as explained in the previous section. A naive approach would be to query the index using each term appearing in all <i>Lih</i><i> </i>to build a <i>Collection</i><i> </i><i>set</i><i> </i><i>C </i>of questions. The best matching question <i>Q</i><i>h</i>will be contained in this collection. We compute the score of each question in C using <i>Score(Q) </i>and the question with highest score is treated as <i>Q</i><i>h</i>. How­ever the naive approach suffers from high runtime cost.</p><p>Inspired by the Threshold Algorithm (Fagin et al., 2001) we propose using a pruning algorithm that maintains a much smaller <i>candidate</i><i> </i><i>set</i><i> </i><i>C </i>of questions that can potentially contain the maximum scoring question. The algorithm is shown in Fig­ure 4. The algorithm works in an iterative manner. In each iteration, it picks the term that has maxi­mum weight among all the terms appearing in the lists <i>Lh, Lh,..., </i>As the lists are sorted in the descending order of the weights, this amounts to picking the maximum weight term amongst the first terms of the <i>n </i>lists. The chosen term <i>t</i><i>h </i>is queried to find the set <i>Qt,</i>. The set <i>Qt, </i>is added to the candidate set C.<page local="6" global="92"/> For each question <i>Q G Qth</i>, we compute its score <i>Score(Q) </i>and keep it along with Q. After this the chosen term <i>t</i><i>h </i>is removed from the list and the next iteration is carried out. We stop the iterative process when a thresholding condition is met and fo­cus only on the questions in the candidate set <i>C. </i>The thresholding condition guarantees that the candidate set C contains the maximum scoring question <i>Q</i><i>h</i><i>. </i>Next we develop this thresholding condition.</p><figure caption="Figure 3: List creation"></figure><p>Let us consider the end of an iteration. Sup­pose <i>Q </i>is a question not included in <i>C. </i>At best, <i>Q </i>will include the current top-most tokens Lh[1], <i>L^[1],..., L</i><i>n</i><i>[1] </i>from every list. Thus, the upper bound UB on the score of <i>Q </i>is <i>Score(Q) </i>&lt; ]T<i>u(Lh[1\).</i><i></i></p><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">i=0</doubt><p>Let <i>Q* </i>be the question in C having the maximum score. Notice that if <i>Q* &gt; UB, </i>then it is guaranteed that any question not included in the candidate set <i>C </i>cannot be the maximum scoring question. Thus, the condition "Q* &gt; UB" serves as the termination cri­terion. At the end of each iteration, we check if the termination condition is satisfied and if so, we can stop the iterative process. Then, we simply pick the question in C having the maximum score and return it.</p><p><b>Procedure </b>Search Algorithm <b>Input: </b>SMS <i>S = s</i><i>i,</i><i> </i><i>s</i><i>2</i><i>,...,sn</i><b>Output: </b>Maximum scoring question <i>Qh</i>. <b>Begin</b> <i>Vsi, </i>construct Le for which <i>y </i><i>(s</i><i>i </i><i>,te) &gt; e</i></p><p>// <i>Li </i>lists variants of <i>si</i>Construct lists <i>Lh, Lh,... ,L% </i>//(see Section 5.2).</p><p>// <i>Lih</i><i> </i>lists cross lingual variants of <i>s</i><i>i </i>in decreasing</p><p>//order of weight. Candidate list C <i>= </i>0. <b>repeat</b> // <i>t*h</i><i> </i>is the term having maximum weight among</p><doubt alpha="42.9" length="7" tooSmall="False" monospace="0.0">t* =th=</doubt><p>// all terms appearing in the <i>n </i>lists. Delete <i>t*h</i><i> </i>from the list Lh.. Retrieve <i>Qt*h</i><i> </i>using the index // <i>Qt*h</i><i> </i>: the set of all questions in <i>Qh</i></p><p>//having the term <i>t*h</i><i></i><b>For each </b><i>Q</i><i> </i><i>G</i><i> </i><i>Qt*h</i></p><p>Compute <i>Scove(Q) </i>and add <i>Q </i>with its score into C</p><doubt alpha="43.8" length="16" tooSmall="False" monospace="0.0">VB =T,Li"(Lh[1])</doubt><p>Q <i>= &amp;igmax</i><i>QeC </i><i>Scoie(Q).</i></p><p><b>if </b><i>Score(Q) &gt; UB, </i><b>then</b></p><p>// Termination condition satisfied Output <i>Q </i>and exit. <b>forever End</b></p><figure caption="Figure 4: Search Algorithm with Pruning6 Experiments"></figure><p>To evaluate our system we used noisy English SMS queries to query a collection of <i>10, 000 </i>Hindi FAQs. These FAQs were collected from websites of vari­ous government organizations and other online re­sources. These FAQs are related to railway reser­vation, railway enquiry, passport application and health related issues. For our experiments we asked 6 human evaluators, proficient in both English and Hindi, to create English SMS queries based on the general topics that our FAQ collection dealt with. We found 60 SMS queries created by the evaluators, had answers in our FAQ collection and we desig­nated these as the in-domain queries. To measure the effectiveness of our system in handling out of domain queries we used a total of <i>380 </i>SMSes part of which were taken from the NUS corpus (How et al., whch metro statn z nr pragati maidan ?<page local="7" global="93"/> dus metro goes frm airpot 2 new delhi rlway statn? is dere any special metro pas 4 delhi uni students? whn is d last train of delhi metro? whr r d auto stands N delhi?</p><figure caption="Figure 5: Sample SMS queries"></figure><p>2005) and the rest from the "out-of-domain" queries created by the human evaluators. Thus the total SMS query data size was 440. Fig 5 shows some of the sample queries.</p><p>Our objective was to retrieve the correct Hindi FAQ response given a noisy English SMS query. A given English SMS query was matched against the list of indexed FAQs and the best matching FAQ was returned by the Pruning Algorithm described in Sec­tion 5. A score of <i>1 </i>was assigned if the retrieved answer was indeed the response to the posed SMS query else we assigned a score of 0. In case of out of domain queries a score of <i>1 </i>was assigned if the output was NULL else we assigned a score of <i>0.</i></p></subsection><subsection number="6.1" title="Translation System"><p>We used the Moses toolkit (Koehn et al., 2007) to build an English-Hindi statistical machine transla­tion system. The system was trained on a collec­tion of <i>150, 000 </i>English and Hindi parallel sentences sourced from a publishing house. The <i>150, 000 </i>sen­tences were on a varied range of subjects such as news, literature, history etc. Apart from this the training data also contained an aligned parallel cor­pus of English and Hindi FAQs. The FAQs were collected from government websites on topics such as health, education, travel services etc.</p><p>Since an MT system trained solely on a collection of sentences would not be very accurate in translat­ing questions, we trained the system on an English-Hindi parallel question corpus. As it was difficult to find a large collection of parallel text consisting of questions, we created a small collection of par­allel questions using <i>240 </i>FAQs and multiplied them to create a parallel corpus of <i>50, 000 </i>sentences. This set was added to the training data and this helped fa­miliarize the language model and phrase tables used by the MT systems to questions. Thus in total the</p><p>MT system was trained on a corpus of <i>200, 000 </i>sen­tences.</p><p>Experiment 1 and 2 form the baseline against which we evaluated our system. For our experi­ments the lexical translation probabilities generated by Moses toolkit were used to build the word trans­lation model. In Experiment 1 the threshold <i>0 </i>de­scribed in Equation 5 is set to 1. In Experiment 2 and 3 this is set to <i>0. 5. </i>The Hindi FAQ collection was indexed using Lucene and a domain dictionary <i>Dh </i>was created from the Hindi words in the FAQ collection.</p></subsection><subsection number="6.2" title="System Evaluation"><p>We perform three sets of experiments to show how each stage of the algorithm contributes in improving the overall results.</p><subsubsection number="6.2.1" title="Experiment 1"><p>For Experiment <i>1 </i>the threshold <i>0 </i>in Equation 5 is set to <i>1 </i>i.e. we consider only those tokens in the query which belong to the dictionary. This setup il­lustrates the case when no noise handling is done. The results are reported in Figure 6.</p></subsubsection><subsubsection number="6.2.2" title="Experiment 2"><p>For Experiment 2 the noisy SMS query was cleaned using the following approach. Given a noisy token in the SMS query it's similarity (Equation 1) with each word in the Dictionary is calculated. The noisy token is replaced with the Dictionary word with the maximum similarity score. This gives us a clean English query.</p><p>For each token in the cleaned English SMS query, we create a list of possible Hindi translations of the token using the statistical translation table. Each Hindi word was assigned a weight according to Equation 4. The Pruning algorithm in Section 5 was then applied to get the best matching FAQ.</p></subsubsection><subsubsection number="6.2.3" title="Experiment 3"><p>In this experiment, for each token in the noisy En­glish SMS we obtain a list of possible English vari­ations. For each English variation a corresponding set of Hindi words from the statistical translation ta­ble was obtained. Each Hindi word was assigned a weight according to Equation 4. As described in Section 5.2, all Hindi words obtained from English variations of a given SMS token are merged to create a list of Hindi words sorted in terms of their weight.<page local="8" global="94"/> The Pruning algorithm as described in Section 5 was then applied to get the best matching FAQ.</p><p>We evaluated our system using two different cri­teria. We used MRR (Mean reciprocal rank) and the best matching accuracy. Mean reciprocal rank is used to evaluate a system by producing a list of possible responses to a query, ordered by probabil­ity of correctness. The reciprocal rank of a query response is the multiplicative inverse of the rank of the first correct answer. The mean reciprocal rank is the average of the reciprocal ranks of results for a sample of queries Q.</p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">Q</doubt><doubt alpha="55.0" length="20" tooSmall="False" monospace="0.0">MRR = 1/\Q\YJ1/ranki</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(7)</doubt><p>Best match accuracy can be considered as a spe­cial case of MRR where the size of the ranked list is 1. As the SMS based FAQ retrieval system will be used via mobile phones where screen size is a ma­jor constraint it is crucial to have the correct result on the top. Hence in our settings the best match ac­curacy is a more relevant and stricter performance evaluation measure than MRR.</p><p>Table 1 compares the MRR scores for all three experiments. Our method reports the highest MRR of 0.83. Figure 6 shows the performance using the strict evaluation criterion of the top result returned being correct.</p><p>We also experimented with different values of the threshold for Score(Q) (Section 5.3). The ROC curve for various threshold is shown in Figure 7. The result for both in-domain and out-of-domain queries for the three experiments are shown in Figure 6 for Score(Q) = 8. The F1 Score for experiments 1, <i>2 </i>and <i>3 </i>are shown in Table 2.</p><figure caption="Figure 6: Comparison of results"></figure><figure caption="Figure 7: ROC Curve forScore(Q)"></figure></subsubsection></subsection><subsection number="6.3" title="Measuring noise level in SMS queries"><p>In order to quantify the level of noise in the col­lected SMS data, we built a character-level language model(LM) using the questions in the FAQ data-set (vocabulary size is 70) and computed the perplexity of the language model on the noisy and the cleaned SMS test-set. The perplexity of the LM on a cor­pus gives an indication of the average number of bits needed per n-gram to encode the corpus. Noise results in the introduction of many previously unseen n-grams in the corpus.<page local="9" global="95"/> Higher number of bits are needed to encode these improbable n-grams which results in increased perplexity. From Table 3 we can see the difference in perplexity for noisy and clean SMS data for the English FAQ data-set. Large per­plexity values for the SMS dataset indicates a high level of noise.</p><table caption="Table 1: MRR Scores" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Experiment 1</p></td><td class="cell"><p>Experiment 2</p></td><td class="cell"><p>Experiment 3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MRR Score</p></td><td class="cell"><p>0.41</p></td><td class="cell"><p>0.68</p></td><td class="cell"><p>0.83</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 2: F1 Measure" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>F1 Score</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Expt 1 (Baseline 1)</p></td><td class="cell"><p>0.23</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Expt 2 (Baseline 2)</p></td><td class="cell"><p>0.68</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Expt 3 (Proposed Method)</p></td><td class="cell"><p>0.72</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 3: Perplexity for Cleaned and Noisy SMS" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Cleaned SMS</p></td><td class="cell"><p>Noisy SMS</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>English FAQ collection</p></td><td class="cell"><p>bigram trigram</p></td><td class="cell"><p>16.64 9.75</p></td><td class="cell"><p>55.19 69.41</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>For each noisy SMS query e.g. "hw 2 prvnt ty-phd" we manually created a clean SMS query "how to prevent typhoid". A character level language model using the questions in the clean English FAQ dataset was created to quantify the level of noise in our SMS dataset. We computed the perplexity ofthe language model on clean and noisy SMS queries.</p></subsection></section><section number="7" title="Conclusion"><p>There has been a tremendous increase in information access services using SMS based interfaces. How­ever, these services are limited to a single language and fail to scale for multilingual QA needs. The ability to query a FAQ database in a language other than the one for which it was developed is of great practical significance in multilingual societies. Au­tomatic cross-lingual QA over SMS is challenging because of inherent noise in the query and the lack of cross language resources for noisy processing. In this paper we present a cross-language FAQ retrieval system that handles the inherent noise in source lan­guage to retrieve FAQs in a target language. Our sys­tem does not require an end-to-end machine transla­tion system and can be implemented using a sim­ple dictionary which can be static or constructed statistically using a moderate sized parallel corpus. This side steps the problem of building full fledged translation systems but still enabling the system to be scaled across multiple languages quickly. We present an efficient algorithm to search and match the best question in the large FAQ corpus of tar­get language for a noisy input question. We have demonstrated the effectiveness of our approach on a real life FAQ corpus.</p></section><references><p>Sreangsu Acharyya, Sumit Negi, L Venkata Subrama-niam, Shourya Roy.  2009.  Language independent unsupervised learning of short message service di­alect. <i>International Journal on Document Analysis and Recognition, </i>pp. 175-184.</p><p>Aiti Aw, Min Zhang, Juan Xiao, and Jian Su. 2006. A phrase-based statistical model for SMS text normaliza­tion. <i>In Proceedings of COLING-ACL, </i>pp. 33-40.</p><p>Peter F. Brown, Vincent J.Della Pietra, Stephen A. Della Pietra, Robert. L. Mercer 1993. The Mathematics of Statistical Machine Translation: Parameter Estimation <i>Computational Linguistics, </i>pp. 263-311.</p><p>Jeunghyun Byun, Seung-Wook Lee, Young-In Song,</p><p>Hae-Chang Rim. 2008. Two Phase Model for SMS</p><p>Text Messages Refinement. <i>AAAI Workshop on En­hanced Messaging.</i></p><p>Monojit Choudhury, Rahul Saraf, Vijit Jain, Animesh Mukherjee, Sudeshna Sarkar, Anupam Basu. 2007. Investigation and modeling of the structure of texting language. <i>International Journal on Document Analy­sis and Recognition, </i>pp. 157-174.</p><p>Philipp Cimiano, Antje Schultz, Sergej Sizov, Philipp Sorg, Steffen Staab. 2009. Explicit versus latent con­cept models for cross-language information retrieval.</p><p><i>In Proceeding ofIJCAI, </i>pp. 1513-1518.</p><p>Danish Contractor, Tanveer A. Faruquie, L. Venkata Sub-ramaniam. 2010. Unsupervised cleansing of noisy text. <i>In Proceeding of COLING 2010: Posters, </i>pp. 189-196.</p><p>R. Fagin, A. Lotem, and M. Naor. 2001. Optimal aggre­gation algorithms for middleware. <i>In Proceedings of</i> <i>the 20th ACM SIGMOD-SIGACT-SIGART symposium</i> <i>on Principles ofdatabase systems, </i>pp. 102-113.</p><p>Yijue How and Min-Yen Kan. 2005. Optimizing pre­dictive text entry for short message service on mobile phones. <i>In M. J. Smith and G. Salvendy (Eds.) Proc. of Human Computer Interfaces International,Lawrence Erlbaum Associates</i></p><p>Valentin Jijkoun and Maarten de Rijke. 2005. Retrieving answers from frequently asked questions pages on the web. <i>In Proceedings of the Tenth ACM Conference on Information and Knowledge Management,CIKM, </i>pp.</p><p>76-83.</p><p>Catherine Kobus, Francois Yvon and Grraldine Damnati. 2008. Normalizing SMS: Are two metaphors better than one? <i>In Proceedings of COLING, </i>pp. 441-448.</p><p>Philipp Koehn, Hieu Hoang, Alexandra Birch Mayne, Christopher Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Chris­tine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, Evan Herbst 2007. Moses: Open source toolkit for statistical machine translation. <i>Annual Meeting </i><i>ofthe</i><i> Association for Computation Linguistics (ACL), Demonstration Session </i>.</p><p>Sunil Kumar Kopparapu, Akhilesh Srivastava and Arun Pande. 2007. SMS based Natural Language Interface to Yellow Pages Directory.<page local="10" global="96"/> <i>In Proceedings of the 4th international conference on mobile technology, appli­cations, and systems and the 1st international sympo­sium on Computer human interaction in mobile tech­nology, </i>pp. 558-563 .</p><p>Govind Kothari, Sumit Negi, Tanveer Faruquie, Venkat Chakravarthy and L V Subramaniam 2009. SMS based Interface for FAQ Retrieval. <i>Annual Meeting ofthe Association for Computation Linguistics (ACL).</i></p><p>I. D. Melamed. 1999. Bitext maps and alignment via pat­tern recognition. <i>Computational Linguistics, </i>pp. 107­130.</p><p>Guimier de Neef, Emilie, Arnaud Debeurme, and</p><p>Jungyeul Park. 2007. TILT correcteur de SMS : Eval­uation et bilan quantitatif. <i>In Actes de TALN, </i>pp. 123­132.</p><p>Douglas W. Oard, Funda Ertunc. 2002. Translation-Based Indexing for Cross-Language Retrieval <i>In Pro­ceedings ofthe ECIR, </i>pp. 324-333.</p><p>A. Pirkola 1998. The Effects of Query Structure and Dictionary Setups in Dictionary-Based Cross-Language Information Retrieval <i>SIGIR '98: Proceed­ings of the 21st Annual International ACM SIGIR Con­ference on Research and Development in Information Retrieval </i>, pp. 55-63.</p><p>E. Prochasson, C. Viard-Gaudin, and E. Morin. 2007. Language models for handwritten short message ser­vices. <i>In Proceedings ofthe 9th International Confer­ence on Document Analysis and Recognition, </i>pp. 83­87.</p><p>Rudy Schusteritsch, Shailendra Rao, Kerry Rodden. 2005. Mobile Search with Text Messages: Designing the User Experience for Google SMS. <i>In Proceedings of ACM SIGCHI, </i>pp. 1777-1780.</p><p>Satoshi Sekine, Ralph Grishman. 2003. Hindi-English cross-lingual question-answering system. <i>ACMTrans-actions on Asian Language Information Processing,</i> pp. 181-192.</p><p>E. Sneiders. 1999. Automated FAQ Answering: Contin­ued Experience with Shallow Language Understand­ing Question Answering Systems. <i>Papers from the 1999 AAAI Fall Symposium. </i>Technical Report FS-99-02, AAAI Press, pp. 97-107.</p><p>W. Song, M. Feng, N. Gu, and L. Wenyin. 2007. Ques­tion similarity calculation for FAQ answering. <i>In Pro­ceeding ofSKG 07, </i>pp. 298-301.</p><doubt alpha="63.0" length="54" tooSmall="False" monospace="0.0">X. Xue, J. Jeon, and W.B Croft. 2008. Retrieval Models</doubt><p>for Question and Answer Archives. <i>In Proceedings of</i> <i>SIGIR, </i>pp. 475-482.</p></references></body></article>