<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="410"/><title>Japanese Abbreviation Expansion with Query and Clickthrough Logs</title><pubinfo>Proceedings of the 5th International Joint Conference on Natural Language Processing,pages 410-419, Chiang Mai, Thailand, November 8 - 13, 2011. ©2011 AFNLP</pubinfo><author surname="Uchiumi" givenname="Kei"><org  name="Nara Institute of Science and Technology" country="Japan" city="Nara"/></author><author surname="Komachi" givenname="Mamoru"><org  name="Nara Institute of Science and Technology" country="Japan" city="Nara"/></author><author surname="Machinaga" givenname="Keigo"><org  name="Nara Institute of Science and Technology" country="Japan" city="Nara"/></author><author surname="Maezawa" givenname="Toshiyuki"><org  name="Nara Institute of Science and Technology" country="Japan" city="Nara"/></author><author surname="Satou" givenname="Toshinori"><org  name="Nara Institute of Science and Technology" country="Japan" city="Nara"/></author><author surname="Kobayashi" givenname="Yoshinori"><org  name="Nara Institute of Science and Technology" country="Japan" city="Nara"/></author></firstpageheader><frontmatter><p><b>Japanese Abbreviation Expansion with Query and Clickthrough Logs </b><b>Kei Uchiumi<footnote anchor="1"/>" Mamoru Komachi* Keigo Machinaga<footnote anchor="1"/>"</b></p><p><b>Toshiyuki Maezawa<footnote anchor="1"/>" Toshinori Satou<footnote anchor="1"/>" Yoshinori Kobayashi<footnote anchor="1"/>" *</b></p><p>| <b>Yahoo Japan Corporation</b></p><p>Midtown Tower, 9-7-1 Akasaka, Minato-ku, Tokyo 107-6211, Japan {kuchiumi,kmachina,tmaezawa,toshsato}@yahoo-corp.jp</p><p>ykobayas@google.com</p><p>J<b>Nara Institute of Science and Technology</b></p><p>8916-5 Takayama, Ikoma, Nara 630-0192, Japan komachi@is.naist.jp</p></frontmatter><abstract>A novel reranking method has been de­veloped to refine web search queries. A label propagation algorithm was applied on a clickthrough graph, and the candi­dates were reranked using a query lan­guage model. Our method first enumer­ates query candidates with common land­ing pages with regard to the given query to create a clickthrough graph. Second, it calculates the likelihood of the candidates, using a language model generated from web search query logs. Finally, the can­didates are sorted by the score calculated from the likelihood and label propagation. As a result, high precision and coverage were achieved in the task of Japanese ab­breviation expansion, without using hand­crafted training data. </abstract></header><body><section number="1" title="Introduction"><p>The query expansion technique has been widely used in recent web-search engines. Query expan­sion significantly improves recall in information retrieval operations. It uses a thesaurus or syn­onym dictionary to reformulate a query, or to cor­rect spelling errors in search queries.</p><p>In the early days of the speller, the dictionary was manually compiled by lexicographers. How­ever, it is time consuming to construct a broad coverage dictionary, and domain knowledge is re­quired to achieve high quality. Moreover, the rapid growth of the web makes it even harder to main­tain an up-to-date dictionary for the web.</p><p>*The work of Kobayashi were performed at Yahoo Japan. Current affiliation: Google Japan, Roppongi Hills Mori Tower, 6-10-1 Roppongi, Minato-ku, Tokyo 106-6126, Japan</p><p>To alleviate this problem, web-search engines often exploit web search query logs to automati­cally generate a thesaurus. A web search query is a query that a web user types into a web search en­gine to find information. It is noisy and sometimes ambiguous to detect query intent, but it is a great way to create a fresh web dictionary at low cost. Hence, the web search queries are widely used in the NLP field. For instance, Hagiwara and Suzuki (2009) used them for a query alteration task, and Sekine and Suzuki (2007) leveraged them for ac­quiring semantic categories.</p><p>More recently, web search clickthrough logs have been explored in the field of lexical acquisi­tion. A web clickthrough is the process of clicking a URL and going to the page it refers. This en­sures that the landing page is appropriate since the web user follows the hyperlink after checking the information displayed, such as 'title', 'URL', and 'summary' of their search. Two distinct queries landing on the same 'URL' are possibly input for the same purpose, meaning that they are likely to be related. In the NLP literature, clickthrough logs have been used to learn semantic categories (Ko-machi et al., 2009) and named entities (Jain and Pennacchiotti, 2010).</p><p>The main contribution of this work is two fold:</p><p>• We propose a novel method to combine web search query logs and clickthrough logs.</p><p>• To the best our knowledge, this work is the first attempt to automatically recognize full spellings given Japanese abbreviations.</p><p>This is a very first step of Japanese abbreviation expansion task using search logs.</p><p>For evaluation of query expansion method, it is desirable to use a set of queries for evaluation.</p><page local="2" global="411"/><p>However, it is difficult to obtain them beforehand, because we have to check query logs to find incor­rect queries and make necessary changes to define their corrections.</p><p>Therefore, in this paper, we focus on query ab­breviation and evaluate our proposed approach in an abbreviation expansion task. Abbreviation ex­pansion itself is difficult for many query expan­sion methods based on edit distance, because the input and output have only a few, if any, charac­ters in common. Our clickthroughlog-based ap­proach can expand even queries that do not share any characters at all with the abbreviated ones <footnote anchor="1"/>. Since our method does not rely on any language, it is applicable to any other languages including Chinese and English.</p><p>The rest of this paper is organized as follows. Section 2 describes previous works in query ex­pansion tasks. In Section 3, we formulate a query expansion task in a noisy channel model frame­work. In Section 4, we show that label propagation on a clickthrough graph can be used as a query ab­breviation model and extract candidates for query correction without preparing correct candidates. Section 5 explains the query language model we use. In Section 6, we evaluate our method in an ab­breviation expansion task and show its efficiency. Section 7 offers conclusions and directions for fu­ture work.</p></section><section number="2" title="Related Work"><p>Query expansion for a web-search query has to handle neologisms and slang on the web. Thus, it is labor-intensive to maintain a list of correctly spelled words for search queries. Additionally, Japanese query expansion includes several tasks, such as word segmentation, word stemming, and acronym expansion. Much of the previous work has focused on each individual task (Ahmad and Kondrak, 2005; Chen et al., 2007; Bergsma and Wang, 2007; Li et al., 2006; Peng et al., 2007;</p><p>Risvik et al., 2003).</p><p>Cucerzan and Brill (2004) clarified problems of spelling correction for search queries, address­ing them using a noisy channel model with a lan­guage model created from query logs. Gao et al. (2010) and Sun et al. (2010) applied a reranking method applying neural net to the search-query spelling correction candidates obtained from the</p><footnote label="1">Note that our method can be applied to query expansion as well.</footnote><p>Cucerzan's method. Their reranking method had the advantageous ability to incorporate click-through logs to a translation model learned as a ranking-feature. However, their methods are based on edit distance, and thus they did not deal with the task of synonym replacement and acronym expan­sion.</p><p>Wei et al. (2009) addressed synonym extraction using similarity based on Jensen-Shannon diver­gence of commonly clicked URL distribution be­tween queries. Their approach is similar to our proposed method, except that they did not use a language model. Also, their method is not scalable and cannot be applied to our task using large-scale data.</p><p>Jain and Pennacchiotti (2010) proposed an un-supervised method for named entity extraction from web search query logs. They performed a clustering method using a combination offeatures based on query logs, web documents, and click-through logs. They showed that clickthrough logs give higher accuracy than query logs as a corpus.</p><p>Guo et al. (2008) proposed a unified approach for query expansion using a discriminative model. They extended feature function of CRFs (Laf-ferty et al., 2001) by adding 'operation' to the triplet variables: 'feature', 'label', and 'operation'. 'Operation' represents a process for query expan­sion. For example, 'operation' can take four states ('deletion', 'insertion', 'substitution', and 'trans­position') on spelling correction. However, their method needs supervised data for training and can­not deal with a word that does not occur in the corpus. In fact, they used only 10,000 queries to learn the query expansion model. Unlike their method, our approach takes advantage of an enor­mous amount of clickthrough logs for learning the query abbreviation model.</p><p>Query suggestion is another task that uses search logs (Mei et al., 2008; Cao et al., 2008).</p><p>Query suggestion differs from our task in that it allows queries to be suggested that are different from the one that the search user types.</p><p>Furthermore, some previous works have ad­dressed acquiring a Japanese abbreviation task. Murayama and Okumura (2008) formulated the process of generating Japanese abbreviations by noisy channel model but they did not handle ab­breviation expansion. Okazaki et al. (2008) dealt with recognizing Japanese abbreviation tasks as a binary classification problem.  They extracted<page local="3" global="412"/></p></section><section number="1" title="Offline part"><doubt alpha="90.9" length="11" tooSmall="True" monospace="0.0">Online part</doubt><figure caption="Figure 1: Combining clickthrough logs and search logs for query abbreviation expansion"></figure><p>pairs of words from the newspaper corpus us­ing a heuristic and then classified them as "ab­breviation" or "not-abbreviation". However, their heuristic for obtaining abbreviation candidates cannot be applied to web search queries.</p></section><section number="3" title="Noisy Channel Model for Abbreviation Expansion"><p>In this section, we explain our noisy channel based approach to query expansion. We define the query expansion problem as follows: Given a user's query q and a set of search logs L, find a cor­rect query c G C that is most relevant to the in­put <i>q. </i>In a probabilistic framework, this can be formulated as finding the argmax P(c|q). Apply­ing Bayes' Rule and dropping the constant de­nominator, we obtain a unnormalized posterior: argmax P(c)P(q|c)(Eq.1). We now have a noisy channel model for query expansion, with two com­ponents: the source model P (c) and the channel model P(q|c).</p><doubt alpha="52.6" length="19" tooSmall="False" monospace="0.0">c* = argmax P (c|q)</doubt><doubt alpha="63.2" length="38" tooSmall="False" monospace="0.0">P (c)P (q|c)argmaXP(q)argmaxP(c)P(q|c)</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(1)</doubt><p>We use a language model estimated from search query logs as the source model, thus P(c) repre­sents likelihood of c as a query. As for the chan­nel model, we use a label propagation method on a clickthrough graph as proposed by Komachi et al. (2009). Figure 1 shows the framework of our approach.</p><p>To find candidates to the input query, we con­struct a bipartite graph from a query and a clicked</p><p>URL using the web search logs. We calculate the relatedness between the queries on this graph to select a set of candidates C. Since the label prop­agation is mathematically identical to the random walk with restart, probability ofthe label propaga­tion can be regarded as the conditional probability P(q|c). If we assume that the relatedness score represents the conditional probability of the typed query q given a candidate c G C, P(q|c), the c* is calculated by argmax P (c) x P (q|c). As a conse­quence, we propose reranking in accordance with the follow equation using two probabilistic mod­els <i>Pqlm </i>and PLP and then output ranked can­didates. In this paper, we will define PLP inter­changeably as a query abbreviation model, <i>Pqam </i>.</p><doubt alpha="54.3" length="35" tooSmall="False" monospace="0.0">score(q, c) =Pqlm(c) xPqam(q|c) (2)</doubt><p>An advantage of our proposed method is that it can correct a query by only using search logs with­out a manually labeled-corpora or any heuristics. Our approach is a versatile framework for query expansion and thus is not specialized for any tasks. We explain the label propagation algorithm on a clickthrough graph and the query language model below.</p></section><section number="4" title="Query Abbreviation Model from Clickthrough Logs"><p>In this section, we describe a label propagation al­gorithm on a clickthrough graph. It is based on a previous work by Komachi et al. (2009). The main difference between their method and ours is that we use the normalized pointwise mutual in­formation and the 1-step approximation ofa click-through graph.</p><p>Graph-based semi-supervised methods such as label propagation can performance well with only a few seeds and scale up to a large dataset. Figure 2 illustrates the process of label propagation using a seed term "abc".</p><p>This is a bipartite graph whose left-hand side nodes are terms and right-hand side nodes are patterns. Starting from "abc", the label prop­agates to other term nodes through the pat­tern "http://abcnews.go.com" that is strongly con­nected to "abc" and thus the label "abc" will be propagated to "american broadcasting corpora­tion".</p><p>In this way, label propagation gradually propa­gates the label of the seed instance to neighboring<page local="4" global="413"/></p><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">abc</doubt><doubt alpha="92.3" length="13" tooSmall="True" monospace="0.0">alphabet song</doubt><p>Figure 2: An illustrative example of Instance-Pattern co-occurrence graph and label propagation process.</p><p>The strength of lines indicates relatedness between each node, whereas the depth of the color of nodes represents relatedness to the seed. The darker a left-hand side node, the more likely it is similar to "abc". The darker a right-hand side node, the more likely it is the characteristic pattern of "abc".</p><p>nodes, and optimal labels are given as the labels at which the label propagation process has con­verged.</p><p>However, the seed instance like that in Figure 2 possibly causes a result to be worse in a task of lexical acquisition, due to an ambiguous instance "abc", which belongs to more than one domain, e.g. "mass media" and "dance". It is expected that the label propagates to unrelated instances if we have highly frequent ambiguous nodes. This problem is called "semantic drift" and has received a lot of attention in NLP research (Komachi et al., 2008).</p><p>Komachi et al. (2008) have reported that boot­strapping algorithms like Espresso (Pantel and Pennacchiotti, 2006) can be viewed as Kleinberg's</p><p>HITS algorithm (Kleinberg, 1999) and the "se­mantic drift" problem on the graph is the same phenomenon as "topic drift" in HITS, which con­verges to the eigenvector of the instance-instance similarity graph created from instance-pattern co­occurrence graph as described in the next subsec­tion.</p><p>Our label propagation method based on Ko-machi et al. (2009) can be used as a relatedness measure that returns a similarity score relative to the seed instance, and thus is suitable for a query correction task.</p><doubt alpha="71.4" length="7" tooSmall="False" monospace="0.0">Input :</doubt><p>Seed instance vector F<i>(0)</i></p><p>Instance similarity matrix A Output : Instance score vector F(t) 1: Construct the normalized Laplacian matrix 2: Iterate</p><doubt alpha="26.3" length="19" tooSmall="False" monospace="0.0">L = I - D-1/2AD-1/2</doubt><doubt alpha="48.0" length="50" tooSmall="False" monospace="0.0">'F(t+ 1) =a(-L)F(t) + (1 -a)F(0) until convergence</doubt><figure caption="Figure 3: Laplacian label propagation"></figure><subsection number="4.1" title="One-step approximation of clickthrough graph"><p>In this paper, we extract queries landing on the same URL as the one related with input query by stopping label propagation after 1-hop. These queries are possibly synonyms with the input query and thus possible to correct without seman­tic transformation.</p><figure caption="Figure 3 shows the label propagation algorithm on a clickthrough graph."></figure><p>Given an instance set X = {xt<i>x\</i>, <i>x%+txn } </i>and a label set L = {1,c}, the first l instances xi (i &lt; l) are labeled as yi G L. The goal is to predict the labels of the unlabeled instances xu (l + 1 &lt; <i>u &lt; </i>n).</p><p>Let <i>F </i>denote the set of n x c matrices with non-negative entries. A matrix <i>F </i>= [Fi,..., Fn]T G F corresponds to a classification on the dataset X by labeling each instance Xj as a label y = <i>argmax</i><i>j</i><i>&lt;cF</i><i>ij</i>. Define Fo as the initial F with <i>F</i><i>ij </i>= 1 if xi is labeled as a label <i>yi </i>= <i>j </i>and Fij = 0 otherwise. The (i, j)-th element of the fi­nal matrix F represents a similarity to the labeled instances. We use these similarities as P(q|c) in Equation 2, where q is a seed instance, c is a la­beled instance by label propagation.</p><p>The instance-instance similarity matrix <i>A </i>in Figure 3 is defined as A = <i>W</i><i>T</i><i>W </i>where <i>W </i>is an instance-pattern matrix. The (i, j)-th element of W<i>ij </i>contains the relative frequency of occurrence of instance x<i>i </i>and pattern p<i>j.</i></p><p>D is a diagonal degree matrix of A where the ( i, j) -th element of D is given as D<i>ii</i><i> </i>= <i>j </i>A<i>ij. </i>Label propagation has a parameter <i>a </i>(0 &lt; <i>a </i>&lt; A-i, where A is a principal eigenvalue of normal­ized Laplacian matrix L) that controls the effect of clamping the label distribution of labeled data.</p><p>http ://www. abc-tokyo.com/<page local="5" global="414"/></p><footnote>http://abcnews.go.com/</footnote><doubt alpha="54.4" length="79" tooSmall="False" monospace="0.0">Ç^r.—/■■http://www.alphabetsong.org/.............."'K^)http://en.wikipedia.org/</doubt><doubt alpha="100.0" length="8" tooSmall="True" monospace="0.0">american</doubt><doubt alpha="100.0" length="12" tooSmall="True" monospace="0.0">broadcasting</doubt><doubt alpha="100.0" length="11" tooSmall="True" monospace="0.0">corporation</doubt><doubt alpha="93.3" length="15" tooSmall="True" monospace="0.0">austrian ballet</doubt><doubt alpha="100.0" length="7" tooSmall="True" monospace="0.0">company</doubt></subsection><subsection number="4.2" title="Normalized PMI"><p>Komachi et al. (2009) suggested that the normal­ized frequency causes semantic drift (Jurafsky and Martin, 2009), and we confirmed this phenomenon in our preliminary experiment. They suggested us­ing relative frequency such as pointwise mutual in­formation (PMI) and log-likelihood ratio as coun-termeasure against semantic drift. Therefore, we used pointwise mutual information (PMI) shown below to handle the aforementioned semantic drift problem.</p><doubt alpha="44.4" length="9" tooSmall="False" monospace="0.0">PM1(x,p)=</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(3)</doubt><p>PMI assigns high scores to low-frequency events. Moreover, using PMI naively makes sparse matrix W dense. Therefore, we used nor­malized PMI (NPMI) (Bouma, 2009) below as the relative frequency and cutoffthe values lowerthan a threshold 9 (9 &gt; 0).</p><doubt alpha="60.0" length="10" tooSmall="False" monospace="0.0">NPMI (x,p)</doubt><doubt alpha="50.0" length="10" tooSmall="False" monospace="0.0">—InP (x,p)</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">Wij</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">0</doubt><doubt alpha="59.3" length="27" tooSmall="False" monospace="0.0">(NPMI (xi,pj) (NPMI (xi,pj)</doubt><doubt alpha="0.0" length="9" tooSmall="False" monospace="0.0">, (9 &gt; 0)</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(4)</doubt><doubt alpha="0.0" length="8" tooSmall="False" monospace="0.0">&lt; 9) (5)</doubt><p>In the web search, neologims appear continu­ously, which make it hard to compute the like­lihood of queries by a word n-gram language model. Moreover, characters themselves carry essential semantic information in Chinese and Japanese. Therefore, we build a character lan­guage model for the search query logs follow­ing observations of the usefulness of character n-grams for Japanese (Asahara and Matsumoto, 2004) and Chinese (Huang and Zhao, 2006). Asa­hara and Matsumoto used a window of two char­acters to the right and to the left ofthe focus char­acter, which results in using character 5-grams. We also used 5-grams for a query language model from the preliminary experiment.</p></subsection></section><section number="6" title="Experiment"><subsection number="6.1" title="Test Set"><p>We collected abbreviations of <i>'Acronym', 'Kanji', 'Kana' </i>from the Japanese version of Wikipedia, and then removed single letters and duplications. Finally, we gathered 1,916 terms and used them in our evaluation.</p></subsection><subsection number="6.2" title="Construction of a Clickthrough Graph"><p>We used queries and clicked links in Japanese clickthrough logs as instances and patterns, re­spectively. We tallied them in the below condi­tions.</p><p>NPMI prevents low-frequency events from be­ing assigned scores that are too high by dividing by —InP(x,p) and heads off excess label propa­gation through them. By cutting off negative val­ues, the range of Wij can be normalized to [0,1]. Additionally, this prevents sparse matrix W from being dense and reduces the noise in the data.</p></subsection></section><section number="5" title="Query Language Model"><p>In this paper, we use a character n-gram language model to obtain the likelihood of the candidates for query expansion in Equation 2.</p><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">N-i</doubt><doubt alpha="45.2" length="31" tooSmall="False" monospace="0.0">P (c) =IIP (xi|xi-N+i,...,xi-i)</doubt><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">i=0</doubt><doubt alpha="54.8" length="42" tooSmall="False" monospace="0.0">Wfreq(xi-N+i,...,xi)Hfreq(xi-N+i,...,xi-i)</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(6)</doubt><p>where consider c is a contiguous sequences of N characters c = {x0, xi,..., xn-i} .</p><p>1. Query and clickthrough are unique with re­spect to each cookie each day.</p><p>If a user input the same query and clicked the same URL any number of times, we do not count it as occurring multiple times, i.e. we do not increase the number of clickthrough.</p><p>2. Alphanumeric characters in a query are uni­fied to one-byte lower-case characters</p><p>3. A sequence of white space in a query is uni­fied to single one-byte white space character</p><p>4. All the URLs included in clickthrough logs are unique, i.e., we did not generalize URLs as Tseng et al. (2009) did.</p><p>The Japanese clickthrough logs were collected from October 22 to November 9, 2009 <footnote anchor="2"/> and from January 1 to 16 in Yahoo Japan web search logs.</p><footnote label="2">A storage device in our experimental environment be­came full when tallying clickthrough logs. As a result, we were not able to use clickthrough logs ofsome periods.</footnote><doubt alpha="37.5" length="8" tooSmall="False" monospace="0.0">P (x,p)_</doubt><doubt alpha="71.4" length="7" tooSmall="False" monospace="0.0">VnP(x)P</doubt><footnote label="5">(p)</footnote><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">//</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">N</doubt><page local="6" global="415"/><p>Links clicked less than 10 times were removed for efficiency reasons. Finally, we obtained 4,428,430 nodes, 16,841,683 patterns, and 16,988,516 edges.</p><p>The threshold 9 of elements Wij was set to 0.1 on the basis of preliminary experimental results.</p><p>The parameter a for label propagation was set to 0.0001.</p><subsection number="6.3" title="Construction of Query Language Model"><p>We used web search query logs for constructing a language model. The search query logs were col­lected from August 1, 2009, to January 27, 2010, in Yahoo Japan web search logs. We removed queries that occurred fewer than 10 times. Finally, we obtained 52,399,621 unique queries as a train­ing corpus.</p><p>In this experiment, we constructed a character 5-gram language model using the query logs, all normalized by the length of the candidate's string.</p></subsection><subsection number="6.4" title="Evaluation"><p>The system output was shown to five search eval­uation specialists. We evaluated all systems using <i>precision </i>and <i>coverage at k. </i>Coverage is defined as the percentage of queries for which the system returned at least one relevant query. Precision at k is the number of relevant queries amongst the top k returned. They are computed as follows:</p><p># of correct output at rank <i>k</i></p><doubt alpha="45.0" length="20" tooSmall="False" monospace="0.0">precision =-——----—-</doubt><p>Number of output at rank <i>k</i> # of queries for which the system gives at least one correct output</p><doubt alpha="36.4" length="22" tooSmall="False" monospace="0.0">coverage = ----———--.-</doubt><p>Number of all input queries</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(7)</doubt><p>In our experiment, the average number ofcandi-dates for each query is about 53. Therefore, we ex­tracted 50 candidates from clickthrough logs and then reranked using three methods:</p><subsubsection number="6.4.1" title="Judgment Guideline"><p>We describe our guidelines to judge system out­puts below. We defined four correction patterns for abbreviation expansion:</p></subsubsection></subsection></section><section number="1." title="Ranking using abbreviation model (AM) only"></section><section number="2." title="Ranking using language model (LM) only"><p>3. Ranking using both language model and ab­breviation model.</p><p>Micro average of edit distance between input abbreviations and its correct expansions is 4.03, while the average length of queries is 3.01. These statistics show that input queries should be re­placed by totally different characters and it is diffi­cult to use edit distance for extracting correct can­didates from web search logs. This is another rea­son clickthrough logs are essential to the query ab­breviation task.</p></section><section number="1." title="acronym for its English expansion"></section><section number="2." title="acronym for its Japanese orthography 4"></section><section number="3." title="Japanese abbreviation for its Japanese or­thography"></section><section number="4." title="Japanese abbreviation for its English orthog­raphy"><p>We collected abbreviation/expansion pairs if and only if they were one of these three types: (1) named entity, (2) common expression, (3) Japanese meaning of the common expression.</p><table caption="Table 1 shows examples of each correction pat­tern along with its output type."></table><p>Ambiguous cases were discarded in the study as exceptions after discussion with experts. To calculate the agreement rate, system outputs for a hundred randomly sampled queries from test set were evaluated by two judges. The agreement rate of judgment of abbreviation/expansion pair is 47.0 percentage and Cohen's kappa measure k = 0.63. Thus, it is considered as an upper bound of the system, and the abbreviation expansion is not con­sidered to be a trivial task.</p><subsection number="6.5" title="Experimental Results"><p>Table 2 shows <i>precision at k </i>and coverage for three systems with <i>k </i>ranging from 1 to 50. Ta­ble 3 shows examples of inputs and outputs. The baseline without reranking is shown at the bottom line (k=50). The result of using only QAM in Ta­ble 2 is equivalent to the method of Komachi et al. (2009) using NPMI instead of raw frequency as elements of an instance-pattern matrix. To<page local="7" global="416"/></p><footnote label="3">Underlined words are correct.</footnote><footnote label="4">Some corrections were dealt with as exceptions. For ex­ample, acronym for its Japanese Hiragana was treated as in­correct, but acronym for its Japanese meaning was treated as correct.</footnote><table caption="Table 1: Abbreviations and its correction"></table><doubt alpha="100.0" length="5" tooSmall="False" monospace="0.0">Input</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">"WW</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">mm</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">vod</doubt><doubt alpha="83.3" length="6" tooSmall="False" monospace="0.0">ilo pr</doubt><p>Candidates</p><doubt alpha="25.0" length="8" tooSmall="False" monospace="0.0">7.79e-i0</doubt><p>our knowledge, their algorithm is the state-of-the-art algorithm in acquiring synonyms using web search logs.</p><p>The proposed ranking method using a query language model and abbreviation model learned from clickthrough logs shows the best precision and coverage within 1 &lt; k &lt; 10. This is because the language and abbreviation model use different sources of information to complement each other.</p><p>The language model estimates probability ofthe candidate as a query, and it assigns high probabil­ity to candidates that appear frequently in query logs. Those candidates tend to co-occur with many clickthrough patterns, which results in cre­ating generic patterns that may cause semantic drift (Komachi et al., 2009). Because we used NPMI instead of raw frequency, our label prop­agation method assigns high weight to instances connected to a seed instance through a few specific patterns. Consequently, low-frequency instances tend to be ranked in higher positions.</p><p>Table 4 shows the significance level between two baselines and the proposed model. We ap­plied Wilcoxon's signed rank test to compare har­monic mean between precision and coverage of each model with <i>k </i>ranging from 1 to 50. The im­provement of adding QAM to QLM is made sta­tistically significant by the Wilcoxon's signed rank test at level p &lt; 0.00001. Our approach outper­forms the QAM without QLM although not as sig­nificant (p &lt; 0.06). These mean that the ranking of our methods is similar to that of QAM. We con­sider the reason or this result to be that QLM intro­duces more information about queries under this experimental setting because the reranking pro­cess is performed after narrowing candidates down to 50 by QAM, even though we do not use QAM scores at all when evaluating QLM.<page local="8" global="417"/></p><table caption="Table 4: P-values of Wilcoxon's signed rank testIIQAMand QAM + QLM | QLM and QAM + QLM"></table><doubt alpha="46.2" length="13" tooSmall="False" monospace="0.0">p-value 0.055</doubt><table caption="Table 2: Precision and coverage atk" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>abbreviation</p></td><td class="cell"><p>correct candidates (descending order of rank)</p></td><td class="cell"><p>output type</p></td><td class="cell"><p>correction pattern</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>adf</p></td><td class="cell"><p>asian dub foundation</p></td><td class="cell"><p>Named Entity: Organization</p></td><td class="cell"><p>acronym for its expansion</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ana</p></td><td class="cell"><p>^BS,^B*S$IWS3#a (All Nippon Airways)</p></td><td class="cell"><p>Named Entity: Organization</p></td><td class="cell"><p>acronym for its Japanese orthography</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ny</p></td><td class="cell"><p><i>— ^ — 3 — ï </i>(New York)</p></td><td class="cell"><p>Named Entity: Location</p></td><td class="cell"><p>acronym for its Japanese orthography</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>tos</p></td><td class="cell"><p>f^*Xt7-&gt;y7* = 7 (Tales of Symphonia)</p></td><td class="cell"><p>Named Entity: Product</p></td><td class="cell"><p>acronym for its Japanese orthography</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>illustrator</p></td><td class="cell"><p>Named Entity: Product</p></td><td class="cell"><p>Japanese abbreviation for its English orthography</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>A7^—Xh7^^ (Hunger Strike)</p></td><td class="cell"><p>Common expression</p></td><td class="cell"><p>Japanese abbreviation for its Japanese orthograpy</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>R»</p></td><td class="cell"><p></p></td><td class="cell"><p>Named Entity: Organization</p></td><td class="cell"><p>Japanese abbreviation for its Japanese orthograpy</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>fyi</p></td><td class="cell"><p>for your information</p></td><td class="cell"><p>Common expression</p></td><td class="cell"><p>acronym to its expansion</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 3: Examples of input and candidates or its correction3" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>k</i></p></td><td class="cell"><p>query abbreviation model (QAM)</p></td><td class="cell"><p>query language model (QLM)</p></td><td class="cell"><p>QLM+QAM</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>precision</p></td><td class="cell"><p>coverage</p></td><td class="cell"><p>precision</p></td><td class="cell"><p>coverage</p></td><td class="cell"><p>precision</p></td><td class="cell"><p>coverage</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p>0.114</p></td><td class="cell"><p>0.114</p></td><td class="cell"><p>0.157</p></td><td class="cell"><p>0.157</p></td><td class="cell"><p><b>0.161</b></p></td><td class="cell"><p><b>0.161</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3</p></td><td class="cell"><p>0.122</p></td><td class="cell"><p>0.256</p></td><td class="cell"><p>0.142</p></td><td class="cell"><p>0.278</p></td><td class="cell"><p><b>0.157</b></p></td><td class="cell"><p><b>0.321</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>5</p></td><td class="cell"><p>0.121</p></td><td class="cell"><p>0.341</p></td><td class="cell"><p>0.128</p></td><td class="cell"><p>0.346</p></td><td class="cell"><p><b>0.142</b></p></td><td class="cell"><p><b>0.392</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>10</p></td><td class="cell"><p>0.114</p></td><td class="cell"><p>0.453</p></td><td class="cell"><p>0.102</p></td><td class="cell"><p>0.425</p></td><td class="cell"><p><b>0.115</b></p></td><td class="cell"><p><b>0.465</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>30</p></td><td class="cell"><p><b>0.087</b></p></td><td class="cell"><p>0.536</p></td><td class="cell"><p>0.078</p></td><td class="cell"><p>0.529</p></td><td class="cell"><p>0.082</p></td><td class="cell"><p><b>0.542</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>50</p></td><td class="cell"><p>0.073</p></td><td class="cell"><p>0.557</p></td><td class="cell"><p>0.073</p></td><td class="cell"><p>0.557</p></td><td class="cell"><p>0.073</p></td><td class="cell"><p>0.557</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Due to time constraints and human resources for evaluation, we were unable to compare NPMI with raw frequency. There is still much room for improvement for assigning appropriate weights to edges in a clickthrough graph.</p></subsection><subsection number="6.6" title="Error Analysis"><p>We conducted error analysis of our proposed method and found that errors can be divided into three types: (1) a partial correct query, (2) a cor­rect query but with an additional attribute word, and (3) a related but not abbreviated term.</p><p><b>A partial correct query </b>The main reason for this error is that the likelihood of the partial query becomes higher than that of its correct spelling. Although we normalized the likelihood of candi­dates by their string length, we still fail to filter fragments of queries. We consider that this issue can be solved by modeling popularity of candi­dates using PageRank from web search logs. Par­tial correct queries do not co-occur with attribute words frequently, while correct queries co-occur with diverse attribute words. Therefore, PageRank on a query graph whose edges represent common co-occurring words between queries, will assign higher scores to correct queries than a query lan­guage model and abbreviation model.</p><p><b>A correct query but with an additional at­tribute word </b>Examples of this error type in­clude the combination of correct queries and com­monly used attribute words in the search (e.g. <i>"* </i>h <i>a</i><i> </i><i>"(what does *mean?), "* meaning),"* </i><i>fâî^J</i><i>i"</i><i> (how to use *)), </i>etc.). There were 857 queries that were classified as incorrect that co-occurred with these attribute words. The similar­ity of these candidates and input query tend to be higher than that of others because these attribute words frequently appear in search query logs, so the likelihood of these candidate being calculated by a language model tends to be higher too.</p><p>We consider that this issue can be solved at some level by generating a language model us­ing the first term only, after splitting queries sep­arated by a space in search query logs. How­ever, attribute words are not always separated by a space, and sometimes appear as the first term in the query<footnote anchor="5"/>. Another way to handle this problem is to use PageRank described earlier to decrease likelihood ofcandidates including attribute words.</p><footnote label="5">Some attributes, (e.g. "KH "(Movie),"' T — ^\Ani-</footnote><p><b>Related but not abbreviated term </b>A number of abbreviations coincide with other general nouns (e.g. "dog"<footnote anchor="6"/>). It is hard to expand these abbrevia­tions correctly at present. In future work, session logs and geo-location information from IP address and GPS can be used to disambiguate the intent of the query.</p><p>Besides above reasons, 280 out of 1,916 queries did not exist in clickthrough logs, resulting in our system not being able to extract the correct query. To solve this problem, we will increase clickthrough logs to improve the coverage of our corpus.</p></subsection></section><section number="7" title="Conclusion"><p>We have proposed a query expansion method us­ing the web search query and clickthrough logs.</p><p>Our noisy channel based method uses charac­ter 5-gram of query logs as a language model and label propagation on a clickthrough graph as a channel model. In our experiment, we found that a combination of label propagation and language model outperformed other methods using either label propagation or language model in reranking of query abbreviation candidates extracted from the web search clickthrough logs.</p><p>In fact, a modified implementation of this method is currently in production use as an assis­tance tool for making a synonym dictionary at Ya­hoo Japan.</p><p>In evaluation of IR systems, Mizzaro (2008) has proposed a normalized mean average preci­sion (NMAP) for considering difficulty of topics in data sets. However, identifying topics in our test set queries and measuring their difficulty are beyond the scope of this paper. Evaluation criteria are important for making production services.</p><p>In the future, we are going to address this task using discriminative learning as a ranking prob­lem.</p></section><references><p>Farooq Ahmad and Grzegorz Kondrak. 2005. Learn­ing a spelling error model from search query logs. In</p><p>mation),"BÄ"(Picture),etc.), occur often at first token in a search query, but some attributes, (e.g.   "{sÊt/^7Ï", ,"M B$",etc.) almost never occur at first token. <footnote anchor="6"/>DOG: Disk Original Group<page local="9" global="418"/></p><p><i>Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Lan­guage Processing, </i>pages 955-962.</p><p>Masayuki Asahara and Yuji Matsumoto. 2004. Japanese unknown word identification by character-based chunking. In <i>Proceedings of the 20th inter­national conference on Computational Linguistics,</i> pages 459-465.</p><p>Shane Bergsma and Qin Iris Wang. 2007. Learning noun phrase query segmentation. In <i>Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning.</i></p><p>Geolof Bouma. 2009. Normalized (pointwise) mutual information in collocation extraction. In <i>Proceed­ings of the Biennial GSCL Conference, </i>pages 31-40.</p><p>Huanhuan Cao, Daxin Jiang, Jian Pei, Qi He, Zhen Liao, Enhong Chen, and Hang Li. 2008. Context-aware query suggestion by mining click-through and session data. In <i>Proceeding of the 14th ACM SIGKDD international conference on Knowledge Discovery and Data Mining, </i>pages 875-883.</p><p>Qing Chen, Mu Li, and Ming Zhou. 2007. Improving query spelling correction using web search results.</p><p>In <i>Proceedings ofthe 2007 Joint Conference on Empirical Methods in Natural Language Process­ing and Computational Natural Language Learning, </i>pages 181-189.</p><p>Silviu Cucerzan and Eric Brill. 2004. Spelling correc­tion as an iterative process that exploits the collective knowledge of web users. In <i>Proceedings of the Con­ference on Empirical Methods in Natural Language Processing, </i>pages 293-300.</p><p>Jianfeng Gao, Xiaolong Li, Daniel Micol, Chris Quirk, and Xu Sun. 2010. A large scale ranker-based sys­tem for search query spelling correction. In <i>Pro­ceedings of the 23rd International Conference on Computational Linguistics, </i>pages 358-366.</p><p>Jianfeng Guo, Gu Xu, Hang Li, and Xueqi Cheng. 2008. A unified and discriminative model for query refinement. In <i>Proceedings of the 31st annual inter­national ACM SIGIR conference on Research and Development in Information Retrieval, </i>pages 379­386.</p><p>Masato Hagiwara and Hisami Suzuki. 2009. Japanese query alteration based on seamntic similarity. In <i>Proceedings of Human Language Technologies:</i><i> The 2009 Annual Conference of the North American Chapter ofthe Association for Computational Lin­guistics, </i>pages 191-199.</p><p>Chang-Ning Huang and Hai Zhao. 2006. Which is essential for chinese word segmentation: character versus word. In <i>Proceedings of the 20th Pacific Asia Conference on Language, Information and Compu­tation (PACLIC-20), </i>pages 1-12.</p><p>Alpa Jain and Marco Pennacchiotti. 2010. Open entity extraction from web search query logs. In <i>Proceed­ings ofthe 23rd International Conference on Com­putational Linguistics, </i>pages 510-518.</p><p>Daniel Jurafsky and James H. Martin. 2009. <i>Speech and language processing: An introduction to natu­ral language processing, computational linguistics, and speech recognition. </i>2nd edition. Prentice Hall, Englewood Cliffs. NJ.</p><p>Jon M. Kleinberg. 1999. Authoritative sources in a hyperlinked environment. <i>Journal ofthe ACM, </i>46:604-632, September.</p><p>Mamoru Komachi, Taku Kudo, Masashi Shimbo, and Yuji Matsumoto. 2008. Graph-based analysis of semantic drift in Espresso-like bootstrapping algo­rithms. In <i>Proceedings of the 2008 Conference on Empirical Methods in Natural Language Process­ing, </i>pages 1011-1020.</p><p>Mamoru Komachi, Shimpei Makimoto, Kei Uchiumi, and Manabu Sassano. 2009. Learning semantic categories from clickthrough logs. In <i>Proceedings</i> <i>ofthe ACL-IJCNLP 2009 Conference Short Papers,</i> pages 189-192.</p><p>John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Prob­abilistic models for segmenting and labeling se­quence data. In <i>Proceedings ofthe Eighteenth In­ternational Conference on Machine Learning, </i>pages 282-289.</p><p>Mu Li, Yang Zhang, Muhua Zhu, and Ming Zhou. 2006. Exploring distributional similarity based models for query spelling correction. In <i>Proceed­ings ofthe 21st International Conference on Compu­tational Linguistics and 44th Annual Meeting ofthe Association for Computational Linguistics, </i>pages 1025-1032.</p><p>Qiazhu Mei, Dengyong Zhou, and Kenneth Church. 2008. Query suggestion using hitting time. In <i>Pro­ceeding ofthe 17th ACMconference on Information and Knowledge Management, </i>pages 469-478.</p><p>Stefano Mizzaro. 2008. The good, the bad, the dif­ficult, and the easy: something wrong with infor­mation retrieval evaluation? In <i>Proceedings of the IR research, 30th European conference on Advances in information retrieval, </i>ECIR'08, pages 642-646, Berlin, Heidelberg. Springer-Verlag.</p><p>Norihumi Murayama and Manabu Okumura. 2008. Statistical model for Japanese abbreviations. In <i>Pro­ceedings of the 10th Pacific Rim International Con­ference on Artificial Intelligence: Trends in Artificial Intelligence, </i>pages 260-272.</p><p>Naoki Okazaki, Mitsuru Ishizuka, and Jun'ichi Tsujii. 2008. A discriminative approach to Japanese ab­breviation extraction. In <i>Proceedings ofthe 3rd In­ternational Joint Conference on Natural Language Processing (IJCNLP-08), </i>pages 889-894.</p><page local="10" global="419"/><p>Patric Pantel and Marco Pennacchiotti. 2006. Espresso: Leveraging generic patterns for automati­cally harvesting semantic relations. In <i>Proceedings ofthe 21st International Conference on Computa­tional Linguistics and the 44th annual meeting ofthe </i>ACL, pages 113-120.</p><p>Fuchun Peng, Nawaaz Ahmed, Xin Li, and Yumao Lu. 2007. Context sensitive stemming for web search. In <i>Proceedings ofthe 30th annual interna­tional ACMSIGIR conference on Research and De­velopment in Information Retrieval, </i>pages 639-646.</p><p>Knut M. Risvik, Tomasz Mikolajewski, and Peter Boros. 2003. Query segmentation for web search.</p><p>In <i>Poster Session in The Twelfth International World Wide Web Conference.</i></p><p>Satoshi Sekine and Hisami Suzuki. 2007. Acquiring ontological knowledge from query logs. In <i>Proceed­ings of the 16th international conference on World Wide Web, </i>pages 1223-1224.</p><p>Xu Sun, Jianfeng Gao, Daniel Micol, and Chris Quirk. 2010. Learning phrase-based spelling error mod­els from clickthrough data. In <i>Proceedings ofthe 48th Annual Meeting ofthe Association for Compu­tational Linguistics, </i>pages 266-274.</p><p>Huihsin Tseng, Longbin Chen, Fan Li, Ziming Zhuang, Lei Duan, and Belle Tseng. 2009. Mining search engine clickthrough log for matching N-gram fea­tures. In <i>Proceedings ofthe 14th Conference on Empirical Methods in Natural Language Process­ing, </i>pages 524-533.</p><p>Xing Wei, Fuchunk Peng, Huishin Tseng, Yumao Lu, and Benoit Dumoulin. 2009. Context sensitive syn­onym discovery for web search queries. In <i>Proceed­ing ofthe 18th ACMconference on Information and Knowledge Management, </i>pages 1585-1588.</p></references></body></article>