<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="524"/><title>Mining Search Engine Clickthrough Log for Matching N-gram Features</title><author surname="Tseng" givenname="Huihsin"><org  name="Yahoo! Inc." country="USA" city="Santa Clara"/></author><author surname="Chen" givenname="Longbin"><org  name="Yahoo! Inc." country="USA" city="Santa Clara"/></author><author surname="Li" givenname="Fan"><org  name="Yahoo! Inc." country="USA" city="Santa Clara"/></author><author surname="Zhuang" givenname="Ziming"><org  name="Yahoo! Inc." country="USA" city="Santa Clara"/></author><author surname="Duan" givenname="Lei"><org  name="Yahoo! Inc." country="USA" city="Santa Clara"/></author><author surname="Tseng" givenname="Belle"><org  name="Yahoo! Inc." country="USA" city="Santa Clara"/></author></firstpageheader><frontmatter><p><b>Mining Search Engine Clickthrough Log for Matching A-gram Features</b></p><p><b>Huihsin Tseng, Longbin Chen, Fan Li, Ziming Zhuang, Lei Duan, Belle Tseng</b></p><p>Yahoo! Inc., Santa Clara, CA 95054</p><p>{huihui,longbin,fanli,ziming,leiduan,belle}@yahoo-inc.com</p></frontmatter><abstract>User clicks on a URL in response to a query are extremely useful predictors of the URL's rele­vance to that query. Exact match click features tend to suffer from severe data sparsity issues in web ranking. Such sparsity is particularly pro­nounced for new URLs or long queries where each distinct query-url pair will rarely occur. To remedy this, we present a set of straightforward yet informative query-url n-gram features that al­lows for generalization of limited user click data to large amounts of unseen query-url pairs. The method is motivated by techniques leveraged in the NLP community for dealing with unseen words. We find that there are interesting regulari­ties across queries and their preferred destination URLs; for example, queries containing "form" tend to lead to clicks on URLs containing "pdf". We evaluate our set of new query-url features on a web search ranking task and obtain improve­ments that are statistically significant at a p-value &lt; 0.0001 level over a strong baseline with exact match clickthrough features. </abstract></header><body><section number="1" title="Introduction"><p>Clickthrough logs record user click behaviors, which are a critical source for improving search relevance (Bilenko and White, 2008; Radlinski et al., 2007; Agichtein and Zheng, 2006; Lu et al. 2006). Previous work (Agichtein et al., 2006) demonstrated that clickthrough features (e.g., IsNextClicked and IsPreviousClicked) can lead to substantial improvements in relevance. Such features summarize query-specific user interac­tions on a search engine. One commonly used clickthrough feature is generated based on the following observation: if a URL receives a large number of first and last clicks across many user sessions, then it indicates that this URL might be a strongly preferred destination of a query. For example, when a user searches for "yahoo", they tend to only click on the URL www.yahoo.com<u> </u>rather than other alternatives. This results in www.yahoo.com being the first and last clicked URL for the query. We refer to such behavior as being navigational clicks <b>(NavClicks). </b>Features that use exact query and URL string matches (e.g., NavClick, IsNextClicked and IsPrevious-Clicked) are referred to as exact match features <b>(ExactM) </b>for the remainder of this paper.</p><p>The coverage of ExactM features is sparse, espe­cially for long queries and new URLs. Many long queries are either unique or very low fre­quency. Hence, the improvements from ExactM features are limited to the more popular queries. In addition, ExactM features tend to be weighted heavily in the ranking of results when they are available. This introduces a bias where the rank­ing models tend to strongly favor older URLs over new URLs even when the latter otherwise appear to be more relevant.</p><p>By inspecting the clickthrough logs, we observed that unseen query-url pairs are often composed of informative previously observed subsequences. Specifically, we saw that query <i>n</i>-grams can be correlated with sequences of URL <i>n</i>-grams. For example, we find that there are interesting regu­larities across queries and URLs, such as queries containing "form" tending to lead to clicks on URLs containing "pdf". This strongly motivates the adoption of an approach similar to the Natu­ral Language Processing <b>(NLP) </b>technique of us­ing <i>n</i>-grams to deal with unseen words. For ex­ample, part-of-speech tagging (Brants, 2000) and parsing (Klein and Manning, 2003) both require dealing with unknown words. By using <i>n</i>-gram substrings, novel items can be dealt with using any informative substrings they contain that were actually observed in the training data.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">524</doubt><p><i>Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, </i>pages 524—533, Singapore, 6-7 August 2009. ©2009 ACL and AFNLP<page local="2" global="525"/></p><p>The remainder of the paper is organized as fol­lows. In Section 2, we introduce our overall me­thodology. Section 2.1 presents a data mining method for building a query-url <i>n</i>-gram diction­ary, Section 2.2 describes the new ranking fea­tures in detail. In section 3, we present our ex­perimental results. Section 4 discusses related work, and Section 5 summarizes the contribution of this work.</p></section><section number="2" title="Methodology"><p>This section describes the detailed methodology used in generating the query-url <i>n</i>-gram features. Our features require association scores to be pre­viously calculated, and, hence, we first introduce a data mining approach that is used to build an association dictionary in Section 2.1. Then, we present the procedure used to generate the query-url <i>n</i>-gram features that use the dictionary in Sec­tion 2.2.</p><p><b>Figure 1: </b>Steps to build a query-url <i>n</i>-gram dic­tionary</p><subsection number="2.1" title="Data Mining on a Query-URL A -gram Dictionary"><p>The steps involved in building the dictionary are shown in Figure 1. We first collect seed query-url pairs from clickthrough data based on Nav-Clicks. The queries and URLs from the collected pairs are tokenized and converted into a collec­tion of paired query-url <i>n</i>-grams. For each pair, we calculate the mutual information of the query <i>n</i>-gram and its corresponding URL <i>n</i>-gram. For our experiment, we collect a total of more than 15M seed pairs and 0.5B query-url <i>n</i>-gram pairs using six months of query log data. The details are described in the following sections.</p><subsubsection number="2.1.1" title="Seed List"><p>We identify the seed list based on characteristic user click behavior. Given a query, we select the URL with the most NavClicks as compared to other URLs returned. During data collection, the rank positions of the top 5 URLs were shuffled to avoid the position bias. We aggregate NavClicks for a URL occurring in these positions in order to both obtain more click data and to avoid the posi­tion bias issue discussed in Dupret and Piwowarski (2008) and Craswell et al. (2008).</p><p>For example, in Figure 1, the numbers of Nav-Clicks for the top three URLs are shown. The URL www.irs.gov/pub/irs-pdf/f1<u>040.pdf</u> receives the largest number of NavClicks, and, therefore, it is used to create the query-url pair:</p><doubt alpha="61.2" length="49" tooSmall="False" monospace="0.0">[irs 1040 form,www.irs.gov/pub/irs-pdf/f1040.pdfl</doubt></subsubsection><subsubsection number="2.1.2" title="Query and URL Segmentation"><p>We segment the seed pairs to <i>n</i>-gram pairs in order to increase the coverage beyond that of ExactM click features. Within NLP, <i>n</i>-grams are typically extracted such that words that are adja­cent in the original sequence are also adjacent in the extracted <i>n</i>-grams. Furthermore, we attempt to achieve additional generalization by using skip <i>n</i>-grams (Lin and Och, 2004). This means we not only extract <i>n</i>-grams for adjacent terms but also for sequences that leave out intermediate terms. This is motivated by the observation that the se­mantics of user queries is often preserved even when some intermediate terms are removed. The details of the segmentation methods are de­scribed below.</p><doubt alpha="63.0" length="27" tooSmall="False" monospace="0.0">2.1.2.1  Query Segmentation</doubt><p>Prior to query segmentation, we normalize raw queries by replacing punctuations with spaces. Queries are then segmented into a sequence of space delimited tokens.<page local="3" global="526"/> From these, we extract all possible query n-grams and skip n-grams for <i>n </i>smaller than or equal to three (i.e., all unigrams, bigrams, and trigrams). For example, given the sequence "irs 1040 form" the adjacent bigrams would be "irs 1040" and "1040 form". With skip n-grams we also extract "irs form" as shown in Table 1. We do not use n-grams longer than 3 in order to avoid problems with overfitting. We will refer to this segmentation method as <b>Affix Seg­mentation.</b></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">525</doubt><p><b><u>Table 1: </u></b><u>An Example of Affix Segmentation</u></p><doubt alpha="62.5" length="24" tooSmall="False" monospace="0.0">2.1.2.2 URL Segmentation</doubt><p>As shown in Table 2, after the queries are seg­mented, URLs are categorized into four groups: domain, URL language, URL region and URL path. In general, a URL is delimited by punctua­tion characters such as "?",      "/", and "=".</p><p><b><u>Table 2: </u></b><u>An Example of URL Segmentation</u></p><p>The domain group includes one domain token, for example, irs.gov. Although domains could be divided into multiple <i>n</i>-grams, we treat them as a single unit, with the exception of encoded lan­guage and region information.</p><p>The language and region groups are based on the language or region part of the URL <i>n</i>-grams such as the suffixes ".en" and ".de". The language and region of a URL <i>n</i>-gram are identified by a table look-up method. The table is created based on the information available at en.wikipedia.org/<u> wiki/List of ISO 639-1 codes</u> and <u>en.wikipedia. org/wiki/ISO 3166</u>. When there is no clear lan­guage or region URL n-gram, we use English (en) as the default language and United States (us) as the default region.</p></subsubsection><subsubsection number="2.1.3" title="Calculation of Mutual Information"><p>After query and URL <i>n</i>-grams are extracted, we calculate mutual information (Gale and Church, 1991) to determine the degree of association between the <i>n</i>-grams. The definition of query-url <i>n-</i>gram mutual information (MI) is given in Equa­tion 1.</p><doubt alpha="48.0" length="25" tooSmall="False" monospace="0.0">tv,tt/    xFreq(q, u) ,1N</doubt><doubt alpha="64.7" length="17" tooSmall="False" monospace="0.0">S2Freq(q) Freq(u)</doubt><p>Here <i>q </i>corresponds to a query n-gram and <i>u </i>cor­responds to a URL <i>n</i>-gram. Freq <i>(q) </i>is the count of q in the seed list normalized by the total num­ber of <i>q. </i>Freq <i>(u)</i><i> </i>is the count of u normalized by the total number of <i>u. </i>Freq <i>(q,</i><i> </i><i>u)</i><i> </i>is the count of <i>q </i>and <i>u </i>that co-occurred in a full query-url pair normalized by the total number of <i>q </i>and <i>u. </i>A pair will be assigned to a MI score of zero if the items occur together no more than expected by chance, under the assumption that the two items are sta­tistically independent. When a pair occurs more than is expected by chance, the MI score is posi­tive. On the other hand, if a pair occurs together less than is expected by chance, the mutual in­formation score is negative. In order to increase the confidence of the MI scores, we remove all <i>n</i>-grams with less than 3 occurrences in the seed list, and assign a zero MI score for any pairs in­volving these n-grams. No smoothing is applied.</p><p>This scoring scheme fits well with the associa­tion properties we would like to have for our query-url <i>n</i>-gram click features. If a query <i>n-</i>gram cues for a certain URL through one of its <i>n-</i>grams, the feature will take on a positive value. Similarly, if a query <i>n</i>-gram cues against a cer­tain URL, the feature will take on a negative val­ue.</p></subsubsection><subsubsection number="2.1.4" title="Analysis of Query-URL VV-gram Association"><p>By examining our dictionary, we observed a number of pairs that are interesting from a rele­vance ranking perspective. To illustrate, we pre­sent four examples of <i>n</i>-gram pairs and intui­tively explore the nature of the <i>n</i>-gram associa­tions in the dictionary.</p><p><b><u>Table 3: </u></b><u>Examples of MI Scores</u></p><p>First, let's examine the association between query <i>n</i>-grams and URL <i>n</i>-grams for the queries "iphone" and "iphone plan".<page local="4" global="527"/> Notice that the query unigram "iphone" is strongly associated with apple.com, but negatively associated with amazon.com. This can be explained by the fact that "iphone" as a product is not only developed by Apple but also strongly associated with the Apple brand. In contrast, while Amazon.com sells iphones, it also sells a large variety of other products, thus is not regarded as a very authorita­tive source of information about the "iphone". However, by adding additional context, the most preferred URL according to MI can change. The two examples in the bottom of Table 3 illustrate the URL preferences for the query bigram "iphone plan". While apple.com is still a strongly preferred destination, there is a much stronger preference for att.com. This preference follows since apple.com has more product information on the "iphone" while the information provided by att.com will be more targeted at visitors who want to explore what rate plans are available.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">526</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>N-gram</p></td><td class="cell"><p>Affix Segmentation</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Unigram</p></td><td class="cell"><p>irs, 1040, form</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Bigram</p></td><td class="cell"><p>irs 1040, 1040 form, irs form</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Trigram</p></td><td class="cell"><p>irs 1040 form</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>URL Groups</p></td><td class="cell"><p>Example</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Domain</p></td><td class="cell"><p>irs.gov</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>URL language</p></td><td class="cell"><p>en</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>URL region</p></td><td class="cell"><p>us</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>URL path</p></td><td class="cell"><p>pub, irs, pdf, f1040, pdf</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Query n-gram</p></td><td class="cell"><p>URL n-gram</p></td><td class="cell"><p>MI score</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>"iphone"</p></td><td class="cell"><p>apple.com</p></td><td class="cell"><p>8.7713</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>"iphone"</p></td><td class="cell"><p>amazon.com</p></td><td class="cell"><p>-0.1555</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>"iphone plan"</p></td><td class="cell"><p>att.com</p></td><td class="cell"><p>11.5388</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>"iphone plan"</p></td><td class="cell"><p>apple.com</p></td><td class="cell"><p>8.9676</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Second, Table 4 shows the association between "kimo", ".tw" and ".us". "Kimo" was a Taiwan­ese start-up acquired by Yahoo!. The mutual in­formation scores accurately reflect the associa­tion between the query <i>n</i>-gram and region ids.</p><p><b><u>Table 4: </u></b><u>Example of MI Scores</u></p><p>Third, Table 5 shows the association between "kanji", and URLs with Language identification of "Japanese", "Chinese" and "English". "Kanji" means "Chinese" in Japanese. Since queries con­taining "Kanji" are typically from users inter­ested in Japanese sites, the mutual information shows higher correlation with Japanese than with English or Chinese.</p><p><b><u>Table 5: </u></b><u>Example of MI Scores</u></p><p>Fourth, Table 6 shows the association between two query <i>n</i>-grams, "form" and "video", that at first glance may not actually look very informa­tive for URL path selection. However, notice that the unigram "form" has a strong preference for pdf documents over more standard web pages with an html extension. Similarly, queries that include "video" convey a preference for URLs containing "watch", a characteristic URL <i>n</i>-gram for many video sharing websites.</p><p>It is reasonable to anticipate that incorporating such associations into a search engine's ranking function should help improve both search quality and user experience. Take the example where, there are two high ranking competing URLs for the query "irs 1040 form". Let's also assume both documents contain the same query relevant keywords, but one is an introduction of the "irs 1040 form" as an htm webpage and the other one is the real filing form given as a pdf document. Since in our dictionary, "form" is more associ­ated with pdf than htm, we predict that most us­ers would prefer the real pdf form directly, so it should be placed first in the list of query results. While click data for the exact query-url pairs confirms this preference, it is reassuring that we could identify it without needing to rely on see­ing the specific query string before. As described in detail below, and motivated by this analysis, we designed our query-url click features based on the contents of the <i>n</i>-gram MI dictionary.</p></subsubsection></subsection><subsection number="2.2" title="Query-URL VV-gram Features"><p>For our feature set, we explored the use of differ­ent query segmentation approaches (concept and affix segmentation) in order to increase the di­versity of <i>n</i>-grams. In the following section, we use an unseen query "irs 1040 forms" and con­trast it with the known query "irs 1040 form" from the last section.</p><subsubsection number="2.2.1" title="Concept Segmentation Features"><p>Query concept segmentation is a weighted query segmentation approach. Each query is analyti­cally interpreted as being a main concept and a sub concept. We search for the unique segmenta­tion of the query that maximizes its cumulative mutual information score with the URL <i>n</i>-grams. Main concepts and sub concepts are <i>n</i>-grams from the query that have the strongest association with URL <i>n</i>-grams and thus assist in identifying relevant landing URL <i>n</i>-grams when the whole query or the whole URL has not been seen.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">527</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Query n-gram</p></td><td class="cell"><p>URL n-gram</p></td><td class="cell"><p>MI score</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>"kimo"</p></td><td class="cell"><p>tw (taiwan)</p></td><td class="cell"><p>12.8303</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>"kimo"</p></td><td class="cell"><p>us (united states)</p></td><td class="cell"><p>0.7209</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Query n-gram</p></td><td class="cell"><p>URL n-gram</p></td><td class="cell"><p>MI score</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>"kanji"</p></td><td class="cell"><p>ja (japanese)</p></td><td class="cell"><p>11.3862</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>"kanji"</p></td><td class="cell"><p>zh (chinese)</p></td><td class="cell"><p>6.2567</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>"kanji"</p></td><td class="cell"><p>en (english)</p></td><td class="cell"><p>4.2110</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Table 6: </b>Example of MI Score</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Query n-gram</p></td><td class="cell"><p>URL n-gram</p></td><td class="cell"><p>MI score</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>"form"</p></td><td class="cell"><p>pdf</p></td><td class="cell"><p>4.9067</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>"form"</p></td><td class="cell"><p>htm</p></td><td class="cell"><p>1.0916</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>"video"</p></td><td class="cell"><p>watch</p></td><td class="cell"><p>5.7192</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>"video"</p></td><td class="cell"><p>htm</p></td><td class="cell"><p>-1.9079</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="5" global="528"/><p><b>Algorithm 1: </b>Concept Segmentation <b>for </b>U = domain, URL language, URL region,</p><p>URL path <b>do</b> <b>if </b>curr_mi_M + curr_mi_S &gt; curr_best adding mi_M as a feature adding mi_S as a feature</p><doubt alpha="46.7" length="15" tooSmall="False" monospace="0.0">forj= 0...n-1do</doubt><doubt alpha="27.3" length="11" tooSmall="False" monospace="0.0">M &lt;= W0...j</doubt><doubt alpha="45.5" length="11" tooSmall="False" monospace="0.0">S&lt;=Wj+l...n</doubt><doubt alpha="53.8" length="13" tooSmall="False" monospace="0.0">fork= 0...mdo</doubt><doubt alpha="54.8" length="73" tooSmall="False" monospace="0.0">curr_mi_M&lt;= arg maxk=1...mMI (M, Uk) curr_mi_S&lt;= arg maxk=1...mMI (S, Uk)</doubt><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">then</doubt><doubt alpha="62.4" length="85" tooSmall="False" monospace="0.0">curr_best = curr_mi_M + curr_mi_S mi_M &lt;^= curr_mi_M mi_S &lt;^= curr_mi_Send if end for</doubt><p><b>end for end for</b></p><p>Pseudo-code for generating query-url n-gram features based on the concept segmentation is given in Algorithm 1. Each query (Q) is com­posed of a number of words, w1, w2, w3...,wn. Each URL is segmented and categorized to four groups: domain, URL language, URL region and URL path. Each URL group has <i>m </i>number of URL <i>n</i>-grams. M is the main concept of Q and S is the sub concept of Q.</p><p>One potential drawback of such concept segmen­tation is data sparsity. When we look for the maximum of cumulative mutual information, we may obtain main concepts with very high mutual information and sub concepts which do not exist in the dictionary. In order to address this problem, we implement a second query segmentation me­thod, affix segmentation, that is discussed in sec­tion 2.2.2.</p><p>Table 7 shows eight concept segmented features. "Coverage" is the percentage of query-url pairs that have valid feature values. Some of the sam­ples do not have values because no clicks for the pairs were seen in the sample of data used to build the dictionary. When a pair does not have a value, the default value of zero is assigned. This default value is based on the assumption that unless we have evidence otherwise, we assume all query-url <i>n</i>-grams are statistically independ­ent and thus provide no preference signal.</p><p><b>Table 7: </b>Eight Features Generated based on <u>Concept Segmentation.</u></p><p><b>Query-URL Domain Features </b>are defined as the mutual information of a query <i>n</i>-gram and the domain level URL. There are two features in this category, one for the query main concept and one for the sub concept. They help to identify the user preferred host given a query.</p><p><b>Table 8: </b>Example of Selecting Query Segmenta­tion</p><p>To illustrate the concept segmentation features, let's examine the query, "irs 1040 forms" in the context of the domain irs.gov. The query "irs 1040 forms" can be segmented either as "irs 1040" and "forms" or as "irs" and "1040 forms". As shown in Table 8, taking the cumulative max­imum, the second segmentation scores higher than the first one. Therefore, the "irs" and "1040 forms" segmentation is preferred. The feature value for the main concept is 11.5550, and the sub concept is then assigned to be 11.2174.</p><p><b>Query-URL Language and Region Features</b> are the mutual information of a query <i>n</i>-gram and URL language/region. They are used for provid­ing language and region information.</p><p><b>Query-URL Path Features </b>are the mutual information of a query <i>n</i>-gram and a URL path <i>n-</i>gram. While there are typically many URL path <i>n</i>-grams, only one URL path <i>n</i>-gram is selected to be paired with each query <i>n</i>-gram. The se­lected <i>n</i>-gram is the one that achieves the highest<page local="6" global="529"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">528</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Feature</p></td><td class="cell"><p>Query N-</p></td><td class="cell"><p>URL N-</p></td><td class="cell"><p>Coverage</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>gram</p></td><td class="cell"><p>gram</p></td><td class="cell"><p>(%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MainDS</p></td><td class="cell"><p>M</p></td><td class="cell"><p>domain</p></td><td class="cell"><p>54.09</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SubDS</p></td><td class="cell"><p>S</p></td><td class="cell"><p>domain</p></td><td class="cell"><p>30.46</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MainLang</p></td><td class="cell"><p>M</p></td><td class="cell"><p>lang.</p></td><td class="cell"><p>94.41</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SubLang</p></td><td class="cell"><p>S</p></td><td class="cell"><p>lang.</p></td><td class="cell"><p>72.40</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MainReg</p></td><td class="cell"><p>M</p></td><td class="cell"><p>reg.</p></td><td class="cell"><p>90.34</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SubReg</p></td><td class="cell"><p>S</p></td><td class="cell"><p>reg.</p></td><td class="cell"><p>68.19</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MainPath</p></td><td class="cell"><p>M</p></td><td class="cell"><p>path</p></td><td class="cell"><p>64.96</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SubPath</p></td><td class="cell"><p>S</p></td><td class="cell"><p>path</p></td><td class="cell"><p>58.76</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MI(q,u)</p></td><td class="cell"><p>irs.gov</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>"irs"</p></td><td class="cell"><p></p></td><td class="cell"><p>11.2174</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>"1040"</p></td><td class="cell"><p>11.6175</p></td><td class="cell"><p></p></td><td class="cell"><p>11.5550</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>"forms"</p></td><td class="cell"><p>7.5049</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Cumulative MI</p></td><td class="cell"><p>19.1224</p></td><td class="cell"><p>22.7724</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Seg. 1</p></td><td class="cell"><p>Seg.2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p><u>Affix Segmentation</u> <b>for </b>U = domain, URL language, URL region,</p><p>URL path <b>do</b></p><p><b>end for end for</b> cumulative maximum MI score. They are used for providing association between query n-grams and url n-grams such as "forms" and "pdf'.</p></subsubsection><subsubsection number="2.2.2" title="Affix Segmentation Features"><p>As previously mentioned, affix segmentation addresses sparsity issues associated with concept segmentation. Here, we introduce the features generated based on affix segmentation. Pseudo­code for generating the features is given in Algo­rithm 2. Two query unigrams (w0 and wn) and one bigram (w0wn) is used. Each URL is seg­mented and categorized to four groups: domain, URL language, URL region and URL path. Each URL group has <i>m </i>number of URL n-grams.</p><p>This approach is complementary to the concept segmentation for long queries. The affix <i>n</i>-grams are in smaller unit, and therefore, are less sparse. In addition, the skip bigrams allow for generali­zations using non-adjacent terms. Table 9 shows the coverage of the twelve affix features.</p><p><b>Algorithm 2: </b>Affix Segmentation</p><doubt alpha="57.6" length="33" tooSmall="False" monospace="0.0">forq = w0,wn,w0wndo fork= 0...mdo</doubt><p>curr_mi_q &lt;= arg maxk=1.m MI (q, Uk) <b>if </b>curr_mi_q &gt; curr_best <b>then </b>curr_best = curr_mi_q</p><p><b>end if end for</b> adding curr_mi_q as a feature <b>Table 9:</b><b> </b>Twelve Features Generated based on <b>Query-url domain affix features </b>has three fea­tures: MI(w0, domain), MI(wn, domain), and MI(w0wn, domain). In the example of "irs 1040 forms" and "irs.gov", the features are MI(irs, irs.gov), MI(forms, irs.gov), and MI(irs forms, irs.gov).</p><p><b>Query-url language and region affix features </b>has three features respectively: MI(w0, language), MI(wn, language), MI(w0wn, language) MI(w0, region), MI(wn, region), and MI(w0wn, region). In the example of "irs 1040 forms", "en" and "us", the features are MI (irs, en), MI (forms, en), MI (irs forms, en), MI (irs, us), MI (forms, us), and MI (irs forms,us).</p><p><b>Query-url path affix features </b>has three fea­tures: MI(w0, path), MI(wn, path), and MI(w0wn, path). In the example of "irs 1040 forms" and "www.irs.gov/pub/irs-pdf/f1040.pdf', there are four URL path <i>n</i>-grams, "pub", "irs", "pdf", and "f1040". The URL path <i>n</i>-gram, irs, gets maxi­mum MI score. Therefore, the query-url path af­fix features are MI (irs, irs), MI (forms, irs), and MI (irs forms, irs).</p><p>We demonstrated the procedure to generate 20 query-url <i>n</i>-gram features, and in Section 3, we will present their effectiveness in relevance rank­ing.</p></subsubsection></subsection></section><section number="3" title="Experiment"><p>We evaluate the performance of query-url <i>n-</i>grams features (8 concept and 12 affix features) on a ranking application and analyze the results from several different perspectives.</p><subsection number="3.1" title="Datasets"><p>For all experiments, our training and test data are query-url pairs annotated with human judgments. In our data, we use five grades to evaluate rele­vance of a query and URL pair.</p><p>The data includes 94K queries for training and 3.4K queries for evaluation, and each query is associated with the top ranked URLs returned from a search engine. Totally, there are 916K query-url pairs for training and 42K pairs for testing. The queries are general and uniformly and randomly sampled with replacement, result­ing in more frequent queries also appearing more frequently in our training and test sets.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">529</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Feature</p></td><td class="cell"><p>Query N-</p></td><td class="cell"><p>URL N-</p></td><td class="cell"><p>Coverage</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>gram</p></td><td class="cell"><p>gram</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PreDS</p></td><td class="cell"><p>w0</p></td><td class="cell"><p>domain</p></td><td class="cell"><p>48.09</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SufDS</p></td><td class="cell"><p>wn</p></td><td class="cell"><p>domain</p></td><td class="cell"><p>47.72</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PresufDS</p></td><td class="cell"><p>w0wn</p></td><td class="cell"><p>domain</p></td><td class="cell"><p>23.57</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PreLang</p></td><td class="cell"><p>w0</p></td><td class="cell"><p>lang.</p></td><td class="cell"><p>55.58</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SufLang</p></td><td class="cell"><p>wn</p></td><td class="cell"><p>lang.</p></td><td class="cell"><p>58.22</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PresufLang</p></td><td class="cell"><p>w0wn</p></td><td class="cell"><p>lang.</p></td><td class="cell"><p>24.91</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PreReg</p></td><td class="cell"><p>w0</p></td><td class="cell"><p>reg.</p></td><td class="cell"><p>93.82</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SufReg</p></td><td class="cell"><p>wn</p></td><td class="cell"><p>reg.</p></td><td class="cell"><p>93.59</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PresufReg</p></td><td class="cell"><p>w0wn</p></td><td class="cell"><p>reg.</p></td><td class="cell"><p>69.29</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PrePath</p></td><td class="cell"><p>w0</p></td><td class="cell"><p>path</p></td><td class="cell"><p>98.15</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SufPath</p></td><td class="cell"><p>wn</p></td><td class="cell"><p>path</p></td><td class="cell"><p>97.80</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PresufPath</p></td><td class="cell"><p>w0wn</p></td><td class="cell"><p>path</p></td><td class="cell"><p>75.81</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="7" global="530"/></subsection><subsection number="3.2" title="Ranking Algorithm"><p>GBRank is a supervised learning algorithm that uses boosted decision trees and incorporates the pair-wise information from the training data (Zheng et al, 2007). It is able to deal with a large amount of training data with hundreds of features. We use an internal C++ implementation of</p><p>GBRank.</p></subsection><subsection number="3.3" title="Evaluation Metric"><p>We use Discounted Cumulative Gain (Järvelin and Kekäläinen, 2002) to evaluate our ranking accuracy. Discounted Cumulative Gain (DCG) has been widely used in evaluating the quality of search engine rankings and is defined as:</p><doubt alpha="57.1" length="7" tooSmall="False" monospace="0.0">DCGk= 2</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">Gi</doubt><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">i=1</doubt><doubt alpha="40.0" length="10" tooSmall="False" monospace="0.0">log2(i +1)</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(2)</doubt><p><i>G</i><i>i </i>represents the editorial judgment of the <i>i</i>-th document. In this paper, we only report <b>normal­ized DCG5, </b>which is an absolute DCG5 normal­ized by a baseline, and <b>relative DCG5 im­provement, </b>which is an improvement normal­ized by the baseline. Note normalized DCG5 is different than NDCG (Normalized Discounted Cumulative Gain defined in Järvelin and Kekäläinen, 2002). We use Wilcoxon signed test (Wilcoxon, 1945) to evaluate the significance for model comparison.</p></subsection><subsection number="3.4" title="Feature Sets"><p>Five feature sets are used in Details are listed in Table 10.</p><p>our experiments.</p><p><b><u>Table 10: </u></b><u>Five Feature Sets</u>_ <u>Tag</u>__<u>Description</u>_</p><p>Base Feature       Core Feature Set and ExactM <u>Set</u>__<u>click features</u>_</p><doubt alpha="64.1" length="78" tooSmall="False" monospace="0.0">Q-U N-gram       Base Feature Set and Q-U N-Feature Set (I)   | gram features_</doubt><p>Core Feature</p><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">Set</doubt><p>NavClick Fea­ture Set</p><p>Q-U <i>N</i>-gram Feature Set (II) query-based, document-based, query-document based fea­<u>tures</u>_</p><p>Core Feature Set and Nav-</p><doubt alpha="100.0" length="5" tooSmall="False" monospace="0.0">Click</doubt><p>Core Feature Set and Q-U N-<u>gram features</u>_ <b>Base Feature Set </b>is a strong baseline feature set from a state-of-the-art commercial search engine. This set includes NavClick features, and other internal ExactM click features. It is used for evaluating Query-URL <i>N</i>-gram Feature Set (I) in order to know whether query-url <i>n</i>-gram features can achieve gains when stacked on top of Ex-actM features.</p><p><b>Core Feature Set </b>is a weaker variant of the baseline system that excludes ExactM click fea­tures. This system is used for evaluating NavClick Feature Set and Query-URL <i>N</i>-gram Feature Set (II) independently in order to study and contrast the effected queries.</p></subsection><subsection number="3.5" title="Experimental Results"><p>We compare the query-URL <i>N</i>-gram feature set (I) with the base feature set in Section 3.5.1, and contrast the NavClick features and the query-URL <i>N</i>-gram features (II) using the Core Feature Set in Section 3.5.2.</p><subsubsection number="3.5.1" title="Query-URL N-gram Feature Set (I) versus Base Feature Set"><p>As shown in Figure 2, Query-URL <i>N</i>-gram Fea­ture Set (I) outperforms Base Feature Set. The additional 20 query-url <i>n</i>-gram features achieve statistically significant gains at a p-value &lt; 0.0001 level, suggesting that they are compli­mentary to ExactM click features. Even though the query-url <i>n</i>-gram features are generated from the same data as the ExactM features, the gain is additive and stackable. The DCG5 impact is 0.53% relative improvement when running GBRank using 2500 trees. Every data point is normalized by the DCG5 of the baseline feature set using 2500 trees. This is represented in the graph as the rightmost point of Base Feature Set curve.</p><p><b>Figure 2: </b>Comparison of the five feature sets on the normalized DCG5 (Y-axis) against number of trees (X-axis).</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1.01</doubt><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">0.99</doubt><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">0.98</doubt><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">0.97</doubt><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">0.96</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">500</doubt><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1000</doubt><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1500</doubt><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">2000</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">530</doubt><page local="8" global="531"/><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">Set</doubt><p><b>3.5.2 NavClick and Query-URL VV-gram Feature Set (II) versus Core Fea­ture Set</b></p><p>We compare NavClick Feature Set and Query-URL VV-gram Feature Set (II) in the context of Core Feature Set, in order to evaluate the two independently. As shown in Figure 2, both NavClick and Query-URL <i>V</i>-gram Feature Set (II) outperform Core Feature Set. It is not sur­prising that NavClick also outperforms Query-URL VV-gram Feature Set (II) since the n-gram features are backoff of NavClick. However, their gains are competitive suggesting the query-url n-gram features are very good relevance indicators. The impact of NavClick and Query-URL <i>V</i>-gram Feature Set (II) is 0.72% and 0.62% relative DCG5 improvement at Tree 2500 respectively.</p></subsubsection><subsubsection number="3.5.3" title="Feature Importance"><p>Using the GBRank model, features are evaluated and sequentially selected to build the boosted decision trees. The split of each node increases the DCG during training. We evaluate a feature's importance by aggregating the DCG impact of the feature over all trees (Zheng et al., 2007). Here, the feature importance is rescaled so that the feature with largest DCG impact is assigned a normalized score of 1. Figure 3 illustrates the relative influence of each of query-url <i>n</i>-gram feature. Of these, n-gram features associated with a domain name (i.e., MainDS) rank highest.</p><p>0.8 -.............................................rj--r-r...................</p><doubt alpha="3.3" length="60" tooSmall="False" monospace="0.0">0.7-...................................................o-p,-</doubt><doubt alpha="0.0" length="20" tooSmall="False" monospace="0.0">0.6 0.50.4 0.30.20.1</doubt><doubt alpha="12.7" length="55" tooSmall="False" monospace="0.0">0111 ! IU U111111111111111U U111111111111111U U1111111,</doubt><doubt alpha="21.2" length="99" tooSmall="True" monospace="0.0">V)     Ul   V)    Ul.CV)Ol    D)    Ol   £    O)    O)   .£    .£    (/)     Dl    SI    Ol   £ (/)</doubt><doubt alpha="19.1" length="47" tooSmall="False" monospace="0.0">«   ï   »   ïû   £Il^   çj=   d   d   .     ^ =</doubt><doubt alpha="14.3" length="14" tooSmall="False" monospace="0.0">««|5=»|q.55|«w</doubt><p><b>Figure 3: </b>Feature importance of query-url <i>n-</i>gram features. The importance (Y axis) is nor­malized so that the most important feature (MainDS)'s importance is 1.</p></subsubsection></subsection><subsection number="3.6" title="Analysis"><p>We access system performance with respect to both query length and frequency using the two click features sets in combination with the Core</p><p>Feature Set in order to gain insight into the ef­fected queries.</p><subsubsection number="3.6.1" title="Query Length"><p>As shown in Table 11, NavClick (NavClick Fea­ture Set) best improves relevance for two word queries. In contrast, Query-url <i>n</i>-gram features in isolation (Query-URL <i>V</i>-gram Feature II) are able to show sizable improvements on longer queries, while slightly degrading performance on short 1-word queries. Using both feature sets to­gether (Query-URL VV-gram Feature I) results in improvement for queries of all lengths.</p><p>These results suggest that the strong signal being provided by NavClick for short queries helps to compensate for any additional noisy introduced by the <i>n</i>-gram features, while allowing the <i>n-</i>gram features to handle longer queries that are less well covered by NavClick. These longer queries are exactly the type of queries our query-url <i>n</i>-gram features were designed to help with.</p><p><b>Table 11: </b>Relative DCG5 Improvement of NavClick, Query-URL <i>V</i>-gram (II), and Query-<u>URL </u><u>VV</u><u>-gram Features (I) vs Core Feature Set</u></p></subsubsection><subsubsection number="3.6.2" title="Query Frequency"><p>We found that query-url <i>n</i>-gram features improve tail queries. Head queries are considered as top two million frequent queries in our traffic and tail queries include anything outside of that range.</p><p><b>Table 12: </b>Relative DCG5 Improvement of NavClick, Query-URL <i>V</i>-gram Features (II) and Query-URL <i>V</i>-gram Features (I) vs Core Feature</p><p>As shown in Table 12, query-url <i>n</i>-gram features (Query-URL Feature Set II) differ from NavClick (NavClick Feature Set) in that they get more gain from tail queries.<page local="9" global="532"/> Together, they (Query-URL Feature Set I) improve both head and tail queries.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">531</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Length</p></td><td class="cell"><p>NavClick</p></td><td class="cell"><p>QU VV-</p></td><td class="cell"><p>QU VV-</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>vs Core</p></td><td class="cell"><p>gram (II)</p></td><td class="cell"><p>gram (I)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>vs Core</p></td><td class="cell"><p>vs Core</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>(%)</p></td><td class="cell"><p>(%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1 word</p></td><td class="cell"><p>0.03</p></td><td class="cell"><p>-0.04</p></td><td class="cell"><p>0.62</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2 words</p></td><td class="cell"><p>1.04</p></td><td class="cell"><p>1.06</p></td><td class="cell"><p>1.58</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3 words</p></td><td class="cell"><p>1.00</p></td><td class="cell"><p>1.44</p></td><td class="cell"><p>2.12</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>4+ words</p></td><td class="cell"><p>0.4</p></td><td class="cell"><p>0.68</p></td><td class="cell"><p>1.01</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>NavClick vs</p></td><td class="cell"><p>QU VV-gram</p></td><td class="cell"><p>QU VV-</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Core (%)</p></td><td class="cell"><p>(II) vs Core</p></td><td class="cell"><p>gram (I) vs</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>Core (%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Head</p></td><td class="cell"><p>0.91</p></td><td class="cell"><p>-0.15</p></td><td class="cell"><p>1.11</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Tail</p></td><td class="cell"><p>0.59</p></td><td class="cell"><p>1.11</p></td><td class="cell"><p>1.40</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></subsubsection></subsection><subsection number="3.7" title="Case Study"><p>Below we examine queries from the test set and analyze the effects of Query-URL <i>V</i>-gram Fea­ture Set (II) versus Core Feature Set.</p><subsubsection number="3.7.1" title="Positive Cases"><p>1) <u>Animal shelter in va</u>: this query targets a spe­cific geographic location. Using the baseline fea­ture set, the root url wvanimalshelter.org is incor­rectly ranked higher than www.netpets.com/<u> cats/catresc/virginia.htm</u>. Without any addition­ally ranking information, general URLs (root) tend to be ranked more highly than more specific URLs (path), as the root pages tend to be more popular. However, our new features express a preference between "va" and "virginia", and this correctly flips the ranking order.</p><p>2) <u>Myspace profile generator</u>: <u>www. myspacgens. com/handler.php?gen=profile</u> was incorrectly ranked higher than www.profilemods.com/<u> myspace-generators</u>. Our new features convey a high user preference association between "profile generator" and the domain profilemods.com, which helps to correctly swap the order.</p></subsubsection><subsubsection number="3.7.2" title="Negative Cases"><p>We determined that negative cases where the baseline feature set outperforms the new features are typically one word navigational queries such as "craigslist". However, after we combine the query-url <i>n</i>-gram features with NavClick, one word navigational queries are ranked correctly.</p></subsubsection></subsection></section><section number="4" title="Related Work"><doubt alpha="66.0" length="47" tooSmall="False" monospace="0.0">Our work is mainly related to Gao et al. (2009)</doubt><doubt alpha="53.2" length="47" tooSmall="False" monospace="0.0">and Bilenko and White (2008). Gao et al. (2009)</doubt><p>addressed the sparsity issue by propagating click information among similar queries in the same cluster. Their idea is based on an observation that similar queries go to similar pages. When two queries have similar clicked URLs, it is likely that they share clicked URLs. In contrast, our idea is to utilize NLP techniques to break down long, infrequent queries into shorter, frequent queries. The two approaches can be mutually beneficial. Bilenko and White (2008) expanded click data with a search engine by using post-search user experience collected from toolbars. Toolbars keep track of users' click behavior both when they are using the search engine directly and beyond. Their relevance features are built based on whole session clicks extracted from the toolbar. In contrast, our <i>n</i>-gram features are built on search engine clicks directly. We should be able to expand our method to integrate the post-search clicks with toolbar data.</p><p>Other related work can be found in the domain of query rewriting. Our <i>n</i>-gram dictionary was orig­inally designed for query rewriting. Query re­writing (Xu and Croft, 1996; Salton and Voor-hees, 1984) reformulates a query to its synonyms or related terms automatically. However, the coverage of query rewriting is normally small, because an inappropriate rewrite can cause sig­nificant decrease in precision. In contrast, our approach can cover a larger number of queries without decreasing precision, because it does not need to make a binary decision whether a query should be reformulated. The association scores between queries and rewrites are used as ranking features which are trained discriminatively to­ward search quality.</p></section><section number="5" title="Conclusion"><p>In this paper, we presented a set of straightfor­ward yet informative query-url <i>n</i>-gram features. They allow for generalization of limited user click data to large amounts of unseen query-url pairs. Our experiments showed such features gave significant improvement over models with­out using the features. In addition, we mined an interesting dictionary which contains informa­tive, but not necessarily obvious, query-url syno­nym pairs such as "form" and "pdf". We are cur­rently extending our work to a variety of exact match features and different sources of click-through logs.</p><p><b>Acknowledgement</b></p><p>Thanks to the anonymous reviewers for detailed suggestion and our colleagues: Jon Degenhardt and Narayanan Sadagopan for assistance on gen­erating clickthrough data, Jiang Chen for devel­oping the decision tree package, Xiangyu Jin for a discussion on map/reduce, Benoît Dumoulin, Fuchun Peng, Yumao Lu, and Xing Wei for pro-ductizing the work, and Rosie Jones, Su-lin Wu, Bo Long, Xin Li and Ruiqiang Zhang for com­ments on an earlier draft.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">532</doubt><page local="10" global="533"/></section><references><p>Agichtein, E., E. Brill, and S. Dumais. 2006. Im­proving web search ranking by incorporating user behavior information. In Proceedings of the ACM SIGIR 29.</p><p>Agichtein, Eugene, Zijian Zheng. 2006. Identify­ing "best bet" web search results by mining past user behavior. In Proceedings of KDD.</p><p>Bilenko, Mikhail and Ryen W. White. 2008.</p><p>Mining the search trails of surfing crowds: identifying relevant websites from user activ­ity. In Proceedings of WWW.</p><p>Brants, T. 2000. Tnt: a statistical part-ofspeech tagger. In Proceedings of ANLP 6.</p><p>Craswell, Nick and Martin Szummer. 2007. Ran­dom walks on the click graph. In Proceedings of SIGIR.</p><p>Craswell, Nick, Onno Zoeter, Michael Taylor, Bill Ramsey. 2008. An experimental compari­son of click position-bias models in WSDM.</p><p>Dupret, Georges, Benjamin Piwowarski. 2008. A user browsing model to predict search engine click data from past observations. In Proceed­ings of SIGIR 31.</p><p>Gale, William A. and Kenneth W. Church. 1991.</p><p>Identifying word correspondence in parallel texts. In Proceedings of HLT 91.</p><p>Gao, Jianfeng, Wei Yuan, Xiao Li, Kefeng Deng, and Jian-Yun Nie. 2009. Smoothing Click-through Data for Web Search Ranking. In Pro­ceedings of SIGIR 32.</p><p>Järvelin, K. and J. Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques, Jour­nal ACM Transactions on Information Sys­tems, 20: 422-446.</p><p>Klein, D. and C. Manning. 2003. Accurate unlex-icalized parsing. In Proceedings of ACL 41.</p><p>Lin, Chin-Yew and Franz Josef Och. 2004. Au­tomatic evaluation of machine translation quality using longest common subsequence and skip-bigram. In In Proceedings of ACL 42.</p><p>Lu, Yumao, Fuchun Peng, Xin Li and Nawaaz Ahmed, 2006, Coupling Feature Selection and Machine Learning Methods for Navigational Query Identification, In Proceeding of CIKM.</p><p>Radlinski, F., Kurup, M. and Joachims, T. 2007. Active exploration for learning rankings from clickthrough data. In <i>SIGKDD.</i></p><p>Salton G. and E. Voorhees. 1984. Comparison of two methods for Boolean query relevancy feedback. Information Processing &amp; Manage­ment, 20(5).</p><p>Wilcoxon, F. 1945. Individual Comparisons by Ranking Methods. Biometrics, 1:80-83.</p><p>Xu Q. and W. Croft. 1996. Query expansion us­ing local and global document analysis. In Proceed of the 19th annual international ACM</p><p>SIGIR.</p><p>Zheng, Z., H. Zha, K. Chen, and G. Sun. 2007. A regression framework for learning ranking functions using relative relevance judgments.</p><p>In Proceedings of SIGIR 30.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">533</doubt></references></body></article>