<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="116"/><title>Inducing Word Senses to Improve Web Search Result Clustering</title><pubinfo>Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing,pages 116-126, MIT, Massachusetts, USA, 9-11 October 2010. ©2010 Association for Computational Linguistics</pubinfo><author surname="Navigli" givenname="Roberto"><org  name="Sapienza Universita di Roma" country="Italy" city="Rome"/></author><author surname="Crisafulli" givenname="Giuseppe"><org  name="Sapienza Universita di Roma" country="Italy" city="Rome"/></author></firstpageheader><frontmatter><p><b>Inducing Word Senses to Improve Web Search Result Clustering</b></p><p><b>Roberto Navigli </b>and <b>Giuseppe Crisafulli</b></p><p>Dipartimento di Informatica Sapienza Universita di Roma</p><p>navigli@di.uniroma1.it, crisafulli.giu@gmail.com</p></frontmatter><abstract>In this paper, we present a novel approach to Web search result clustering based on the au­tomatic discovery of word senses from raw text, a task referred to as Word Sense Induc­tion (WSI). We first acquire the senses (i.e., meanings) of a query by means of a graph-based clustering algorithm that exploits cycles (triangles and squares) in the co-occurrence graph of the query. Then we cluster the search results based on their semantic similarity to the induced word senses. Our experiments, conducted on datasets of ambiguous queries, show that our approach improves search result clustering in terms of both clustering quality and degree of diversification. </abstract></header><body><section number="1" title="Introduction"><p>Over recent years increasingly huge amounts of text have been made available on the Web. Popular search engines such as Yahoo! and Google usually do a good job at retrieving a small number of rel­evant results from such an enormous collection of Web pages (i.e. retrieving with high precision, low recall). However, current search engines are still fac­ing the lexical ambiguity issue (Furnas et al., 1987) - i.e. the linguistic property owing to which any particular word may convey different meanings. In a recent study (Sanderson, 2008) - conducted us­ing WordNet (Miller et al., 1990) and Wikipedia as sources of ambiguous words - it was reported that around 3% of Web queries and 23% of the most frequent queries are ambiguous. Examples include: "buy B-52" (a cocktail? a bomber? a DJ worksta­tion? tickets for a band?), "Alexander Smith quotes" (the novelist? the poet?), "beagle search" (dogs? the Linux search tool? the landing spacecraft?).</p><p>Ambiguity is often the consequence of the low number of query words entered on average by Web users (Kamvar and Baluja, 2006). While average query length is increasing - it is now estimated at around 3 words per query<footnote anchor="1"/> - many search engines such as Google have already started to tackle the query ambiguity issue by reranking and diversify­ing their results, so as to prevent Web pages that are similar to each other from ranking too high on the list.</p><p>In the past few years, Web clustering engines (Carpineto et al., 2009) have been proposed as a solution to the lexical ambiguity issue in Web In­formation Retrieval. These systems group search re­sults, by providing a cluster for each specific aspect (i.e., meaning) of the input query. Users can then se­lect the cluster(s) and the pages therein that best an­swer their information needs. However, many Web clustering engines group search results on the ba­sis of their lexical similarity. For instance, consider the following snippets returned for the <i>beagle search </i>query:</p><p>1. <i>Beagle </i>is a <i>search </i>tool that ransacks your...</p><p>2. ...the <i>beagle </i>disappearing in <i>search </i>of game...</p><p>3. <i>Beagle </i>indexes your files and <i>searches...</i></p><p>While snippets 1 and 3 both concern the Linux search tool, they do not have any content word in common except our query words.<page local="2" global="117"/> As a result, they will most likely be assigned to two different clusters.</p><footnote label="1">http://www.hitwise.com/us/press-center/ press-releases/google-searches-apr-09</footnote><p>In this paper we present a novel approach to Web search result clustering which is based on the auto­matic discovery of word senses from raw text - a task referred to as Word Sense Induction (WSI). At the core of our approach is a graph-based algorithm that exploits cycles in the co-occurrence graph of the input query to detect the query's meanings. Our experiments on two datasets of ambiguous queries show that our WSI approach boosts search result clustering in terms of both clustering quality and de­gree of diversification.</p></section><section number="2" title="Related Work"><p><b>Web directories. </b>A first, historical solution to query ambiguity is that of Web directories, that is taxonomies providing categories to which Web pages are manually assigned (e.g., the Open Direc­tory Project - http://dmoz.org). Given a query, search results are organized by category. This ap­proach has three main weaknesses: first, it is static, thus it needs manual updates to cover new pages; second, it covers only a small portion of the Web; third, it classifies Web pages based on coarse cate­gories. This latter feature of Web directories makes it difficult to distinguish between instances of the same kind (e.g., pages about artists with the same surname classified as Arts:Music:Bands and Artists). While methods for the automatic clas­sification of Web documents have been proposed (e.g., (Liu et al., 2005b; Xue et al., 2008)) and some problems have been effectively tackled (Bennett and Nguyen, 2009), these approaches are usually super­vised and still suffer from relying on a predefined taxonomy of categories.</p><p><b>Semantic Information Retrieval (SIR). </b>A dif­ferent direction consists of associating explicit se­mantics (i.e., word senses or concepts) with queries and documents, that is, performing Word Sense Dis­ambiguation (WSD, see Navigli (2009)). SIR is per­formed by indexing and/or searching concepts rather than terms, thus potentially coping with two linguis­tic phenomena: expressing a single meaning with different words <i>(synonymy) </i>and using the same word to express various different meanings <i>(polysemy). </i>Over the years, different methods for SIR have been proposed (Krovetz and Croft, 1992; Voorhees, 1993; Mandala et al., 1998; Gonzalo et al., 1999; Kim et al., 2004; Liu et al., 2005a, inter alia). However, con­trasting results have been reported on the benefits of these techniques: it has been shown that WSD has to be very accurate to benefit Information Retrieval (Sanderson, 1994) - a result that was later debated (Gonzalo et al., 1999; Stokoe et al., 2003). Also, it has been reported that WSD has to be very pre­cise on minority senses and uncommon terms, rather than on frequent words (Krovetz and Croft, 1992; Sanderson, 2000).</p><p>SIR relies on the existence of a reference dictio­nary to perform WSD (typically, WordNet) and thus suffers from its static nature and its inherent paucity of most proper nouns. This latter problem is partic­ularly important for Web searches, as users tend to retrieve more information about named entities (e.g., singers, artists, cities) than concepts (e.g., abstract information about singers or artists).</p><p><b>Search Result Clustering. </b>A more popular ap­proach to query ambiguity is that of search result clustering. Typically, given a query, the system starts from a flat list of text snippets returned from one or more commonly-available search engines and clus­ters them on the basis of some notion of textual simi­larity. At the root of the clustering approach lies van Rijsbergen's (1979) cluster hypothesis: "closely as­sociated documents tend to be relevant to the same requests", whereas documents concerning different meanings of the input query are expected to belong to different clusters.</p><p>Approaches to search result clustering can be classified as data-centric or description-centric (Carpineto et al., 2009). The former focus more on the problem of data clustering than on presenting the results to the user. A pioneering example is Scat­ter/Gather (Cutting et al., 1992), which divides the dataset into a small number of clusters and, after the selection of a group, performs clustering again and proceeds iteratively. Developments of this approach have been proposed which improve on cluster qual­ity and retrieval performance (Ke et al., 2009). Other data-centric approaches use agglomerative hierar­chical clustering (e.g., LASSI (Yoelle Maarek and</p><doubt alpha="55.1" length="49" tooSmall="False" monospace="0.0">Pelleg, 2000)), rough sets (Ngo and Nguyen, 2005)</doubt><p>or exploit link information (Zhang et al., 2008). Description-centric approaches are, instead, more focused on the description to produce for each cluster of search results.<page local="3" global="118"/> Among the most popular and successful approaches are those based on suf­fix trees (Zamir et al., 1997; Zamir and Etzioni, 1998), including later developments (Crabtree et al., 2005; Bernardini et al., 2009). Other methods in the literature are based on formal concept analy­sis (Carpineto and Romano, 2004), singular value decomposition (Osinski and Weiss, 2005), spectral clustering (Cheng et al., 2005), spectral geometry and graph connectivity measures (Di Giacomo et al., 2007). Search result clustering has also been viewed as a supervised salient phrase ranking task (Zeng et al., 2004).</p><doubt alpha="50.9" length="55" tooSmall="False" monospace="0.0">(Liu et al., 2008), link analysis (Gelgi et al., 2007),</doubt><p><b>Diversification. </b>Another recent research topic dealing with the query ambiguity issue is diversifi­cation, which aims to rerank top search results based on criteria that maximize their diversity. One of the first examples of diversification algorithms is based on the use of similarity functions to measure the diversity among documents and between document and query (Carbonell and Goldstein, 1998). Other techniques use conditional probabilities to deter­mine which document is most different from higher-ranking ones (Chen and Karger, 2006) or use affinity ranking (Zhang et al., 2005), based on topic variance and coverage. More recently, an algorithm called Es­sential Pages (Swaminathan et al., 2009) has been proposed to reduce information redundancy and re­turn Web pages that maximize coverage with respect to the input query.</p><p><b>Word Sense Induction (WSI). </b>In contrast to the above approaches, we perform WSI to dynamically acquire an inventory of senses of the input query. Instead of performing clustering on the basis of the surface similarity of Web snippets, we use our in­duced word senses to group snippets. Very little work on this topic exists: vector-based WSI was suc­cessfully shown to improve bag-of-words ad-hoc In­formation Retrieval (Schütze and Pedersen, 1995) and preliminary studies (Udani et al., 2005; Chen et al., 2008) have provided interesting insights into the use of WSI for Web search result clustering. A more recent attempt at automatically identify­ing query meanings is based on the use of hidden topics (Nguyen et al., 2009). However, in this ap­proach topics - estimated from a universal dataset are query-independent and thus their number needs to be established beforehand. In contrast, we aim to cluster snippets based on a dynamic and finer-grained notion of sense.</p></section><section number="3" title="Approach"><p>Web search result clustering is usually performed in three main steps:</p><doubt alpha="64.1" length="103" tooSmall="False" monospace="0.0">1. Given a queryq,a search engine (e.g., Yahoo!) is used to retrieve a list of resultsR = (r\,..., rn);</doubt><p>2. A clustering <i>C </i><i>= (C0, C\,..., Cm) </i>of the results in <i>R </i>is obtained by means of a clustering algo­rithm;</p><p>3. The clusters in C are optionally labeled with an appropriate algorithm (e.g., see Zamir and Etzioni (1998) and Carmel et al. (2009)) for visualization purposes.</p><p>Our key idea is to improve step 2 by means of a Word Sense Induction algorithm: given a query <i>q, </i>we first dynamically induce, from a text corpus, the set of word senses of q (Section 3.1); next, we clus­ter the Web results on the basis of the word senses previously induced (Section 3.2).</p><subsection number="3.1" title="Word Sense Induction"><p>Word Sense Induction algorithms are unsupervised techniques aimed at automatically identifying the set of senses denoted by a word. These methods in­duce word senses from text by clustering word oc­currences based on the idea that a given word -used in a specific sense - tends to co-occur with the same neighbouring words (Harris, 1954). Several approaches to WSI have been proposed in the litera­ture (see Navigli (2009) for a survey), ranging from clustering based on context vectors (e.g., Schütze (1998)) to word clustering (e.g., Lin (1998)) and co-occurrence graphs (e.g., Widdows and Dorow (2002)).</p><p>Successful approaches such as HyperLex (Veronis, 2004) - a graph algorithm based on the identification of hubs in co-occurrence graphs -have to cope with a high number of parameters to be tuned (Agirre et al., 2006). To deal with this issue we propose two variants of a simple, yet effective, graph-based algorithm for WSI, that we describe hereafter.<page local="4" global="119"/> The algorithm consists of two steps: graph construction and identification of word senses.</p><subsubsection number="3.1.1" title="Graph construction"><p>Given a target query <i>q, </i>we build a co-occurrence graph <i>Gq = (V, E) </i>such that V is a set of context words related to q and E is the set of undirected edges, each denoting a co-occurrence between pairs of words in <i>V. </i>To determine the set of co-occurring words <i>V, </i>we use the Google Web1T corpus (Brants and Franz, 2006), a large collection of n-grams <i>(n = I,..., </i>5) - i.e., windows of n consecutive tokens -occurring in one terabyte of Web documents. First, for each content word w we collect the total num­ber c(w) of its occurrences and the number of times c(w, <i>w<footnote anchor="1"/>) </i>that w and w' occur together in any 5-gram (we include inflected forms in the count); second, we use the Dice coefficient to determine the strength of co-occurrence between w and w':</p><doubt alpha="30.0" length="10" tooSmall="False" monospace="0.0">')2c(w,w')</doubt><p><i>DlCe(w,w </i>) = <b>——r-</b><b>~.</b><b>-jr-</b>.</p><doubt alpha="29.4" length="17" tooSmall="False" monospace="0.0">v'    c(w)+ c(w')</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(1)</doubt><p>The rationale behind Dice is that dividing by the sum of total counts of the two words drastically de­creases the ranking of words that tend to co-occur frequently with many other words (e.g., new, <i>old, nice, </i>etc.).</p><p>The graph <i>G</i><i>q </i>= <i>(V, E) </i>is built as follows:</p><p>• Our initial vertex set V(0) contains all the con­tent words from the snippet results of query <i>q </i>(excluding stopwords); then, we add to V(0) the highest-ranking words co-occurring with <i>q </i>in the Web1T corpus, i.e., those words <i>w </i>for which <i>Dice(q, w) &gt; ö </i>(the threshold ö is established ex­perimentally, see Section 4.1). We set V := V(0)and E := 0.</p><p>• For each word w G V(0), we select the high­est ranking words co-occurring with <i>w </i>in Web1T, that is those words w' for which <i>Dice(w, w') &gt; ö. </i>We add each of these words to <i>V </i>(note that some w' might already be in V(0)) and the corresponding edge {w,w'} to E with weight <i>Dice(w,w'). </i>Finally, we remove disconnected vertices.</p></subsubsection><subsubsection number="3.1.2" title="Identification of word senses"><p>The main idea behind our approach is that edges in the co-occurrence graph participating in cycles are likely to connect vertices (i.e., words) belonging to the same meaning component. Specifically, we fo­cus on cycles of length 3 and 4, called respectively triangles and squares in graph theory.</p><p>For each edge e, we calculate the ratio of triangles in which e participates:</p><doubt alpha="63.2" length="38" tooSmall="False" monospace="0.0">. .        # triangleseparticipates in</doubt><doubt alpha="17.4" length="23" tooSmall="False" monospace="0.0">Tri(e) = -—---—-—-— (2)</doubt><p># triangles <i>e </i>could participate in where the numerator is the number of cycles of length 3 in which <i>e = { w, w'} </i>participates, and the denominator is the total number of neighbours of <i>w </i>and w'. Similarly, we define a measure <i>Sqr(e) </i>of the ratio of squares (i.e., cycles of length 4) an edge <i>e </i>participates in to the number of possible squares <i>e </i>could potentially participate in:</p><doubt alpha="50.0" length="8" tooSmall="False" monospace="0.0">Sqr(e) =</doubt><p># squares <i>e </i>participates in # squares <i>e </i>could participate in where the numerator is the number of squares con­taining <i>e </i>and the denominator is the number of pos­sible distinct pairs of neighbours of <i>w </i>and <i>w'.</i><i> </i>If no triangle (or square) exists for <i>e, </i>the value of the cor­responding function is set to 0.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(3)</doubt><p>In order to disconnect the graph and determine the meaning components, we remove all the edges whose <i>Tri </i>(or Sqr) value is below a threshold a. The resulting connected components represent the word senses induced for the query <i>q. </i>Notice that the num­ber of senses is dynamically chosen based on the co­occurrence graph and the algorithm's thresholds.</p><p>Our triangular measure is the edge counterpart of the clustering coefficient (or curvature) for ver­tices, previously used to perform WSI (Widdows and Dorow, 2002). However, it is our hunch that measuring the ratio of squares an edge participates in provides a stronger clue of how important that edge is within a meaning component. In Section 4, we will corroborate this idea with our experiments.</p></subsubsection><subsubsection number="3.1.3" title="An example"><p>As an example, let <i>q = beagle. </i>Two steps are per­formed:</p><page local="5" global="120"/><p>1. <b>Graph construction. </b>We build the co-occurrence graph <i>G </i><i>beagle </i><i>= </i>(V, E), an excerpt of which is shown in Figure 1(a).</p><p>2. <b>Identification of word senses. </b>We calculate the <i>Sqr </i>values of each edge in the graph. The edges e whose <i>Sqr(e) &lt; a </i>are removed (we assume <i>a = </i>0.25). For instance, Sqr({ <i>dog, breed </i>}) = <i>^</i>, as the edge participates in the square <i>dog </i>- <i>breed</i> - <i>puppy </i>- <i>canine </i>- <i>dog, </i>but it could also have participated in the potential square <i>dog </i>- <i>breed</i> - <i>puppy </i>- <i>search </i>- <i>dog.</i><i> </i>In fact, the other neigh­bours of <i>dog </i>are <i>canine, puppy </i>and <i>search, </i>and the other neighbour of <i>breed </i>is <i>puppy, </i>thus the square can only be closed by connecting <i>puppy </i>to either <i>canine </i>or <i>search. </i>In our example, the only edges whose <i>Sqr </i>is below <i>a </i>are: { <i>dog, puppy </i>} , { <i>dog, search </i>} and { <i>linux, mission </i>} (they participate in no square). We remove these edges and select the resulting connected compo­nents as the senses of the query <i>beagle </i>(shown in Figure 1(b)). Note that, if we selected triangles as our pruning measure, we should also remove the following edges { <i>search, index </i>}, { <i>index, linux </i>} , { <i>linux, system </i>} and { <i>system, search </i>} . In fact, these edges do not participate in any tri­angle (while they do participate in a square). As a result, we would miss the computer science sense of the query.</p></subsubsection></subsection><subsection number="3.2" title="Clustering of Web results"><p>Given our query <i>q, </i>we submit it to a search engine, which returns a list of relevant search results <i>R = (r\,... ,rn). </i>We process each result <i>ri </i>by consid­ering the corresponding text snippet and transform­ing it to a bag of words <i>bi </i>(we apply tokenization, stopwords and target word removal, and lemmatiza-tion<footnote anchor="2"/>). For instance, given the snippet:</p><p>"the <i>beagle </i>is a breed of medium-sized dog", we produce the following bag of words:</p><p>{ <i>breed, medium, size, dog </i>} .</p><p>As a result of the above processing, we obtain a list of bags of words <i>B = (b\,... ,bn)</i>. Now, our aim is to cluster our Web results <i>R, </i>i.e., the corre­sponding bags of words <i>B. </i>To this end, rather than</p><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">(a)</doubt><p>Inde^ <b>e</b><u>mission</u>^</p></subsection></section><section number="4" title="ander^"><p>spacecraft 'dogg) <b>cinde^ e</b><u>mission</u>^</p><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">(b)</doubt><doubt alpha="100.0" length="5" tooSmall="False" monospace="0.0">puppy</doubt></section><section number="4" title="andet^"><p>Figure 1: The <i>beagle </i>example: (a) graph construction, "weak" edges (according to Sqr) drawn in bold, (b) the word senses induced after edge removal.</p><p>considering the interrelationships between them (as is done in traditional search result clustering), we intersect each bag of words <i>bi </i><i>G </i><i>B </i>with the sense clusters <i>{S\,..., Sm} </i>acquired as a result of our Word Sense Induction algorithm (cf. Section 3.1). The sense cluster with the largest intersection with <i>b</i><i>i </i>is selected as the most likely meaning of <i>r</i><i>i. </i>For­mally:</p><doubt alpha="63.6" length="11" tooSmall="False" monospace="0.0">Sense(ri) =</doubt><doubt alpha="54.3" length="35" tooSmall="False" monospace="0.0">argmax |6j n Sj|if max |6j n Sj|&gt; 0</doubt><doubt alpha="30.0" length="10" tooSmall="False" monospace="0.0">j=1,...,mj</doubt><doubt alpha="66.7" length="6" tooSmall="False" monospace="0.0">0 else</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(4)</doubt><p>where 0 denotes that no sense is assigned to result <i>ri, </i>as the intersection is empty for all senses <i>Sj</i>. Oth­erwise the function returns the index of the sense having the largest overlap with <i>bi </i>- the bag of words associated with the search result <i>ri. </i>As a result of sense assignment for each <i>ri </i>G R, we obtain a clus­tering C = <i>(C0, </i><i>C\,...,</i><i> Cm) </i>such that:</p><doubt alpha="56.0" length="25" tooSmall="False" monospace="0.0">Cj={riGR : Sense(ri)= j},</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(5)</doubt><footnote label="2">We use the WordNet lemmatizer.</footnote><p>that is, <i>Cj</i><i> </i>contains the search results classified with the <i>j</i>-th sense of query <i>q</i><i> </i><i>(C</i>o includes unassigned results). Finally, we sort the clusters in our clus­tering C based on their "quality". For each cluster <i>Cj</i><i> </i><i>G C \ </i><i>{C0},</i><i> </i>we determine its similarity with the corresponding meaning <i>S</i><i>j </i>by calculating the fol­lowing formula:<page local="6" global="121"/></p><p><i>Y</i><i>^neC' </i><i>sim(r</i><i>i</i><i>,S</i><i>j)</i></p><doubt alpha="33.3" length="30" tooSmall="False" monospace="0.0">avgsim(Cj,Sj) =-1—3.    .-.(6)</doubt><doubt alpha="80.0" length="5" tooSmall="False" monospace="0.0">lCj I</doubt><p>The formula determines the average similarity be­tween the search results in cluster <i>C</i><i>j </i>and the corre­sponding sense cluster <i>S</i><i>j.</i><i> </i>The similarity between a search result <i>r</i><i>i </i>and <i>S</i><i>j </i>is determined as the normal­ized overlap between its bag of words <i>b</i><i>i</i><i> </i>and <i>S</i><i>j:</i></p><doubt alpha="44.7" length="38" tooSmall="False" monospace="0.0">sim(ri, Sj) = sim(b.j,,Sj) =^z^Sj1.(7)</doubt><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">IbiI</doubt><p>Finally, we rank the elements <i>r</i><i>i </i>within each clus­ter <i>C</i><i>j</i><i> </i>by their similarity <i>sim(r</i><i>i,</i><i> </i><i>S</i><i>j).</i><i> </i>We note that the ranking and optimality of clusters can be im­proved with more sophisticated techniques (Crabtree et al., 2005; Kurland, 2008; Kurland and Domshlak, 2008; Lee et al., 2008, inter alia). How­ever, this is outside the scope of this paper.</p></section><section number="4" title="Experiments"><subsection number="4.1" title="Experimental Setup"><p><b>Test Sets. </b>We conducted our experiments on two datasets:</p><p>• AMBIENT (AMBIguous ENTries), a recently</p><p>released dataset which contains 44 ambiguous queries<footnote anchor="3"/>. The sense inventory for the mean­ings (i.e., subtopics)<footnote anchor="4"/> of queries is given by Wikipedia disambiguation pages. For instance, given the <i>beagle </i>query, its disambiguation page in Wikipedia provides the meanings of dog, Mars lander, computer search service, beer brand, etc. The top 100 Web results of each query returned by the Yahoo! search engine were tagged with the most appropriate query senses according to Wikipedia (amounting to 4400 sense-annotated search results). To our knowledge, this is cur­rently the largest dataset of ambiguous queries available on-line. Other datasets, such as those from the TREC competitions, are not focused on distiguishing the subtopics of a query.</p><footnote label="3">http://credo.fub.it/ambient  4 In the following, we use the terms subtopic and word sense interchangeably.</footnote><p>• MORESQUE (MORE Sense-tagged QUEry re­sults), a new dataset of 114 ambiguous queries which we developed as a complement to AMBI­ENT following the guidelines provided by its au­thors. In fact, our aim was to study the behaviour of Web search algorithms on queries of differ­ent lengths, ranging from 1 to 4 words. How­ever, the AMBIENT dataset is composed mostly of single-word queries. MORESQUE provides dozens of queries of length 2, 3 and 4, together with the 100 top results from Yahoo! for each query annotated as in the AMBIENT dataset (overall, we tagged 11,400 snippets). We decided to carry on using Yahoo! mainly for homogeneity reasons.</p><p>We report the statistics on the composition of the two datasets in Table 1. Given that the snippets could possibly be annotated with more than one Wikipedia subtopic, we also determined the average number of subtopics per snippet. This amounted to 1.01 for AMBIENT and 1.04 for MORESQUE for snippets with at least one subtopic annotation (51% and 53% of the respective datasets). We can thus conclude that multiple subtopic annotations are infrequent.</p><p><b>Parameters. </b>Our graph-based algorithms have two parameters: the Dice threshold <i>ö </i>for graph construction (Section 3.1.1) and the threshold <i>a </i>for edge removal (Section 3.1.2). The best pa­rameters, used throughout our experiments, were <i>(ö</i><i> = 0.00033, a = 0.45) </i>with triangles and <i>(ö</i><i> </i><i>=</i><i> 0.00033, a = 0.33) </i>with squares. The parameter values were obtained as a result of tuning on a small in-house development dataset. The dataset was built by automatically identifying monosemous words and creating pseudowords following the scheme proposed by Schütze (1998).</p><p><b>Systems. </b>We compared Triangles and Squares against the best systems reported by Bernardini et<page local="7" global="122"/></p><doubt alpha="42.3" length="26" tooSmall="False" monospace="0.0">al. (2009, cf. Section 2):</doubt><table caption="Table 1: Statistics on the datasets ofambiguous queries." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>dataset</p></td><td class="cell"><p>queries</p></td><td class="cell"><p>queries by length</p></td><td class="cell"><p>avg.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p>2</p></td><td class="cell"><p>3</p></td><td class="cell"><p>4</p></td><td class="cell"><p>polys.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>AMBIENT</p></td><td class="cell"><p>44</p></td><td class="cell"><p>35</p></td><td class="cell"><p>6</p></td><td class="cell"><p>3</p></td><td class="cell"><p>0</p></td><td class="cell"><p>17.9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MORESQUE</p></td><td class="cell"><p>114</p></td><td class="cell"><p>0</p></td><td class="cell"><p>47</p></td><td class="cell"><p>36</p></td><td class="cell"><p>31</p></td><td class="cell"><p>6.7</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>• <b>Lingo </b>(Osinski and Weiss, 2005): a Web clus­tering engine implemented in the Carrot<footnote anchor="2"/> open-source framework<footnote anchor="5"/> that clusters the most frequent phrases extracted using suffix arrays.</p><p>• <b>Suffix Tree Clustering (STC) </b>(Zamir and Et-zioni, 1998): the original Web search clustering approach based on suffix trees.</p><p>• <b>KeySRC </b>(Bernardini et al., 2009): a state-of-the-art Web clustering engine built on top of STC with part-of-speech pruning and dynamic selection of the cut-off level of the clustering dendrogram.</p><p>• <b>Essential Pages (EP) </b>(Swaminathan et al., 2009): a recent diversification algorithm that selects fun­damental pages which maximize the amount of information covered for a given query.</p><p>• <b>Yahoo!: </b>the original search results returned by the Yahoo! search engine.</p><p>The first three of the above are Web search result clustering approaches, whereas the last two produce lists of possibly diversified results (cf. Section 2).</p></subsection><subsection number="4.2" title="Experiment 1: Clustering Quality"><p><b>Measure. </b>While assessing the quality of cluster­ing is a notably hard problem, given a gold standard <i>G </i>we can calculate the <b>Rand index </b>(RI) of a cluster­ing C, a common quality measure in the literature, determined as follows (Rand, 1971; Manning et al., where W is the union set of all the words in C and <i>ö(w, w') = 1 </i>if any two words <i>w </i>and w' are in the same cluster both in C and in the gold standard G or they are in two different clusters in both C and G, otherwise <i>ö(w, w') = 0.</i><i> </i>In other words, we calcu­late the percentage of word pairs that are in the same configuration in both C and G. For the gold standard G we use the clustering induced by the sense annota­tions provided in our datasets for each snippet (i.e., each cluster contains the snippets manually associ­ated with a particular Wikipedia subtopic). Similarly to what was done in Section 3.2, untagged results are grouped together in a special cluster of G.</p><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">2008):</doubt><doubt alpha="52.2" length="23" tooSmall="False" monospace="0.0">Y(w,w')eWxW,w=w&gt;ö(w,w')</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(8)</doubt><doubt alpha="34.6" length="26" tooSmall="False" monospace="0.0">I{(w,w') G W x W : w =w'}|</doubt><p><b>Results. </b>The results of all systems on the AM­BIENT and MORESQUE datasets according to the average Rand index are shown in Table 2<footnote anchor="6"/>. In accordance with previous results in the litera­ture, KeySRC performed generally better than the other search result clustering systems, especially on smaller queries. Our Word Sense Induction sys­tems, Squares and Triangles, outperformed all other systems by a large margin, thus showing a higher clustering quality (with the exception of KeySRC performing better than Triangles on AMBIENT). Interestingly, all clustering systems perform more poorly on longer queries (i.e., on the MORESQUE dataset), however our WSI systems, and especially Triangles, are more robust across query lengths. Compared to Triangles, the Squares algorithm per­forms better, confirming our hunch that Squares is a more solid graph pattern.</p></subsection><subsection number="4.3" title="Experiment 2: Diversification"><p><b>Measure. </b>Search result clustering can also be used to diversify the top results returned by a search en­gine. Thus, for each query <i>q, </i>one natural way of measuring a system's performance is to calculate the <b>subtopic recall-at</b>-K (Zhai et al., 2003) given by the number of different subtopics retrieved for <i>q </i>in the top <i>K </i>results returned:</p><p>I U<i>K==1</i><i> subtopics(n)I</i> where <i>subtopics(r</i><i>i)</i><i> </i>is the set of subtopics manually assigned to the search result <i>r</i><i>i </i>and <i>M </i>is the number of subtopics for query <i>q </i>(note that in our experiments <i>M </i>is the number of subtopics occurring in the 100 results retrieved for q, so S-recall@100 = 1). How­ever, this measure is only suitable for systems re­turning ranked lists (such as Yahoo! and EP). Given a clustering <i>C = </i>(C0, Ci,.<page local="8" global="123"/>.., <i>Cm), </i>we flatten it to a list as follows: we add to the initially empty list the first element of each cluster <i>C</i><i>j </i>(j <i>= </i>1,..., m); then we iterate the process by selecting the second ele­ment of each cluster Cj such that |Cj | &gt; 2, and so on. The remaining elements returned by the search engine, but not included in any cluster of C \ {C0}, are appended to the bottom of the list in their orig­inal order. Note that the elements are selected from each cluster according to their internal ranking (e.g., for our algorithms we use Formula 7 introduced in Section 3.2).</p><doubt alpha="66.7" length="12" tooSmall="False" monospace="0.0">S-recall@K =</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">M</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(9)</doubt><footnote>http://project.carrot2.org</footnote><footnote label="6">For reference systems we used the implementations of Bernardini et al. (2009) and Osinski and Weiss (2005).</footnote><table caption="Table 2: Results by Rand index (percentages)." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>System</p></td><td class="cell"><p>AMBIENT</p></td><td class="cell"><p>MORESQUE</p></td><td class="cell"><p>All</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Squares</p></td><td class="cell"><p>72.59</p></td><td class="cell"><p>65.41</p></td><td class="cell"><p>67.28</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Triangles</p></td><td class="cell"><p>66.13</p></td><td class="cell"><p>64.47</p></td><td class="cell"><p>64.93</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Lingo</p></td><td class="cell"><p>62.75</p></td><td class="cell"><p>52.68</p></td><td class="cell"><p>55.49</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>STC</p></td><td class="cell"><p>61.48</p></td><td class="cell"><p>51.52</p></td><td class="cell"><p>54.29</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>KeySRC</p></td><td class="cell"><p>66.49</p></td><td class="cell"><p>55.82</p></td><td class="cell"><p>58.78</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 3: S-recall@Kon all queries (percentages)."></table><p><b>Results. </b>For the sake of clarity and to save space, we selected the best systems from our previous ex­periment, namely Squares, Triangles and KeySRC, and compared their output with the original snippet list returned by Yahoo! and the output of the EP di­versification algorithm (cf. Section 4.1).</p><p>The S-recall@K (with K = 3, 5,10,15,20) cal­culated on AMBIENT+MORESQUE is reported in Table 3. Squares and Triangles show the high­est degree of diversification, with a subtopic recall greater than all other systems, and with Squares con­sistently performing better than Triangles. It is inter­esting to observe that KeySRC performs worse than Yahoo! with low values of K and generally better with higher values of K.</p><p>Given that the two datasets complement each other in terms of query lengths (with AMBIENT having queries of length &lt; 2 and MORESQUE with many queries of length &gt; 3), we studied the S-recall@K trend for the two datasets. The results are shown in Figures 2 and 3. While KeySRC does not show large differences in the presence of short and long ambiguous queries, our graph-based algorithms do. For instance, as soon as K = 3 the Squares al­gorithm obtains S-recall values of 37% and 57.5% on AMBIENT and MORESQUE, respectively. The</p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">i</doubt><doubt alpha="0.0" length="31" tooSmall="False" monospace="0.0">0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2</doubt><doubt alpha="20.0" length="5" tooSmall="False" monospace="0.0">0.i 0</doubt><doubt alpha="0.0" length="20" tooSmall="False" monospace="0.0">30   40   50   60 70</doubt><p>number of results K</p><doubt alpha="9.1" length="11" tooSmall="False" monospace="0.0">80   90 i00</doubt><figure caption="Figure 2: Results by S-recall@Kon AMBIENT."></figure><doubt alpha="3.2" length="31" tooSmall="False" monospace="0.0">0.9 0.8 0.7 0.6 0.5 0.4 0.3h0.2</doubt><doubt alpha="19.5" length="82" tooSmall="False" monospace="0.0">10     20     30     40     50     60     70     80     90 100 number of results K</doubt><figure caption="Figure 3: S-recall@Kon MORESQUE."></figure><p>difference decreases as K increases, but is still sig­nificant when K = 10. We hypothesize that, because they are less ambiguous, longer queries are easier to diversify with the aid of WSI. However, we note that, even with low values of K, Squares and Tri­angles obtain higher S-recall than the other systems (with KeySRC competing on AMBIENT when K &lt; 15). Finally, we observe that - with low values of K - the Squares algorithm performs significantly better than Triangles on shorter queries, and only slightly better on longer ones.</p></subsection></section><section number="5" title="Discussion"><p><b>Results. </b>Our results show that our graph-based al­gorithms are able to consistently produce clusters of better quality than all other systems tested in our experiments. The results on S-recall@K show that our approach can also be used effectively as a diver­sification technique, performing better than a very recent proposal such as Essential Pages.<page local="9" global="124"/> The lat­ter outperforms Yahoo! and KeySRC when K &gt; 30 on AMBIENT, whereas on MORESQUE it per­forms generally worse until higher values of K are reached. If we analyze the entire dataset of 158 queries by length, EP works best after examining at least 20 results on 1- and 2-word ambiguous queries, whereas on longer queries a larger number of docu­ments (&gt; 30) needs to be analyzed before surpassing Yahoo! performance.</p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">o</doubt><doubt alpha="20.0" length="5" tooSmall="False" monospace="0.0">i0 20</doubt><table caption="Table 3: S-recall@K on all queries (percentages)." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>System</p></td><td class="cell"><p>K=3</p></td><td class="cell"><p>K=5</p></td><td class="cell"><p>K=10</p></td><td class="cell"><p>K=15</p></td><td class="cell"><p>K=20</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Squares</p></td><td class="cell"><p>51.9</p></td><td class="cell"><p>63.4</p></td><td class="cell"><p>75.8</p></td><td class="cell"><p>83.3</p></td><td class="cell"><p>87.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Triangles</p></td><td class="cell"><p>50.8</p></td><td class="cell"><p>62.4</p></td><td class="cell"><p>75.2</p></td><td class="cell"><p>82.7</p></td><td class="cell"><p>86.6</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Yahoo!</p></td><td class="cell"><p>49.2</p></td><td class="cell"><p>60.0</p></td><td class="cell"><p>72.9</p></td><td class="cell"><p>78.5</p></td><td class="cell"><p>82.7</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>EP</p></td><td class="cell"><p>40.6</p></td><td class="cell"><p>53.2</p></td><td class="cell"><p>68.6</p></td><td class="cell"><p>77.2</p></td><td class="cell"><p>83.3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>KeySRC</p></td><td class="cell"><p>44.3</p></td><td class="cell"><p>55.8</p></td><td class="cell"><p>72.0</p></td><td class="cell"><p>79.1</p></td><td class="cell"><p>83.2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>llll........'</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>- ss^</i></p></td><td class="cell"><p><b><i>•***</i></b></p><p>Squares Triangles Yahoo! "</p><p>EP * Keysrc ♦</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b><i>ft/</i></b></p><p><i>- </i><b>•&lt;+</b></p><p>•V</p><p><b>_ ^ /</b></p><p><b>x+</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>■+</b></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>-</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>The above considerations might not seem intu­itive at first glance, as the average polysemy of longer queries is lower (17.9 on AMBIENT vs. 6.7 on MORESQUE according to our gold standard). However, we note that while the kind of ambigu­ity of 1-word queries is generally coarser (e.g., <i>bea­gle </i>as dog vs. lander vs. search tool), with longer queries we often encounter much finer sense distinc­tions (e.g., <i>Across the Universe </i>as song by The Bea­tles vs. a 2007 film based on the song vs. a Star Trek novel vs. a rock album by Trip Shakespeare, etc.). Word Sense Induction is able to deal better with this latter kind of ambiguity as discriminative words be­come part of the meanings acquired.</p><p><b>Performance issues. </b>Inducing word senses from the query graph comes at a higher computational cost than other non-semantic clustering techniques. Indeed, the most time-consuming phase of our ap­proach is the construction of the query graph, which requires intensive querying of our database of co­occurrences calculated from the Web1T corpus. While graphs can be precomputed or cached, previ­ously unseen queries will still require the construc­tion of new graphs. Instead, triangles and squares, as well as the resulting connected components, can be calculated on the fly.</p></section><section number="6" title="Conclusions"><p>In this paper we have presented a novel approach to Web search result clustering. Our key idea is to induce senses for the target query automatically by means of a graph-based algorithm focused on the notion of cycles. The results of a Web search engine are then mapped to the query senses and clustered accordingly.</p><p>The paper provides three novel contributions. First, we show that WSI boosts the quality of search result clustering and improves the diversification of the snippets returned as a flat list. We provide a clear indication on the usefulness of a loose notion of sense to cope with ambiguous queries. This is in contrast to research on Semantic Information Re­trieval, which has obtained contradictory and often inconclusive results. The main advantage of WSI lies in its dynamic production of word senses that cover both concepts (e.g., <i>beagle </i>as a breed of dog) and instances (e.g., <i>beagle </i>as a specific instance of a space lander). In contrast, static dictionaries such as WordNet - typically used in Word Sense Dis­ambiguation - by their very nature encode mainly concepts. Second, we propose two simple, yet ef­fective, graph algorithms to induce the senses of our queries. The best performing approach is based on squares (cycles of length 4), a novel graph pat­tern in WSI. Third, we contribute a new dataset of 114 ambiguous queries and 11,400 sense-annotated snippets which complements an existing dataset of ambiguous queries<footnote anchor="7"/>. Given the lack of ambiguous query datasets available (Sanderson, 2008), we hope our new dataset will be useful in future compara­tive experiments. Finally, we note that our approach needed very little tuning. Moreover, its requirement of a Web corpus of n-grams is not a stringent one, as such corpora are available for several languages and can be produced for any language of interest.</p><p>As regards future work, we intend to combine our clustering algorithm with a cluster labeling al­gorithm. We also aim to implement a number of Word Sense Induction algorithms and compare them in the same evaluation framework with more Web search and Web clustering engines. Finally, it should be possible to use precisely the same approach pre­sented in this paper for document clustering, by grouping the contexts in which the target query oc­curs - and we will also experiment on this in the future.</p></section><section title="Acknowledgments"><p>We thank Google for providing the Web1T corpus for research purposes. We also thank Massimiliano D'Amico for producing the output of KeySRC and EP, and Stanislaw Osinski and Dawid Weiss for their help with Lingo and STC.<page local="10" global="125"/> Additional thanks go to Jim McManus, Senja Pollak and the anonymous re­viewers for their useful comments.</p><footnote label="7">The MORESQUE dataset is available at the following URL: http://lcl.uniroma1.it/moresque</footnote></section><references><p>Eneko Agirre, David Martinez, Oier Lopez de Lacalle, and Aitor Soroa. 2006. Evaluating and optimizing the parameters of an unsupervised graph-based WSD algorithm. In <i>Proc. of TextGraphs '06, </i>pages 89-96, New York, USA.</p><p>Paul N. Bennett and Nam Nguyen. 2009. Refined ex­perts: improving classification in large taxonomies. In <i>Proc. ofSIGIR '09, </i>pages 11-18, Boston, MA, USA.</p><p>Andrea Bernardini, Claudio Carpineto, and Massimil-iano D'Amico. 2009. Full-subtopic retrieval with keyphrase-based search results clustering. In <i>Proc. of WI '09, </i>pages 206-213, Milan, Italy.</p><p>Thorsten Brants and Alex Franz. 2006. Web 1t 5-gram, ver. 1, ldc2006t13. In <i>LDC, </i>PA, USA.</p><p>Jaime Carbonell and Jade Goldstein. 1998. The use of mmr, diversity-based reranking for reordering docu­ments and producing summaries. In <i>Proc. ofSIGIR '98, </i>pages 335-336, Melbourne, Australia.</p><p>David Carmel, Haggai Roitman, and Naama Zwerdling. 2009. Enhancing cluster labeling using Wikipedia. In <i>Proc. ofSIGIR '09, </i>pages 139-146, MA, USA.</p><p>Claudio Carpineto and Giovanni Romano. 2004. Ex­ploiting the potential of concept lattices for informa­tion retrieval with CREDO. <i>Journal of Universal Computer Science, </i>10(8):985-1013.</p><p>Claudio Carpineto, Stanislaw Osmski, Giovanni Ro­mano, and Dawid Weiss. 2009. A survey of web clus­tering engines. <i>ACM Computing Surveys, </i>41(3):1-38.</p><p>Harr Chen and David R. Karger. 2006. Less is more: probabilistic models for retrieving fewer relevant doc­uments. In <i>Proc. ofSIGIR 06, </i>pages 429-436, Seat­tle, WA, USA.</p><p>Jiyang Chen, Osmar R. Zaiane, and Randy Goebel. 2008. An unsupervised approach to cluster web search re­sults based on word sense communities. In <i>Proc. of WI-IAT2008, </i>pages 725-729, Sydney, Australia.</p><p>David Cheng, Santosh Vempala, Ravi Kannan, and Grant Wang. 2005. A divide-and-merge methodology for clustering.   In <i>Proc. of PODS '05, </i>pages 196-205,</p><p>New York, NY, USA.</p><p>Daniel Crabtree, Xiaoying Gao, and Peter Andreae. 2005. Improving web clustering by cluster selec­tion. In <i>Proc. ofWI '05, </i>pages 172-178, Compiegne, France.</p><p>Douglass R. Cutting, David R. Karger, Jan O. Pedersen, and John W. Tukey. 1992. Scatter/gather: A cluster-based approach to browsing large document collec­tions. In <i>Proc. ofSIGIR '92, </i>pages 318-329, Copen­hagen, Denmark. Emilio Di Giacomo, Walter Didimo, Luca Grilli, and Giuseppe Liotta. 2007. Graph visualization tech­niques for web clustering engines. <i>IEEE Transactions on Visualization and Computer Graphics, </i>13(2):294304.</p><p>George W. Furnas, Thomas K. Landauer, Louis Gomez, and Susan Dumais. 1987. The vocabulary problem in human-system communication. <i>Communications of the ACM, </i>30(11):964-971. Fatih Gelgi, Hasan Davulcu, and Srinivas Vadrevu. 2007. Term ranking for clustering web search results. In <i>Proc. ofWebDB '07, </i>Beijing, China. Julio Gonzalo, Anselmo Penas, and Felisa Verdejo. 1999. Lexical ambiguity and Information Retrieval revisited. In <i>Proc. of EMNLP/VLC 1999, </i>pages 195-202, Col­lege Park, MD, USA.</p><p>Zellig Harris. 1954. Distributional structure. <i>Word, </i>10:146-162.</p><p>Maryam Kamvar and Shumeet Baluja. 2006. A large scale study of wireless search behavior: Google mo­bile search. In <i>Proc. of CHI '06, </i>pages 701-709, New</p><p>York, NY, USA.</p><p>Weimao Ke, Cassidy R. Sugimoto, and Javed Mostafa. 2009. Dynamicity vs. effectiveness: studying online clustering for scatter/gather. In <i>Proc. ofSIGIR 09, </i>pages 19-26, MA, USA. Sang-Bum Kim, Hee-Cheol Seo, and Hae-Chang Rim. 2004. Information Retrieval using word senses: root sense tagging approach. In <i>Proc. ofSIGIR '04, </i>pages 258-265, Sheffield, UK. Robert Krovetz and William B. Croft. 1992. Lexical am­biguity and Information Retrieval. <i>ACM Transactions on Information Systems, </i>10(2):115-141. Oren Kurland and Carmel Domshlak.  2008.  A rank­aggregation approach to searching for optimal query-specific clusters. In <i>Proc. of SIGIR '08, </i>pages 547­554, Singapore. Oren Kurland. 2008. The opposite of smoothing: a lan­guage model approach to ranking query-specific doc­ument clusters. In <i>Proc. ofSIGIR 08, </i>pages 171-178, Singapore.</p><p>Kyung Soon Lee, W. Bruce Croft, and James Allan. 2008. A cluster-based resampling method for pseudo-relevance feedback. In <i>Proc. ofSIGIR 08, </i>pages 235­242, Singapore. Dekang Lin. 1998. Automatic retrieval and clustering of similar words. In <i>Proc. of the 17th COLING, </i>pages 768-774, Montreal, Canada.</p><page local="11" global="126"/><p>Shuang Liu, Clement Yu, and Weiyi Meng. 2005a. Word Sense Disambiguation in queries. In <i>Proc. of CIKM 05, </i>pages 525-532, Bremen, Germany.</p><p>Tie-Yan Liu, Yiming Yang, Hao Wan, Hua-Jun Zeng, Zheng Chen, and Wei-Ying Ma. 2005b. Support vec­tor machines classification with a very large-scale tax­onomy. <i>SIGKDD Explor. Newsl., </i>7(1):36-43.</p><p>Ying Liu, Wenyuan Li, Yongjing Lin, and Liping Jing. 2008. Spectral geometry for simultaneously clustering and ranking query search results. In <i>Proc. ofSIGIR 08, </i>pages 539-546, Singapore.</p><p>Rila Mandala, Takenobu Tokunaga, and Hozumi Tanaka. 1998. The use of WordNet in Information Retrieval. In <i>Proc. of the COLING-ACL workshop on Usage of Wordnet in Natural Language Processing, </i>pages 31­37, Montreal, Canada.</p><p>Christopher D. Manning, Prabhakar Raghavan, and Hin-rich Schutze. 2008. <i>Introduction to Information Re­trieval. </i>Cambridge University Press, New York, NY,</p><p>USA.</p><p>George A. Miller, Richard T. Beckwith, Christiane D. Fellbaum, Derek Gross, and Katherine Miller. 1990. WordNet: an online lexical database. <i>International Journal ofLexicography, </i>3(4):235-244.</p><p>Roberto Navigli. 2009. Word Sense Disambiguation: a survey. <i>ACM Computing Surveys, </i>41(2):1-69.</p><p>Chi Lang Ngo and Hung Son Nguyen. 2005. A method of web search result clustering based on rough sets. In <i>Proc. ofWI '05, </i>pages 673-679, Compiegne, France.</p><p>Cam-Tu Nguyen, Xuan-Hieu Phan, Susumu Horiguchi, Thu-Trang Nguyen, and Quang-Thuy Ha. 2009. Web search clustering and labeling with hidden top­ics. <i>ACM Transactions on Asian Language Informa­tion Processing, </i>8(3):1-40.</p><p>Stanislaw Osinski and Dawid Weiss. 2005. A concept-driven algorithm for clustering search results. <i>IEEE Intelligent Systems, </i>20(3):48-54.</p><p>William M. Rand. 1971. Objective criteria for the eval­uation of clustering methods. <i>Journal ofthe American Statistical Association, </i>66(336):846-850.</p><p>Mark Sanderson. 1994. Word Sense Disambiguation and Information Retrieval. In <i>Proc. ofSIGIR 94, </i>pages 142-151, Dublin, Ireland.</p><p>Mark Sanderson. 2000. Retrieving with good sense. <i>In­formation Retrieval, </i>2(1):49-69.</p><p>Mark Sanderson. 2008. Ambiguous queries: test collec­tions need more sense. In <i>Proc. ofSIGIR 08, </i>pages 499-506, Singapore.</p><p>Hinrich Schuitze and Jan Pedersen. 1995. Information Retrieval based on word senses. In <i>Proceedings of SDAIR '95, </i>pages 161-175, Las Vegas, Nevada, USA.</p><p>Hinrich Schuitze. 1998. Automatic word sense discrimi­nation. <i>Computational Linguistics, </i>24(1):97-124.</p><p>Christopher Stokoe, Michael J. Oakes, and John I. Tait. 2003. Word Sense Disambiguation in Information Re­trieval revisited. In <i>Proc. of SIGIR '03, </i>pages 159­166, Canada.</p><p>Ashwin Swaminathan, Cherian V. Mathew, and Darko Kirovski. 2009. Essential pages. In <i>Proc. ofWI 09, </i>pages 173-182, Milan, Italy.</p><p>Goldee Udani, Shachi Dave, Anthony Davis, and Tim Sibley. 2005. Noun sense induction using web search results. In <i>Proc. ofSIGIR 05, </i>pages 657-658, Sal­vador, Brazil.</p><p>Cornelis Joost van Rijsbergen. 1979. <i>Information Re­trieval. </i>Butterworths, second edition.</p><p>Jean Veronis. 2004. HyperLex: lexical cartography for Information Retrieval. <i>Computer Speech and Lan­guage, </i>18(3):223-252.</p><p>Ellen M. Voorhees. 1993. Using WordNet to disam-biguate word senses for text retrieval. In <i>Proc. ofSI-GIR '93, </i>pages 171-180, Pittsburgh, PA, USA.</p><p>Dominic Widdows and Beate Dorow. 2002. A graph model for unsupervised lexical acquisition. In <i>Proc. of the </i>19th <i>COLING, </i>pages 1-7, Taipei, Taiwan.</p><p>Gui-Rong Xue, Dikan Xing, Qiang Yang, and Yong Yu. 2008. Deep classification in large-scale text hierar­chies. In <i>Proc. ofSIGIR ' 08, </i>pages 619-626, Singa­pore.</p><p>Israel Ben-Shaul Yoelle Maarek, Ron Fagin and Dan Pel-leg. 2000. Ephemeral document clustering for web applications. <i>IBM Research Report RJ10186.</i></p><p>Oren Zamir and Oren Etzioni. 1998. Web document clustering: a feasibility demonstration. In <i>Proc. ofSI-GIR 98, </i>pages 46-54, Melbourne, Australia.</p><p>Oren Zamir, Oren Etzioni, Omid Madani, and Richard M. Karp. 1997. Fast and intuitive clustering of web docu­ments. In <i>Proc. ofKDD 97, </i>pages 287-290, Newport Beach, California.</p><p>Hua-Jun Zeng, Qi-Cai He, Zheng Chen, Wei-Ying Ma, and Jinwen Ma. 2004. Learning to cluster web search results. In <i>Proc. ofSIGIR '04, </i>pages 210-217, Sheffield, UK.</p><p>ChengXiang Zhai, William W. Cohen, and John Lafferty. 2003. Beyond independent relevance: Methods and evaluation metrics for subtopic retrieval. In <i>Proc. of SIGIR '03, </i>pages 10-17, Toronto, Canada.</p><p>Benyu Zhang, Hua Li, Yi Liu, Lei Ji, Wensi Xi, Weiguo Fan, Zheng Chen, and Wei-Ying Ma. 2005. Improv­ing web search results using affinity graph. In <i>Proc. of SIGIR '05, </i>pages 504-511, Salvador, Brazil.</p><p>Xiaodan Zhang, Xiaohua Hu, and Xiaohua Zhou. 2008. A comparative evaluation of different link types on en­hancing document clustering. In <i>Proc. ofSIGIR 08, </i>pages 555-562, Singapore.</p></references></body></article>