<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="1037"/><title>Relevance Feedback using Latent Information</title><pubinfo>Proceedings of the 5th International Joint Conference on Natural Language Processing,pages 1037-1045, Chiang Mai, Thailand, November 8 - 13, 2011. ©2011 AFNLP</pubinfo><author surname="Harashima" givenname="Jun"><org  name="Kyoto University" country="Japan" city="Kyoto"/></author><author surname="Kurohashi" givenname="Sadao"><org  name="Kyoto University" country="Japan" city="Kyoto"/></author></firstpageheader><frontmatter><p><b>Relevance Feedback using Latent Information</b></p><p><b>Jun Harashima Sadao Kurohashi</b></p><p>Graduate School of Informatics, Graduate School of Informatics,</p><p>Kyoto University, Kyoto University,</p><p>Yoshida-honmachi, Sakyo-ku, Yoshida-honmachi, Sakyo-ku,</p><p>Kyoto, 606-8501, Japan Kyoto, 606-8501, Japan</p><p>harashima@nlp.kuee.kyoto-u.ac.jp kuro@i.kyoto-u.ac.jp</p></frontmatter><abstract>We present a novel relevance feedback (RF) method that uses not only the sur­face information in texts, but also the la­tent information contained therein. In the proposed method, we infer the latent topic distribution in user feedback and in each document in the search results using latent Dirichlet allocation, and then we modify the search results so that documents with a similar topic distribution to that of the feedback are re-ranked higher. Evaluation results show that our method is effective for both explicit and pseudo RF, and that it has the advantage of performing well even when only a small amount of user feed­back is available. </abstract></header><body><section number="1" title="Introduction"><p>The main purpose of information retrieval (IR) is to provide the user with documents that are rele­vant to his/her information needs. However, it is difficult to achieve this by one-off retrieval, since user queries are typically short and often ambigu­ous (Jansen et al., 2000).</p><p>Relevance feedback (RF) is a technique to solve this problem. The basic procedure of RF is as fol­lows. First, a system obtains initial search results for a given query, and presents them to the user. The user then annotates some of the documents in the search results as being relevant or not, and the system modifies the search results using this feed­back.</p><p>There are a variety of RF methods that depend on different retrieval models. Rocchio's algorithm (Rocchio, 1971) and the Ide dec-hi method (Ide, 1971) are well-known RF methods for the vector space model (Salton et al., 1975). In the prob­abilistic model (Sparck Jones et al., 2000), the weight of terms can be modified by feedback. For language modeling approaches (Ponte and Croft, 1998), Zhai and Lafferty (2001) proposed a fun­damental RF method.</p><p>As described above, many methods have been proposed for RF. However, most of the previous methods use only the surface information in texts. That is, they ignore the latent information in texts, which could assist in improving IR performance. For example, they do not and cannot use the infor­mation of words for RF that do not appear in user feedback even if these words are highly probable from the latent topics of the feedback.</p><p>In this paper, we explore a novel RF method for language modeling approaches. In the proposed method, we use not only the surface information in texts, but also the latent information contained therein. More specifically, we infer the latent topic distribution in user feedback and in each document in the search results using latent Dirichlet alloca­tion (LDA), and then we modify the search results so that documents with a similar topic distribution to that of the feedback are re-ranked higher. Evalu­ation results show that our method is effective for both explicit and pseudo RF, and that it has the advantage of performing well even when only a small amount of user feedback is available.</p></section><section number="2" title="Language Modeling Approaches to IR"><p>In this section, we describe the language model­ing approaches to IR that form the basis of our method.</p><subsection number="2.1" title="Overview"><p>Language modeling approaches can be classi­fied into three types: the query likelihood model (Ponte and Croft, 1998), the document likeli­hood model (Lavrenko and Croft, 2001), and the Kullback-Leibler (KL) divergence retrieval model (Lafferty and Zhai, 2001).<page local="2" global="1038"/> In the query likelihood model, a document language model is constructed for each document in the collection. When a query is submitted by a user, the query likelihood is com­puted using the document model for each docu­ment. Then, the documents in the collection are ranked according to their likelihoods. In the docu­ment likelihood model, a query language model is constructed for a given query, and this is then used to compute the document likelihood for each doc­ument in the collection. The documents are then ranked by their likelihoods. In the KL-divergence retrieval model, both a query model and a docu­ment model are constructed, and the documents in the collection are ranked according to the KL-divergence between these models.</p></subsection><subsection number="2.2" title="Language Model Construction"><p>There are several ways of constructing a query model and a document model. One method is maximum likelihood estimation (MLE). The MLE of a word <i>Wj</i><i> </i>with respect to a text t (e.g., query, document) is computed as where tf (w<b>j</b>, t) represents the frequency of W<b>j </b>in</p><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">MLE</doubt><doubt alpha="40.0" length="5" tooSmall="False" monospace="0.0">(wj )</doubt><doubt alpha="20.0" length="5" tooSmall="False" monospace="0.0">|t| '</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(1)</doubt><p>t.</p><p>Dirichlet smoothed estimation (DIR) (Zhai and Lafferty, 2004) is also a well-known construction method. The DIR of W<b>j </b>with respect to t is com­puted as follows.</p><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">DIR</doubt><doubt alpha="47.6" length="21" tooSmall="False" monospace="0.0">tf (Wj,t)+ßP^f(|t| +ß</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(2)</doubt><p>where Da11 represents a collection, and ß repre­sents the smoothing parameter that controls the de­gree of confidence in the frequency in Da11 rather than in the frequency in t.</p></subsection><subsection number="2.3" title="RF for Language Modeling Approaches"><p>Zhai and Lafferty proposed a fundamental RF method for the language modeling approaches (Zhai and Lafferty, 2001). When user feedback is given, they construct a language model for the feedback. Then, a new query model is constructed by interpolating the feedback model with the orig­inal query model, which is used to obtain the ini­tial search results. Finally, they modify the search results using the new query model. They show the effectiveness of their method through their exper­iments, and report that the performance is better than that of Rocchio's algorithm.</p><doubt alpha="60.0" length="5" tooSmall="False" monospace="0.0">3 LDA</doubt><p>In this section, we explain LDA, which is em­ployed in the proposed method.</p></subsection><subsection number="3.1" title="Overview"><p>LDA (Blei et al., 2003) is one of the most popular topic models, and is viewed as an Bayesian ex­tension of Probabilistic Latent Semantic Indexing (PLSI) (Hofmann, 1999). In PLSI, it is assumed that each document has a unique topic proportion 6 <b><i>= </i></b><i>(9\,</i>..., <i>9K</i>). In contrast, LDA posits that 6 can take any values in the <i>(K</i><i> </i><i>— </i>1) simplex, a topic proportion that means a point of the sim­plex is drawn from a Dirichlet distribution Dir(a). Note that the parameter <b><i>a </i></b>is a K-vector with com­ponents <i>ak &gt; </i>0. In LDA, the probability of a document d» in the training data is calculated as follows.</p><doubt alpha="45.5" length="11" tooSmall="False" monospace="0.0">P (di|a, ß)</doubt><doubt alpha="33.3" length="30" tooSmall="False" monospace="0.0">=)P(6|a) P(wjI**'ß) P(**|6)J J</doubt><doubt alpha="50.0" length="2" tooSmall="False" monospace="0.0">d6</doubt><p>where <i>zk</i><i>(k</i><i> </i><b><i>= </i></b>1,..., K) represents a topic, and ß <b><i>=</i></b><b><i> </i></b><b><i>(ß</i></b><b><i>i,</i></b><i>..., </i><b><i>ßK</i></b><b><i>)</i></b><b><i> </i></b>represents the distributions over words for each topic zk.</p></subsection><subsection number="3.2" title="Parameter Estimation"><p>In LDA, the expectation-maximization algorithm cannot be used to estimate the parameters, since the computation of the posterior distribution of la­tent variables is intractable. Thus, a wide vari­ety of techniques using the variational method and Gibbs sampling, have been proposed to estimate the parameters (Blei et al., 2003; Griffiths and Steyvers, 2004). Here, we explain the technique using the variational method, as this is employed in the proposed method.</p><p>First, variational parameters <b><i>y% </i></b><b><i>= </i></b><i>(Y</i><b><i>ii,</i></b><i>Y</i><b><i>iK ) </i></b>and 0» = <b><i>(&lt;f</i></b><b><i>&gt;n,</i></b>..., <b><i>0»j </i></b>) are introduced for each document d» in the training data. Then, the optimal values of these are found by repeatedly computing the following pair of update equations:<page local="3" global="1039"/></p><doubt alpha="46.2" length="26" tooSmall="False" monospace="0.0">&lt;/&gt;ijfc   «ßkjexp *(7fc)—*</doubt><doubt alpha="75.0" length="4" tooSmall="False" monospace="0.0">7ifc</doubt><doubt alpha="25.0" length="8" tooSmall="False" monospace="0.0">^ k'=i '</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">J</doubt><doubt alpha="45.8" length="24" tooSmall="False" monospace="0.0">ak + ^ ^ijktf (wj,di)(4)</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">j=i</doubt><p>where * is the first derivative of the log r func­tion.</p><p>Next, a and ß are updated using and 0i for each d<b>i. </b>In the original paper, a Newton-Raphson method was used to estimate a (Blei et al., 2003). However, estimation with the Newton-Raphson method has the disadvantages that it takes a long time and each estimated <i>ak </i>can be a negative value under certain circumstances. It is known that the fixed-point iteration method (Minka, 2000) is a better estimation technique, and hence, we present the update equations based on this method. The update equations for a and ß are given below.</p><doubt alpha="54.2" length="24" tooSmall="False" monospace="0.0">ßkj    «X]^ijktf (wj,di)</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">i=i</doubt><doubt alpha="53.8" length="26" tooSmall="False" monospace="0.0">aEI=l(*(«k+nik)—*(ak)}aold</doubt><doubt alpha="40.0" length="25" tooSmall="False" monospace="0.0">EI=i{*(ac+|di|) — *(ao)}k</doubt><p>where <i>n</i><b><i>ik </i></b>= <i>Y,'jJ=l </i><i>4</i><b><i>&gt;ijk </i></b><i>tf</i><i> </i><i>(w</i><b><i>j</i></b><i>,</i><i> </i><b><i>d</i></b><b><i>i), </i></b><i>a</i><b><i>o </i></b>= <i>Y^K'=1</i><i> </i><i>ak</i>, and <b>akld </b>represents <b>ak </b>before the up­date.</p><p>Finally, the updates of and 0i for each di and those of a and ß are iterated until convergence. Once all the parameters have been estimated, we can obtain the probability of a word <b>wj </b>given a document d<b>i </b>as</p><doubt alpha="58.3" length="12" tooSmall="False" monospace="0.0">PdjDA(wj ) *</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(5)</doubt></subsection><subsection number="3.3" title="Inference of Unseen Texts"><p>One major advantage of LDA over PLSI is that it has a natural way of inferring the probabilities of unseen texts, which are not included in the training data. When we compute the probabilities of an unseen text t, the variational parameters jt and <b><i>4&gt;t</i></b><b><i> </i></b>are estimated using Eqs.(3) and (4). Then, for example, the probabilities of words given t can be obtained using Eq.(5).</p></subsection><subsection number="3.4" title="LDA in IR"><p>Certain works using LDA for IR are closely re­lated to our work. Wei and Croft (2006) incor­porate LDA into a query likelihood model, while</p><p>Zhou and Wade's work can be viewed as a study that incorporates LDA into a KL-divergence re­trieval model (Zhou and Wade, 2009). Such works successfully utilize the latent information in texts through LDA, and report that the latent informa­tion is effective for ad-hoc retrieval. Although there are many differences between our work and those mentioned above, one of the biggest differ­ences is that whereas the other works explored the effectiveness of the latent information for ad-hoc retrieval, we explore it for RF beyond ad-hoc re­trieval.</p></subsection></section><section number="4" title="Proposed Method"><subsection number="4.1" title="Overview"><p>An overview of the proposed method is illustrated in Figure 1. First, when a query is submitted by a user, we obtain the initial search results (Step 1). Next, for each document in the search results, we construct a hybrid language model that contains not only the surface information, but also the la­tent information in the document (Step 2). Then, when user feedback is given, we also construct a hybrid language model for it (Step 3). Finally, we construct a new query model by interpolating the original query model with the feedback model. We also re-rank the initial search results using this new model so that documents with a similar topic dis­tribution to that of the user feedback are re-ranked higher (Step 4). In the following subsections, we describe each step in detail.</p></subsection><subsection number="4.2" title="Acquisition of Initial Search Results"><p>In the proposed method, we employ a KL-divergence retrieval model (Lafferty and Zhai, 2001) to obtain the initial search results for a given query. First, we construct the MLE-based query model <i>P'MLE</i>(•) for a query q using Eq.(1). Then, for each document containing q in the collection, the KL-divergence between the DIR-based docu­ment model and the MLE-based query model is computed. That is, the score of a document d for a query q is defined as follows.</p><p><i>initialscore</i><b><i>(d' q)</i></b></p><doubt alpha="52.2" length="23" tooSmall="False" monospace="0.0">-KL(PqMLE(•)||PdDIR(^))</doubt><p>Finally, the initial search results <b><i>Dq</i></b><b><i> </i></b>= (dl,..., d|Dq |) are obtained by ranking the docu­ments according to their scores.</p><doubt alpha="12.5" length="8" tooSmall="False" monospace="0.0">7k'))(3)</doubt><page local="4" global="1040"/><p>Step 1. Acquisition of Initial Search Results</p><doubt alpha="100.0" length="5" tooSmall="False" monospace="0.0">query</doubt><footnote label="1">q</footnote><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(■)</doubt><p>search results collection</p><p>Step 3. Hybrid Feedback Model Construction</p><doubt alpha="100.0" length="5" tooSmall="False" monospace="0.0">Imark</doubt><p>feedback</p><p>Step 2. Hybrid Document Model Construction</p><p>Step 4. Search Results Re-ranking</p><doubt alpha="55.6" length="9" tooSmall="False" monospace="0.0">pqLE&lt;-)^P</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">PL</doubt><doubt alpha="75.0" length="4" tooSmall="True" monospace="0.0">,NEW</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">q</doubt><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">(■ )</doubt><doubt alpha="52.0" length="25" tooSmall="False" monospace="0.0">frp^,re-rank %L[[DqJ -&gt; ■</doubt><doubt alpha="100.0" length="7" tooSmall="False" monospace="0.0">initial</doubt><doubt alpha="100.0" length="7" tooSmall="False" monospace="0.0">results</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">D</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">d</doubt><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">user</doubt><figure caption="Figure 1: Overview of the proposed method."></figure></subsection><subsection number="4.3" title="Hybrid Document Model Construction"><p>For each document in the search results, we con­struct a hybrid language model, which we call the HYB-based language model. In this model, we take into account the latent information in docu­ments as well as the surface information.</p><p>First, an LDA-based document model that con­tains the latent information in the document is con­structed for each document. We perform LDA on D<b><i>q </i></b>to infer the topic distribution in each docu­ment di, and estimate the parameters a, ß and <b><i>ji</i></b><b><i> </i></b>for each di as described in Section 3.2. Then, the LDA-based document model is constructed by computing the probabilities of words given di us­ing Eq.(5). In this model, we can allocate high probability to words that are highly probable from the latent topic distribution of a document.</p><p>Next, for each di, we construct an HYB-based document model Pd[YB(•) by interpolating the DIR-based document model with the LDA-based document model as follows.</p><doubt alpha="51.2" length="43" tooSmall="False" monospace="0.0">PdHYB(wj ) = (1 — a)PdDIR(wj ) + aPLDA(wj )</doubt><p>where a is a parameter that controls the reliability of the LDA-based document model.</p><p>This interpolation is motivated by significant improvements reported in (Wei and Croft, 2006). They also interpolate a DIR-based document model with an LDA-based document model, and perform LDA on the whole collection to construct the LDA-based document model. However, exe­cuting LDA throughout the whole collection re­quires high computational cost. In the proposed method, we can avoid this problem by performing LDA only on the set of search results, which is much smaller than the whole collection.</p></subsection><subsection number="4.4" title="Hybrid Feedback Model Construction"><p>When feedback is given, we also construct an HYB-based language model for it. First, we ob­tain feedback F = <b><i>(fl</i></b><i>,..., </i>f|F|) that is relevant to the user's information need. Note that, in this study, we are not concerned with whether <b><i>F </i></b>is ex­plicit, implicit, or pseudo feedback. Moreover, we have no preference of whether each <b><i>f</i></b><b>i </b>is a whole document or part of a document (e.g., title, snip­pet).</p><p>Next, we perform LDA on <b><i>F </i></b>to infer the topic distribution in <b><i>F, </i></b>and construct the LDA-based feedback model Pj;DA(-). To be more precise, we generate a virtual relevant document <b><i>f </i></b>by com­bining each <b><i>f</i></b><b>i,</b><b> </b>and infer the variational parameter <b><i>jf </i></b>as described in Section 3.3. Then, P<b>|iDA( </b>) is constructed using Eq.(5).</p><p>Finally, we construct the HYB-based feedback model <i>P^YB</i>(•), which contains the surface and latent information in F. <i>P^YB</i>(•) is constructed in the same manner as     <b>YB</b>(•). That is,</p><p>P#YB (w<b>j </b>) = (1 — a)Pj?IR(w<b>j </b>) + aP|DA(w<b>j </b>) where P <b>DIR</b>( ) is constructed using Eq.(2).</p></subsection><subsection number="4.5" title="Search Results Re-ranking"><p>We construct a new query model, which is used to re-rank the initial search results. The new query model <i>P^EW</i>(•) is constructed by interpolating the original query model Pq^LE(•) with the hybrid feedback model <i>P^YB</i>(•) as follows.</p><doubt alpha="50.0" length="40" tooSmall="False" monospace="0.0">PqNEW(wj) = (1 — b)PqMLE(wj) + 6P#YB(wj)</doubt><p>where b is a parameter that controls the reliability of the feedback model. This interpolation is based on Zhai and Lafferty's linear combination method (see Section 2.<page local="5" global="1041"/>3).</p><p>Then, for each document <b><i>d</i></b><b>i </b>in the search re­sults <b><i>Dq, </i></b>we compute the KL-divergence between PdTB ( ) and P^EW (•). That is, the score of doc­ument di for query q and feedback F is defined as <i>re-rankingscore</i><b><i>(di' q, F</i></b>) = —KL(PqNEW (•)||Pd[YB (•)).</p><p>Finally, we obtain the revised search results by re-ranking the documents in <b><i>Dq</i></b><b><i> </i></b>according to their re-ranking scores.</p></subsection></section><section number="5" title="Experiments 5.1 Overview"><p>We conducted three experiments to evaluate the performance of our method. An overview of each experiment is given below.</p><p><b>Experiment 1. Effectiveness with Respect to Explicit and Pseudo RF</b></p><p>We examined how well our method performed in re-ranking the initial search results using explicit and pseudo feedback. In the experiment with ex­plicit RF, we obtained the top 100 documents with the highest initial scores as the initial search re­sults for a given query, and re-ranked them using our method with two relevant documents that were given explicitly. In our experiments, we employed the queries and the relevant documents provided by NTCIR (see Section 5.3). Then, we compared the results with the following three (re-)ranking re­sults.</p><p><b>INIT </b>This is the ranking of the initial search re­sults obtained based on the KL-divergence retrieval model.</p><p><b>WORD </b>This is the ranking of the search results obtained after simple RF, where we used only the surface information (i.e., words) in the feedback and the documents in the search re­sults. This ranking is equivalent to the rank­ing obtained using our method with a = 0.</p><p><b>REPR </b>This is the re-ranking result obtained us­ing Zhai and Lafferty's RF method (Zhai and Lafferty, 2001). The process of the method is almost the same to that of WORD. The main difference is that it modifies the probabilities of words in feedback by a background word distribution (see their paper). We chose their method as it is a representative RF method for language modeling approaches.</p><p>Our method can also be applied to pseudo RF. Hence, we also explored the effectiveness of our method in this regard. With pseudo RF, the top n documents in the initial search results are as­sumed to be relevant, and the search results are re-ranked based on this assumption. We imple­mented pseudo RF using our method for n = 10, and compared the results with the three (re-)ranking results described above.</p><p><b>Experiment 2. Effect of the Amount of</b> <b>Feedback</b></p><p>It is important to know how well our method per­forms when only a small amount of feedback is obtained, because in practice users generally can­not be bothered to provide feedback, and thus sufficient feedback is rarely obtained. We inves­tigated how the amount of explicit feedback af­fected the performance of our method. To be more precise, we reduced the amount of available ex­plicit feedback little by little, and observed the change in precision at 10 top re-ranked documents (P@10). For this experiment, we used seven dif­ferent amounts of explicit feedback: 2<footnote anchor="1"/>, 2<footnote anchor="0"/>, 2-1, 2-2, 2-3, 2-4, and 2-5 relevant documents. Note that, for example, 2-1 documents means that we used half a document's worth of words in the rele­vant documents given as explicit feedback. In this case, half the words were sampled randomly from the feedback, and only these words were used for</p><p>RF.</p><p><b>Experiment 3. Sensitivity to Parameters</b></p><p>It is also important to know how the reliability of the LDA-based document model and the HYB-based feedback model affect the performance of our method. Hence, we investigated how sensitive our method is to parameters a and b. We re-ranked the initial search results using different values for these parameters ranging from 0 to 1 in steps of 0.1, and measured how the performance of our method changed according to these values.</p><subsection number="5.2" title="Configuration of Our Method"><p>The configuration of our method is given below. For the DIR estimation, we set the smoothing pa­rameter ß = 1,000. This setting was also em­ployed in other works (Zhai and Lafferty, 2001; Wei and Croft, 2006). The number of topics K for LDA was set to 20, since with this setting we obtained better results in the preliminary experi­ments, in which we performed LDA with K rang<b>Experiment 1.</b><page local="6" global="1042"/><b> Effectiveness with Respect to Explicit and Pseudo RF</b> ing from 10 to 100 in steps of 10. We set the ini­tial values of a<b>k</b>(k = 1, . . . , K) to 1, and the ini­tial values of P(w<b>j| </b>z<b>k,</b><b> </b><b><i>ß) </i></b>to random values. The number of iterations for the variational parame­ters and that for <b><i>a </i></b>and <b><i>ß </i></b>were set to 10. Addi­tionally, we limited the size of the vocabulary in LDA, designated as J in Section 3, to 1,000. We selected 1, 000 words based on their importance to the search results. Note that the importance of a word w<b>j </b>to the search results <b>Dq </b>is defined as <i>df </i><i>(w</i><b><i>j</i></b><i>,</i><i> </i><b><i>Dq) </i></b>* log<b>(|Daii</b>|/df <i>(w</i><b><i>j</i></b><i>,</i><i> </i><b>Daii)), </b>where <i>df </i><i>(w</i><b><i>j</i></b><i>,</i><i> </i><b><i>D) </i></b>represents the document frequency of <i>w</i><b>j </b>in documents <b><i>D.</i></b></p></subsection><subsection number="5.3" title="Data Set"><p>In our experiments, we employed the test collec­tion used in the Web Retrieval Task in the Third</p><doubt alpha="62.2" length="45" tooSmall="False" monospace="0.0">NTCIR Workshop (Eguchi et al., 2002). The NT-</doubt><p>CIR Workshops are a series of evaluation work­shops designed to enhance research in informa­tion access technologies. The test collection con­sists of 11,038, 720 Japanese Web pages and 47 information needs. For each information need, about 2, 000 documents are rated as highly rele­vant, fairly relevant, partially relevant, or irrele­vant. We used only 40 information needs in our experiments. The remaining 7 (with identification numbers: 0011, 0018, 0032, 0040, 0044, 0047, and 0061) were not used, because we could not retrieve 100 documents for each (see Section 5.1).</p><p>Figure 2 gives an example of an information need for the Web Retrieval Task in the Third NT-CIR Workshop. The meaning of each element is given below.</p><p>(NUM) gives the identification number of the in­formation need.</p><p>( TITLE) provides up to three terms that are sim­ilar to the actual query submitted to a real search engine.</p><p>( DESC) describes the user's information need in a single sentence.</p><p>(RDOC) provides up to three identification num­bers of examples of relevant documents for the information need.</p><p>We employed the terms in the ( TITLE) tag as the query, and the documents in the ( RDOC) tag as explicit feedback. Note that since the numbers of terms and documents differed depending on the in­formation need, we employed the first two terms in the (TITLE) tag and the first two documents in the ( RDOC) tag for each information need.</p></subsection><subsection number="5.4" title="Evaluation Method"><p>We used P@10, mean average precision (MAP), normalized discounted cumulative gain at 10 top (re-)ranked documents (NDCG@10), and NDCG@100 (Jarvelin and Kekalainen, 2002) in the evaluation. In the calculation of P@10 and MAP, documents that were rated as highly rele­vant, fairly relevant and partially relevant were re­garded as relevant, while documents rated as irrel­evant and unrated documents were regarded as ir­relevant. Note that MAP was calculated using all the (re-)ranked documents (i.e., 100 documents). In calculating NDCG, we assessed the relevance score of documents rated highly relevant, fairly relevant and partially relevant as 3, 2, and 1 re­spectively.</p><p>To evaluate the effectiveness of explicit RF, we decided in advance which documents would be used as explicit feedback as described in Section 5.3, and if these were included in the initial search results and the re-ranked results, we removed them from both sets of results. One common problem in the evaluation of the effectiveness of explicit RF is how to handle documents that users have marked as relevant (i.e., the input to RF methods) (Hull, 1993). If the initial search results and the re-ranked results are compared in a straightforward manner, the latter have an advantage. This is due to the fact that documents that are known to be rel­evant tend to be re-ranked higher. However, if we remove them from the re-ranked results, they have a disadvantage. This is especially true if there are few relevant documents. Therefore, we removed the documents used as explicit feedback from both the initial search results and the re-ranked results in the experiments with explicit RF.</p><p>In contrast, we did not apply extra care in Ex­periment 1 with pseudo RF, and measured the per­formance of each method using its raw (re-)ranked results.</p></subsection><subsection number="5.5" title="Experimental Results"><p>Table 1 gives the results for explicit RF. Owing to space limitations, we only show the optimal re­sults in terms of P@10 for each method. The op­timal results for WORD were obtained using our<page local="7" global="1043"/></p><p>RF.</p><doubt alpha="35.3" length="17" tooSmall="False" monospace="0.0">(NUM) 0008 (/NUM)</doubt><p>(TITLE) Salsa, learn, methods (/TITLE)</p><doubt alpha="56.6" length="136" tooSmall="False" monospace="0.0">(DESC) I want to find out about methods for learning how to dance the salsa (/DESC)(rDOC) NW011992774, NW011992731, NW011992734 (/RDOC)_</doubt><figure caption="Figure 2: Example of an information need for a Web Retrieval Task in the Third NTCIR Workshop."></figure><table caption="Table 1: Effectiveness with respect to explicit RF."></table><p>method with <i>a </i>= 0 and <i>b = </i>0.7, while those for OURS (our method) were obtained with <i>a </i>= 0.2 and b = 0.7.</p><p>Based on this table, we can confirm that our method is effective with respect to explicit RF. Our method significantly improved the initial search results across all metrics. Additionally, it outper­formed two other baseline RF methods, with the differences in all metrics being statistically signifi­cant (Wilcoxon signed-rank test, <i>p &lt; </i>0.05). There were no significant differences between the base­line methods, since they were similar in process to each other. These results suggest that the latent in­formation in the user feedback and each document in the search results is useful for explicit RF.</p><p>As a result of the investigation, we found that our method made good use of the words that did not appear in the feedback but were highly prob­able from the latent topic distribution of the feed­back. Consider the information need in Figure 2 as an example. The documents employed as user feedback did not contain the words "technique" or "level", which are related to the information need. As such, the baseline methods could not use these words. In contrast, our method allocated a cer­tain degree of probability to these highly probable words using LDA, despite the words not appearing in the feedback, and hence raised the score of rel­evant documents in the search results containing these words.</p><p>Table 2 gives the results for pseudo RF. The val­ues of the parameters for WORD and OURS were determined as: <i>a </i>= 0, b = 0.7, and <i>a = </i>0.1, b = 0.6, respectively. Note that the results of INIT in Table 2 differ from those in Table 1. This is be­cause although we removed the documents used as user feedback from the initial search results (and the re-ranked results) in the experiment with ex­plicit RF, we did not remove them from any of the results in this experiment.</p><table caption="Table 2: Effectiveness with respect to pseudo RF."></table><p>From this table, we can see that our method sig­nificantly improved the initial search results. Ad­ditionally, our method outperformed the baseline methods. From these results, we can conclude that our method is also effective with respect to pseudo <b>Experiment 2.</b><b> Effect of the Amount of Feedback</b></p><p>Figure 3 shows the effect of the amount of ex­plicit feedback on the performance of our method. For comparison with baseline methods, we also present their results. The parameters for each method were identical to those used in Experiment 1 with explicit RF.</p><p>From this figure, we can see that our method achieved consistently high performance. For ex­ample, when 2<footnote anchor="0"/> relevant documents (i.e., one rel­evant document) were given as user feedback, our method improved the initial search results by about 35% in P@10. Additionally, a notable fea­ture is that although the improvements in the base­line methods almost disappeared, our method per­formed well when only a small amount of feed­back was obtained. For example, improvement of about 18% was achieved even with only 2-5 doc­uments, which constituted an average of 52 words in our experiment. The reason for this is, once again, that our method is able to use not only the surface words in the feedback, but also the highly probable words from its latent topic distribution.</p><p>As described above, our method can re-rank search results using a small amount of feedback. This suggests that our method is practically use­ful, and that it performs well even if only a part of a document (e.g., title, snippet), the relevance of which is easier to determine than that of the whole document, are given as user feedback.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>P@10</p></td><td class="cell"><p>MAP</p></td><td class="cell"><p>NDCG@10</p></td><td class="cell"><p>NDCG@100</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>INIT</p></td><td class="cell"><p>0.278</p></td><td class="cell"><p>0.106</p></td><td class="cell"><p>0.220</p></td><td class="cell"><p>0.249</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>WORD</p></td><td class="cell"><p>0.310</p></td><td class="cell"><p>0.111</p></td><td class="cell"><p>0.228</p></td><td class="cell"><p>0.250</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>REPR</p></td><td class="cell"><p>0.303</p></td><td class="cell"><p>0.107</p></td><td class="cell"><p>0.236</p></td><td class="cell"><p>0.249</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>OURS</p></td><td class="cell"><p>0.383</p></td><td class="cell"><p>0.117</p></td><td class="cell"><p>0.284</p></td><td class="cell"><p>0.255</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>P@10</p></td><td class="cell"><p>MAP</p></td><td class="cell"><p>NDCG@10</p></td><td class="cell"><p>NDCG@100</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>INIT</p></td><td class="cell"><p>0.298</p></td><td class="cell"><p>0.112</p></td><td class="cell"><p>0.243</p></td><td class="cell"><p>0.268</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>WORD</p></td><td class="cell"><p>0.303</p></td><td class="cell"><p>0.111</p></td><td class="cell"><p>0.258</p></td><td class="cell"><p>0.274</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>REPR</p></td><td class="cell"><p>0.300</p></td><td class="cell"><p>0.112</p></td><td class="cell"><p>0.250</p></td><td class="cell"><p>0.270</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>OURS</p></td><td class="cell"><p>0.330</p></td><td class="cell"><p>0.112</p></td><td class="cell"><p>0.283</p></td><td class="cell"><p>0.278</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="8" global="1044"/><p><b>Experiment 3. Sensitivity to Parameters</b></p><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">INIT</doubt><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">REPR</doubt><doubt alpha="80.0" length="5" tooSmall="False" monospace="0.0">-WORD</doubt><doubt alpha="80.0" length="5" tooSmall="False" monospace="0.0">-OURS</doubt><doubt alpha="13.3" length="30" tooSmall="False" monospace="0.0">■ 2A(-5)   -♦-2A(-3)-A--2A(-1)</doubt><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">-2*1</doubt><doubt alpha="0.0" length="44" tooSmall="True" monospace="0.0">0.40 0.38 0.36 0.34 0.32 0.30 0.28 0.26 0.24</doubt><doubt alpha="50.0" length="4" tooSmall="True" monospace="0.0">—T n</doubt><doubt alpha="0.0" length="19" tooSmall="True" monospace="0.0">2-52-42-32-22-12021</doubt><p>amount of explicit feedback (document) Figure 3: Effect of the amount of feedback.</p><doubt alpha="0.0" length="7" tooSmall="False" monospace="0.0">-2*(-5)</doubt><doubt alpha="0.0" length="7" tooSmall="False" monospace="0.0">-2*(-3)</doubt><doubt alpha="0.0" length="7" tooSmall="False" monospace="0.0">•2*(-1)</doubt><doubt alpha="0.0" length="79" tooSmall="True" monospace="0.0">0.0     0.1     0.2     0.3     0.4     0.5     0.6     0.7     0.8     0.9 1.0</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">a</doubt><figure caption="Figure 4: Sensitivity to parameter a."></figure><p>The sensitivity of the performance to a is illus­trated in Figure 4. Each line in the figure rep­resents the results with different amounts of ex­plicit feedback: 2-5, 2-3, 2-1, and 2<footnote anchor="1"/> relevant documents. The value of the other parameter b was fixed to 0.7. From this figure, we can see that the performance of our method is sensitive to the value of a, and that the optimal value is about 0.2. Despite the goal and setting being different to ours, this optimal value is similar to that re­ported in (Wei and Croft, 2006), where the DIR-and LDA-based document models were interpo­lated as in our work.</p><p>The sensitivity to b is depicted in Figure 5. The value of <i>a </i>was fixed to 0.2. According to this fig­ure, we can see that the setting of b also affected the performance of our method. Additionally, this figure shows that if we set the value appropriately, the interpolated new query model is more effective than both the original query model on its own (i.e., b = 0.0) and the feedback model on its own (i.e., b = 1.0). These findings concur approximately with the results presented in (Zhai and Lafferty, 2001).</p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">b</doubt><figure caption="Figure 5: Sensitivity to parameter b.5.6 Discussion"></figure><p>Although our method achieved good performance in our experiments, we also encountered a prob­lem in that the method took a long time to execute. More specifically, our method required about one minute to estimate the parameters of LDA in Step 2. (On the other hand, the time required for Steps 1, 3, and 4 was only a few seconds.) Thus, we need to explore ways of reducing the time for parame­ter estimation so that our method can be used in real situations. One way of doing this is to choose a faster estimation technique. For example, col­lapsed variational methods may provide a viable solution (Teh et al., 2006; Asuncion et al., 2009). Another alternative is to decrease the size of the vocabulary, designated as J in Section 3. For ex­ample, we conducted an additional experiment, in which we set J = 100, and confirmed that the time for parameter estimation fell to about 10 seconds without any significant change in performance.</p></subsection></section><section number="6" title="Conclusion"><p>In this paper, we proposed a novel RF method us­ing latent information, and discussed the effective­ness thereof. Using LDA, our method infers the distributions over latent topics in the feedback and in each document in the search results. Then, doc­uments whose topic distribution resembles that of the feedback are regarded as being relevant to the user's information need, and are re-ranked higher. Through our experiments, we confirmed that our method achieves good performance for both ex­plicit and pseudo RF, and that it provides the ben­efit of performing well even when only a small amount of feedback can be obtained. As future work, we aim to explore ways of reducing the ex­ecution time of our method so that it can be used in practical situations.</p><page local="9" global="1045"/></section><references><p>Arthur Asuncion, Max Welling, Padhraic Smyth, and Yee Whye Teh. 2009. On smoothing and inference for topic models. In <i>Proceedings of UAI2009.</i></p><p>David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. <i>Jounal of Machine Learning Research, </i>3:993-1022.</p><p>Koji Eguchi, Keizo Oyama, Emi Ishida, Kazuko Kuriyama, and Noriko Kando. 2002. The web re­trieval task and its evaluation in the third ntcir work­shop. In <i>Proceedings ofSIGIR 2002, </i>pages 375­376.</p><p>Thomas L. Griffiths and Mark Steyvers. 2004. Finding scientific topics. In <i>Proceedings ofNAS 2004, </i>pages 5228-5235.</p><p>Thomas Hofmann. 1999. Probabilistic latent semantic indexing. In <i>Proceedings ofSIGIR 1999, </i>pages 50­57.</p><p>David Hull. 1993. Using statistical testing in the eval­uation of retrieval experiments. In <i>Proceedings of SIGIR 1993, </i>pages 329-338.</p><p>Eleanor Ide. 1971. New experiments in relevance feedback. In <i>The SMART Retrieval System: Ex­periments in Automatic Document Processing, </i>pages 337-354. Prentice-Hall Inc.</p><p>Bernard James Jansen, Amanda Spink, and Tefko Saracevic. 2000. Real life, real users, and real needs: a study and analysis of user queries on the web.   <i>Information Processing and Management,</i> 36(2):207-227.</p><p>Kalervo Jarvelin and Jaana Kekalainen. 2002. Cumu­lated gain-based evaluation of ir techniques. <i>ACM Transactions on Information Systems, </i>20(4):422446.</p><p>John Lafferty and Chengxiang Zhai. 2001. Document language models, query models, and risk minimiza­tion for information retrieval. In <i>Proceedings ofSI-GIR 2001, </i>pages 111-119.</p><p>Victor Lavrenko and W. Bruce Croft. 2001. Relevance-based language models. In <i>Proceedings</i> <i>ofSIGIR 2001, </i>pages 120-127.</p><p>Thomas P. Minka. 2000. Estimating a dirichlet distri­bution. Technical report, Microsoft.</p><p>Jay M. Ponte and W. Bruce Croft. 1998. A language modeling approach to information retrieval. In <i>Pro­ceedings ofSIGIR 1998, </i>pages 275-281.</p><p>Joseph John Rocchio. 1971. Relevance feedback in information retrieval. In <i>The SMART Retrieval Sys­tem: Experiments in Automatic Document Process­</i>ing, pages 313-323. Prentice-Hall Inc.</p><p>Gerard Salton, Anita Wong, and Chung-Shu Yang. 1975. A vector space model for automatic indexing. <i>Communications of the ACM, </i>18(11):613-620.</p><p>Karen Sparck Jones, S. Walker, and Stephen E. Robert­son. 2000. A probabilistic model of informa­tion retrieval: Development and comparative exper­iments. <i>Information Processing and Management,</i> 36(6):779-808,809-840.</p><p>Yee Whye Teh, David Newman, and Max Welling. 2006. A collapsed variational bayesian inference al­gorithm for latent dirichlet allocation. In <i>Proceed­ings of NIPS 2006.</i></p><p>Xing Wei and W.Bruce Croft. 2006. Lda-based docu­ment models for ad-hoc retrieval. In <i>Proceedings of SIGIR 2006, </i>pages 178-185.</p><p>Chengxiang Zhai and John Lafferty. 2001. Model­based feedback in the language modeling approach to information retrieval. In <i>Proceedings </i><i>ofCIKM</i> <i>2001, </i>pages 403-410.</p><p>Chengxiang Zhai and John Lafferty. 2004. A study of smoothing methods for language models applied to information retrieval. <i>ACM Transactions on Infor­mation Systems, </i>22(2):179-214.</p><p>Dong Zhou and Vincent Wade. 2009. Latent document re-ranking. In <i>Proceedings ofEMNLP 2009, </i>pages 1571-1580.</p></references></body></article>