<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="100"/><title>Exploring Abbreviation Expansion for Genomic Information Retrieval</title><pubinfo>Proceedings of the Australasian Language Technology Workshop 2007, pages 100-108</pubinfo><author surname="Stokes" givenname="Nicola"><org  name="University of Melbourne" country="Australia" city="Melbourne"/></author><author surname="Li" givenname="Yi"><org  name="University of Melbourne" country="Australia" city="Melbourne"/></author><author surname="Cavedon" givenname="Lawrence"><org  name="University of Melbourne" country="Australia" city="Melbourne"/></author><author surname="Zobel" givenname="Justin"><org  name="University of Melbourne" country="Australia" city="Melbourne"/></author></firstpageheader><frontmatter><p><b>Exploring Abbreviation Expansion for Genomic Information Retrieval*</b></p><p><b>Nicola Stokes, Yi Li, Lawrence Cavedon and Justin Zobel</b></p><p>National ICT Australia, Victoria Research Laboratory Department of Computer Science and Software Engineering The University of Melbourne, Victoria 3010, Australia. (nstokes, yli8,  lcavedon, jz}@csse.unimelb.edu.au</p></frontmatter><abstract>Abbreviations are commonly found in­stances of synonymy in Biomedical journal papers. Information retrieval systems that index paragraphs rather than full-text arti­cles are more susceptible to term variation of this kind, since abbreviations are typi­cally only defined once at the beginning of the text. One solution to this problem is to expand the user query automatically with all possible abbreviation instances for each query term. In this paper, we compare the effectiveness of two abbreviation expansion techniques on the TREC 2006 Genomics Track queries and collection. Our results show that for highly ambiguous abbrevia­tions the <i>query collocation </i>effect isn't strong enough to deter the retrieval of erroneous passages. We conclude that full-text ab­breviation resolution prior to passage index­ing is the most appropriate approach to this problem. </abstract></header><body><section number="1" title="Introduction"><p>Query expansion is a well-known technique used in Information Retrieval (IR) to address the problem of lexical variation between the query and semantically related terms in relevant documents (Efthimiadis, 1996). While on average query expansion methods,</p><p>National ICT Australia is funded by the Australian Govern­ment's Department of Communications, Information Technol­ogy, and the Arts and the Australian Research Council through Backing Australia's Ability and the ICT Research Centre ofEx-cellence programs.</p><p>such as <i>relevance feedback </i>(Ruthven and Lalmas, 2003), have been shown to improve retrieval perfor­mance, there are many examples where query effec­tiveness has been significantly downgraded. How­ever, in terminology rich domains where word sense distributions are heavily skewed, query expansion has been shown to have more of a consistent pos­itive effect on retrieval performance. This trend is particularly evident in the passage retrieval task in­vestigated at the TREC (Text REtrieval Conference) Genomics Track (Hersh et al., 2006).</p><p>In this paper, we investigate the impact of vari­ous expansion term types on passage retrieval ef­fectiveness in the biomedical domain. Our re­sults show that expanding with ontologically related words (synonyms, hypernyms, hyponyms) signif­icantly improves performance; however, abbrevia­tion expansion shows more inconsistent results sim­ilar to those seen in general domain expansion ex­periments. One would expect that the performance of IR systems that index paragraphs rather than full-text articles would greatly benefit from this sort of expansion, since abbreviations are typically only de­fined once in an entire document.</p><p>We report the results of our investigation on the TREC 2006 Genomic retrieval task. We compare two abbreviation expansion techniques: the first adds abbreviations found in the ADAM database of abbreviations (Zhou et al., 2006a); the second, uses a pseudo relevance feedback strategy to iden­tify query term abbreviations in the full-text docu­ments of an initial set of retrieved passages. Despite the benefit of mutual disambiguation across query terms, referred to as the <i>query term collocation effect</i> (Krovetz and Croft, 1992), both approaches reduce retrieval effectiveness, leading to the conclusion that abbreviation resolution in the document collection is more appropriate than expansion.<page local="2" global="101"/></p><p>Another contribution of this paper is our novel concept-based IR ranking method. This ranking method is an adaptation of the Okapi method, en­hanced so as to deal with multi-concept queries de­rived from natural language questions. Our method ensures that passages containing at least one occur­rence of all the query concepts out-rank passages that contain many occurrences of only one of the concepts. We also describe a paragraph reduction strategy that increases the TREC defined answer ex­traction accuracy score of our system. Finally, we discuss our plans for future work.</p></section><section number="2" title="Information Retrieval for Functional Genomics"><p>Biomedical text retrieval is a very active area of re­search, driven by the biomedical community's need for high precision systems that answer specific bi­ological questions not captured in the plethora of database resources (of varying quality) containing different types of biological information. Two dis­tinct user information needs have been recently in­vestigated by the IR community: <i>clinical text re­trieval </i>(which supports patient-centred clinical re­search or care) and <i>functional genomic text retrieval </i>(which supports researchers involved in laboratory experiments). In this paper, we focus on genomic retrieval. An interesting overview of evidence-based medical retrieval in the clinical domain can be found in (Lin and Demner-Fushman, 2006).</p><p>Functional Genomics is the study of gene and pro­tein function and interaction at a molecular level, and the effects of this interaction on biological pro­cesses that results in phenotypic outcomes (such as disease) in organisms. An important yet very time-consuming part of the functional genomics pipeline for researchers involves arriving at bio­logically motivated explanations for the output of bioinformatics-based clustering techniques such as gene expression profiling. Since a single experiment can involve thousands of genes, even a competent biologist needs to turn to a search engine to deter­mining whether the functional dependencies found in these clusters make sense.</p><p>The TREC Genomics Track was established in 2003 with the aim of supporting the evaluation of information retrieval systems capable of answering the types of questions typically posed by genomi-cists such as:</p><p>• What is the role of gene A in disease B?</p><p>• What effect does gene A have on a particular biological process?</p><p>• How do genes A and B interact in the function of a specific organ?</p><p>• How do mutations in gene A influence a partic­ular biological process?</p><p>Each of these four query templates were investi­gated at the 2006 Genomics Track. In all, 28 queries were evaluated on a collection of full-text journal pa­pers, where the task was to retrieved relevant answer passages rather than full-text documents. In the fol­lowing section we describe our novel genomic re­trieval system.</p></section><section number="3" title="System Description"><p>In this section, we describe the different components in our Genomic IR architecture. Our IR system is a version of the Zettair engine<footnote anchor="1"/> that we have specifi­cally modified for passage retrieval and biomedical query term expansion.</p><p><b>Collection Preprocessing</b></p><p>The TREC collection consists of full-text journal ar­ticles obtained by crawling the Highwire site<footnote anchor="2"/>. The full collection contains 162,259 documents and is about 12.3 GB in size when uncompressed. Af­ter preprocessing, the whole collection becomes 7.9 GB. The collection is pre-processed as follows:</p><p>Paragraph Segmentation: for evaluation purposes the Genomics Track requests that the ranked answer passages must be within specified para­graph boundaries.</p><footnote label="1">http://www.seg.rmit.edu.au/zettair/ 2 http://www.highwire.org</footnote><page local="3" global="102"/><p>Sentence Segmentation: all sentences within para­graphs are segmented using an open source</p><doubt alpha="66.7" length="6" tooSmall="False" monospace="0.0">tool.3</doubt><p>Character Replacement: Greek characters repre­sented by gifs are replaced by textual encod­ings; accented characters such as "A" or "A" are replaced by "A"; Roman numbers are re­placed by Arabic numerals. These replace­ments are very important for capturing varia­tions in gene names.</p><p>Removal: all HTML tags, very short sentences, paragraphs with the heading <i>Abbreviations, </i>fig­ures, tables and some special characters such as hyphens, slashes and asterisks are removed: (Trieschnigg et al., 2006) has shown that small changes in the tokenisation strategy such as these improve the performance of biomedical</p><p>IR.</p><p><b>Query Expansion</b></p><p>Once the collection has been indexed, querying can begin. In the 2006 Genomics Track, each query or topic contains at least two biological concepts or en­tities which could be a gene ("NM23"), a protein ("p53"), a disease ("ovarian cancer") or a biologi­cal process ("ethanol metabolism"). TREC simpli­fies the query preprocessing task by ensuring that all topics conform to the query templates discussed in Section 2. The following is a sample query, Topic 173 from the 2006 track, which contains two con­cepts: "PrnP" (a gene) and "mad cow disease" (a disease):</p><p><i>What is the role of </i><b>PrnP </b><i>in </i><b>mad cow disease?</b></p><p>Our query expansion process proceeds as follows. First, each gene or protein in the query is expanded with entries from the Entrez Gene database.<footnote anchor="4"/> Since the same gene may occur in many different species, and many of their synonyms only differ with re­spect to capitalisation, we choose the first entry re­trieved that belongs to the species type <i>Homo sapien. </i>Then, terms in the <i>Official Symbol, Name, Other</i> <i>Aliases </i>and <i>Other Designations </i>fields, for the gene, are added to the query.</p><footnote label="3">http://l2r.cs.uiuc.edu/~cogcomp/atool. php?tkey=SS</footnote><footnote label="4">http://www.ncbi.nlm.nih.gov/sites/ entrez?db=gene</footnote><p>For all disease and biological process mentions in the query, we use the MeSH<footnote anchor="5"/> taxonomy of medical terms to find their synonyms (using the <i>Entry Terms </i>and <i>See Also </i>fields). The terms' hyponyms (descen­dants) and hypernyms (ancestors) in the MeSH tree structure are also used as expansion terms.</p><p><b>Gene Variant Generation</b></p><p>As well as expanding with synonyms, we use a "gene variant" generation tool to generate all the possible variants for both original query terms and expanded terms. Our segmentation rules are similar to those used by (Buttcher et al., 2004). We describe our rules as follows:</p><p>Given a gene name containing a hyphen or punc­tuation, or a change from lower case to upper case, or from a character to a number (or vice versa), or a Greek character (e.g. "alpha"), we call this a <i>split point. </i>A word is split according to all its split points, and all variants are generated by concatenating all these split parts, optionally with a space inserted. Greek characters are also mapped to English vari­ants, e.g. "alpha" is mapped to "a".</p><p>For example, on the query term "Sec61alpha", we would generate the following lexical variants which are also commonly used forms of this term in the collection: "Sec 61alpha", "Sec61 alpha", "Sec 61 alpha", "Sec 61a", "Sec61 a", "Sec 61 a", "Sec61a";</p><p>In phrases, we replace hyphens ("-"), slashes ("/") and asterisks ("*") in the queries with spaces. For example, "subunit 1 BRCA1 BRCA2 containing complex" is a variant of "subunit 1 BRCA1/BRCA2-containing complex".</p><p><b>Concept-based Query Normalisation</b></p><p>Our document ranking method is based on the Okapi model (Robertson et al., 1994). Many participant systems at the TREC Genomics track use the Okapi method for ranking documents with respect to their similarity to the query. However, there are two fun­damental problems with using this model on TREC Genomic queries.</p><p>The first problem regards Okapi not differentiat­ing between concept terms and general query terms in the query.<page local="4" global="103"/> For example, consider two documents, one containing the terms "mad cow disease" and "PrnP", and the other containing the terms "role" and "PrnP". Clearly the first document containing the two biological concepts is more relevant. The second problem occurs because TREC 2006 topics contain more than one concept term. It is possi­ble that a short paragraph that discusses one concept only will be ranked higher than a longer paragraph which mentions two concepts. Again this is an un­desirable outcome.</p><footnote label="5">http://www.nlm.nih.gov/mesh</footnote><p>To overcome these problems, a <i>Conceptual IR </i>model was proposed in (Zhou et al., 2006b). In this paper we propose another method called the <i>concept-based query normalisation </i>which is based on the Okapi model and similar to the method intro­duced in (Li, 2007; Stokes et al., 2008) for geospatial IR.</p><p>The first problem is solved by dividing query terms into two types: <i>general terms tg </i>and <i>concept terms tc. </i>Given a query with both concept and gen­eral terms, the similarity between a query Q and a document Dd is measured as follows:</p><doubt alpha="64.5" length="31" tooSmall="False" monospace="0.0">sim(Q,Dd)=gsim(Q,Dd)+csim(Q,Dd)</doubt><p>where <i>gsim(Q, D</i><i>d) </i>is the <i>general similarity score </i>and <i>csim(Q, D</i><i>d)</i><i> </i>is the <i>concept similarity score. </i>The general similarity score is given by:</p><doubt alpha="50.0" length="44" tooSmall="False" monospace="0.0">gsim(Q, Dd)=simt(Q, Dd)=       rd,t• wt•rq,t</doubt><doubt alpha="75.0" length="4" tooSmall="False" monospace="0.0">t&amp;Qg</doubt><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">tEQg</doubt><p>where <i>Qg </i>is the aggregation of all general terms/phrases in the query. The concept similarity score is given by:</p><doubt alpha="63.6" length="22" tooSmall="False" monospace="0.0">csim(Q,Dd)= simc(Q,Dd)</doubt><doubt alpha="54.3" length="35" tooSmall="False" monospace="0.0">= Norm(simt1(Q,Dd),...,simtN(Q,Dd))</doubt><p><i>t</i><i>e</i><i>c,c</i><i>e</i><i>Qc</i></p><p><i>tEC,CEQc</i></p><doubt alpha="50.0" length="18" tooSmall="False" monospace="0.0">simt2+     +simtN)</doubt><doubt alpha="20.0" length="15" tooSmall="False" monospace="0.0">-+       +aN-l)</doubt><p>where <i>Q</i><i>c </i>is the aggregation of all concepts in the query, <i>C </i>is one concept in Qc, and <i>U </i>is a term/phrase in the query, after expansion, which belongs to the concept C; the tj are listed in descending order ac­cording to their Okapi similarity scores <i>simtl</i>, ..., <i>sim</i><i>tN </i>:</p><doubt alpha="44.1" length="34" tooSmall="False" monospace="0.0">simt(Q, Dd)   =   rd,t • wt • rq&gt;t</doubt><doubt alpha="100.0" length="5" tooSmall="False" monospace="0.0">where</doubt><doubt alpha="75.0" length="4" tooSmall="False" monospace="0.0">rd,t</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">wt</doubt><doubt alpha="75.0" length="4" tooSmall="False" monospace="0.0">rg,t</doubt><doubt alpha="38.5" length="13" tooSmall="False" monospace="0.0">(kl+ 1) •fd,t</doubt><p><i>avgWd </i><i>] + f</i><i>d,t</i></p><doubt alpha="42.9" length="21" tooSmall="False" monospace="0.0">N - max(ft,ftq) + 0.5</doubt><doubt alpha="40.7" length="27" tooSmall="False" monospace="0.0">max(ft,ft,)+0.5(ka + 1) •Ut</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(1)</doubt><doubt alpha="60.0" length="5" tooSmall="False" monospace="0.0">ka +f</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">q,t</doubt><p>where <i>k\ </i>and <i>b </i>are usually set to 1.2 and 0.75 re­spectively, and <i>k3 </i>can be taken to be oo. Variable <i>Wd </i>is the length of the document <i>d </i>in bytes; <i>avgWd</i>is the average document length in the entire collec­tion; <i>N </i>is the total number of documents in the col­lection; <i>ft </i>is the number of documents in which term t occurs; and <i>f</i><i>{d,q],t </i>is the frequency of term t in ei­ther a document d or query <i>q.</i></p><p>Note that (1) is an adjustment of the calculation for the weight wt of an <i>expansion </i>term t appear­ing in the query: for expansion term <i>t,</i><i> </i>its own term frequency <i>f</i><i>t </i>and the corresponding original query term's frequency <i>f</i><i>tq </i>are compared, and the larger value used — this ensures the term contributes an appropriately normalised "concept weight".</p><p>To solve the second problem, we use the follow­ing rules to ensure that for two passages <i>P\</i><i> </i>and <i>P2, </i>where one contains more unique concepts than the other, the number of concepts <i>ConceptNum(P) </i>will override the Okapi score <i>Score(P) </i>and assign a higher rank to the passage with more unique con­cepts:</p><p><b>if </b><i>ConceptNum(P1) &gt; ConceptNum(P2) </i><b>then</b></p><p><i>Rank(P1) &gt; Rank(P2) </i><b>else if </b><i>ConceptNum(P1) &lt; ConceptNum(P2) </i><b>then</b><page local="5" global="104"/></p><doubt alpha="64.7" length="51" tooSmall="False" monospace="0.0">Rank(P2) &gt; Rank(Pi)else ifScore(P1) &gt; Score(P2)then</doubt><doubt alpha="61.1" length="18" tooSmall="False" monospace="0.0">Rank(Pi)&gt; Rank(P2)</doubt><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">else</doubt><doubt alpha="61.1" length="18" tooSmall="False" monospace="0.0">Rank(P2)&gt; Rank(Pi)</doubt><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">ki •</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">l</doubt><doubt alpha="29.4" length="17" tooSmall="False" monospace="0.0">=       53(simtt+</doubt><p><b>Abbreviation Finder</b></p><p>Although MeSH and Entrez Gene contain many syn­onyms and related terms, one important type of lex­ical variant, <i>abbreviations, </i>has very low coverage in both databases. For example, "AD" is a com­monly used abbreviation for "Alzheimer's Disease". Since the long and short form ("Alzheimer's Disease (AD)") only appear together at the beginning ofeach journal document, many relevant passages will con­tain "AD" only and so will appear less relevant than they should against a query containing "Alzheimer's Disease". Hence, expanding the given query with "AD" should improve retrieval effectiveness.</p><p>As already mentioned, there are two methods for collecting abbreviations from the literature: the first uses the static resource ADAM (Zhou et al., 2006a), while the second uses our pseudo relevance feed­back method for extraction these abbreviations dur­ing run time. The advantage of the latter approach is that it dynamically collects abbreviations and so does not suffer from the coverage and update prob­lems of static resources like ADAM. The follow­ing is an overview of how our abbreviation feedback step contributes to the retrieval process:</p><p>1. Retrieve the first 1000 documents which in­clude at least one instance of each concept in the query.</p><p>2. From this subset of documents, find terms which fit the pattern <i>"Term </i>(Abbr)", where <i>"Term" </i>is a concept in the query (original or ex­panded) and <i>"Abbr" </i>is the abbreviation or syn­onym defined in the text.</p><p>3. Among all the detected abbreviations or syn­onyms, remove all the multi-word terms, terms that do not have any overlapping characters with the original term, and terms which occur less than three times.</p><p>4. For all remaining abbreviations or synonyms, use the above generation tool to formulate all their lexical variants, and add them to the query. The expanded query is then re-submitted to the retrieval engine, and the passage extraction step, described below, is applied.</p><p><b>Passage Extraction</b></p><p>As already mentioned the 2006 Genomics Track de­fined a new question answering-type task that re­quires short full-sentence answers to be retrieved in response to a particular query. However, before answer passages can be generated, we first retrieve the first 1000 ranked paragraphs for each topic, and use the following simple rules to reduce these para­graphs to answer spans.</p><p>Two methods are examined in this paper which are best described with an example. Given a para­graph consisting of a set of sentences <i>{(s\,</i><i> </i>i), (s2, i), (S3, r), (s4, r), (S5, i), (sa, r), (S7, i), (sg, i), (sg, r), (sio, i)}, where r is relevant (that is, mentions at least one query term) and i is irrelevant. <i>Method A </i>shortens a paragraph by removing irrelevant sen­tences from its start and end until a relevant sentence is detected. Hence, it would produce the following passage of sentences: {(s3, r), (s4, r), (s5, i), (sa, r), (s7, i), (sg, i), (sg, r)}.</p><p>This extraction method does not split a paragraph into multiple passages if irrelevant sentences occur within the resultant passage. <i>Method B, </i>on the other hand, addresses this issue by splitting a passage if there are two or more consecutive irrelevant sen­tences within this span. Hence, Method B would produce the following two passages for this para­graph: {(s3, r), (s4, r), (s5, i), (sa, r)} and {(sg,</p><p>r)}.</p><p>After one of these passage extraction techniques has been applied for a particular topic, we re-rank passages by re-indexing them, and re-querying the topic against this new index, using the global statis­tics from the original indexed collection, i.e. using term frequency f<i>t </i>and the average paragraph length <i>avgWd.</i></p></section><section number="4" title="Experimental Methodology"><subsection number="4.1" title="Data and Evaluation Metrics"><p>We used the TREC 2006 Genomics Track evaluation resources to determine the effectiveness of our sys­tem. The TREC 2006 collection consists of 162,259 full-text documents from 49 journals publish elec­tronically via the Highwire Press website<footnote anchor="6"/>. The track also provided 28 topics expressed as natural language questions, formatted with respect to seven general topic templates.<page local="6" global="105"/> Participants were asked to submit the first 1,000 ranked passages returned by their system for each of the topics (Hersh et al., 2006). Passages in this task are defined as text se­quences that cannot cross paragraph boundaries (de­limited by HTML tags), and are subsets of the origi­nal paragraphs in which they occur. As is the custom at TREC, human judges were used to decide the rel­evance of passages in the pooled participating sys­tem results. These judges also defined exact passage boundaries, and assigned topic tags called <i>aspects </i>from a control vocabulary of MeSH terms to each relevant answer retrieved.</p><footnote label="6">More information on the TREC dataset can be found at: http://ir.ohsu.edu/genomics/2 00 6data.html</footnote><p>Mean Average Precision, or MAP, is a popular IR metric for evaluating system effectiveness. The TREC Genomics Track defines three versions of the MAP score calculated at various levels of granular­ity: <i>Document, Passage </i>and <i>Aspect. </i>Traditionally the MAP score is defined as follows: first, the av­erage of all the precision values at each recall point on a topic's <i>document </i>ranked list is calculated; then, the mean of all the topic average precisions is de­termined. Since the retrieval task at the Genomics Track is a question answering-style task, a metric that is sensitive to the length of the answer retrieved was developed.</p><p>Passage MAP is similar to document MAP ex­cept average precision is calculated as the fraction of characters in the system passage overlapping with the gold standard answer, divided by the total num­ber of characters in every passage retrieved up to that point in the ranked list. Hence, a system is penalised for all additional characters retrieved that are not members of the human evaluated answer passage.</p><p>The TREC organisers also wanted to measure to what extent a particular passage captured all the nec­essary information required in the answer. Judges were asked to assign at least one MeSH heading to all relevant passages. Aspect average precision is then measured as the number of aspects (MeSH headings) captured by all the relevant documents up to the recall point in the ranked list for a particular query. Relevant passages that did not contribute any new aspect to the aspects retrieved by higher ranked passages were removed from the ranking. Aspect MAP is defined as the mean of these average topic precision scores.</p></subsection><subsection number="4.2" title="Experimental Results"><p>In this section, we examine the increased effective­ness obtained when different expansion information is added to the original query. We also evaluate the effect of our proposed abbreviation feedback tech­nique, and our novel answer expansion module, on system performance.</p><p>As explained in Section 3, our system uses Entrez Gene for expansion of genes to their synonymous instances. In addition, all term variants are gener­ated for their abbreviations as described in Section 3, while other biological entities in the query (e.g., dis­eases) are expanded using MeSH. Table 1 presents the MAP scores for the following system runs:</p><p><b>• Baseline: </b>Zettair system with no expansion</p><p><b>• SYN: </b>query expansion using Entrez gene and MeSH expansion <i>(Synonym </i>and <i>See Also </i>en­tries in MeSH) of query terms</p><p><b>• SYN+HYPO: </b>query expansion using Entrez gene and MeSH expansion, including <i>Hy-ponyms </i>(i.e., specialisations)</p><p><b>• SYN+HYPER: </b>query expansion using Entrez gene and MeSH expansion, including <i>Hyper-nyms </i>(i.e., generalisations)</p><p><b>• SYN+HYPER+VAR: </b>query expansion using En­trez gene, <i>Gene Variant Generation, </i>and MeSH expansion, including <i>Hypernyms</i></p><p>All expansion run MAP scores show a statistically significant<footnote anchor="7"/> improvement over the baseline MAP. The only expansion experiment that does not incre­mentally improve the results is the addition of hy-ponym terms (i.e. specialisation) from MeSH. On the other hand, hypernyms (i.e. generalisations) im­prove the performance of the <b>SYN </b>run by nearly 5%. This result may be explained by the fact that at a pas­sage level, generalised expressions are commonly used to refer to query terms that have been discuss earlier in the document. For example, the following sentence is clearly relevant to the <i>mad cow disease </i>query presented in Section 3: "These <i>prion diseases </i>are characterised by the accumulation of an abnor­mal (aberrantly folded) isoform of a cellular host<page local="7" global="106"/></p><footnote label="7">We use a paired Wilcoxon signed-rank test at the 0.05 con­fidence level to determine significance.</footnote><p>Table 1: Table showing improvement in MAP score obtained over baseline MAP when the query is expanded with various combinations of related terms: synonyms (SYN), hyponyms (HYPO), hypernyms (HYPER) and gene lexical variants (VAR) protein PrPC". However, it would only be ranked highly if the generalisation relationship from <i>mad cow disease </i>to <i>prion disease </i>has been established. Expanding the query term with the immediate par­ent terms in the different MeSH hierarchies usually results in a few focussed terms being added to the query. In contrast, adding specialisations may result in a much larger number of term additions, depend­ing on the generality of the query term. For exam­ple, the term <i>neurons </i>has 18 unique subcategories one level below its position in the MeSH hierarchy and many more beyond this level.</p><p>Our best system run <b>(SYN+HYPER+VAR) </b>used ontological and gene variant expansion, and achieved a 97.6% increase in Passage MAP over the baseline run. Similarly large increases in Aspect and Document MAP were also observed. A detailed analysis showed that many passages had been either missed or ranked lower than expected by our system due to the occurrence of query term abbreviations in the relevant passage. These abbreviations were not captured in either of our ontological resources.</p><p>Table 2 compares the performance of the two abbreviation expansion strategies described in Sec­tion 3. Ontological expansion using the ADAM ab­breviation database reduces our best Passage MAP score by 36%. Our abbreviation feedback loop per­forms better, producing a small increase in Docu­ment MAP over the baseline, but slightly lower As­pect and Passage MAPs. In some respects, this feed­back result is disappointing as a manual analysis of the added abbreviations shows that many useful syn­onyms were added to the query, which should, in theory, help to retrieve additional passages and boast the rankings of other relevant passages.</p><p>However, there is one big drawback to abbre­viation expansion that isn't characteristic in other types of expansion we have explored: abbreviations are much more ambiguous. For example, the ab­breviation "AD" is a very commonly used refer­ence to "Alzheimer's disease"; however, according to ADAM, "AD" has 35 unique long forms defined in MedLINE abstracts. For example, "AD" can also refer to the phrases "after discharge", "autosomal dominant", "autistic disorder", and other unrelated concepts.</p><p>IR researchers have found that query-term ambi­guity is less of a problem than one might expect be­cause of the <i>query term collocation effect </i>(Krovetz and Croft, 1992): query terms mutually disam-biguate each other because their intended senses tend to co-occur together in relevant documents in the collection. For example, for the query term "cell", adding the term "blood" to the query ensures that documents using the biological sense are ranked higher. Hence, one would expect that despite ab­breviation ambiguity, great gains in IR effectiveness would be possible using expansion. However, when the total number of possible unabbreviated forms is factored into the expansion process, it is clear that an excessive amount of ambiguity is added in.</p><p>A manual analysis of the results backs up this observation: although new relevant passages con­taining abbreviations are being retrieved, paragraph ranking is being affected to such an extent that pre­viously retrieved passages are "dropping out" of the top 1000 items in the ranked list.</p><p>However, our results also show that dynamic ab­breviation expansion does not degrade performance as dramatically as expansion with ADAM. The feed­back process ensures that only abbreviations that occur in documents of high ranked passages, men­tioning all query concepts, are added to the query. Thus, these abbreviations have the highest potential for providing positive impact on retrieval effective­ness.</p><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Run</p></td><td class="cell"><p>Passage MAP</p></td><td class="cell"><p>Aspect MAP</p></td><td class="cell"><p>Document MAP</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Baseline</b></p></td><td class="cell"><p>0.0480</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>0.1838</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>0.3355</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>SYN</b></p></td><td class="cell"><p>0.0888f</p></td><td class="cell"><p>+85.0%</p></td><td class="cell"><p>P</p></td><td class="cell"><p>=</p></td><td class="cell"><p>0.005</p></td><td class="cell"><p>0.3499f</p></td><td class="cell"><p>+90.3%</p></td><td class="cell"><p>P</p></td><td class="cell"><p>&lt;</p></td><td class="cell"><p>0.001</p></td><td class="cell"><p>0.4711t</p></td><td class="cell"><p>+40.4%</p></td><td class="cell"><p>P</p></td><td class="cell"><p>=</p></td><td class="cell"><p>0.008</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>SYN+HYPO</b></p></td><td class="cell"><p>0.0878f</p></td><td class="cell"><p>+83.0%</p></td><td class="cell"><p><i>P</i></p></td><td class="cell"><p>=</p></td><td class="cell"><p>0.007</p></td><td class="cell"><p>0.3417t</p></td><td class="cell"><p>+85.9%</p></td><td class="cell"><p>P</p></td><td class="cell"><p>=</p></td><td class="cell"><p>0.001</p></td><td class="cell"><p>0.4632t</p></td><td class="cell"><p>+38.1%</p></td><td class="cell"><p>P = 0.02</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>SYN+HYPER</b></p></td><td class="cell"><p>0.0933f</p></td><td class="cell"><p>+ 94.4%</p></td><td class="cell"><p><i>P</i></p></td><td class="cell"><p>&lt;</p></td><td class="cell"><p>0.001</p></td><td class="cell"><p>0.3695f</p></td><td class="cell"><p>+101%</p></td><td class="cell"><p>P</p></td><td class="cell"><p>&lt;</p></td><td class="cell"><p>0.001</p></td><td class="cell"><p>0.4843t</p></td><td class="cell"><p>+44.3%</p></td><td class="cell"><p>P</p></td><td class="cell"><p>=</p></td><td class="cell"><p>0.002</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>SYN+HYPER+VAR</b></p></td><td class="cell"><p>0.0949j</p></td><td class="cell"><p>+ 97.6%</p></td><td class="cell"><p><i>P</i></p></td><td class="cell"><p>&lt;</p></td><td class="cell"><p>0.001</p></td><td class="cell"><p>0.3827t</p></td><td class="cell"><p>+108%</p></td><td class="cell"><p>P</p></td><td class="cell"><p>&lt;</p></td><td class="cell"><p>0.001</p></td><td class="cell"><p>0.5080t</p></td><td class="cell"><p>+51.4%</p></td><td class="cell"><p>P</p></td><td class="cell"><p>&lt;</p></td><td class="cell"><p>0.001</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="8" global="107"/><p>Table 2: Table showing effect on system performance when additional expansion terms are added from the ADAM abbreviation (+Adam) database and our system Abbreviation feedback loop (+Abbr).</p><p><u>Run</u> <b><u>I</u> </b><u>Passage MAP</u> <u>|</u> <u>Aspect MAP</u> <u>|</u> <u>Document MAP</u></p><doubt alpha="33.3" length="33" tooSmall="False" monospace="0.0">SYN+HYPER+VAR0.0949 0.3827 0.5080</doubt><doubt alpha="24.1" length="87" tooSmall="False" monospace="0.0">SYN+HYPER+VAR+Adam0.0600f-36.8%P&lt;0.001 0.2387f-37.6%P&lt;0.001 0.4105t-19.2%     P = 0.001</doubt><doubt alpha="17.1" length="105" tooSmall="False" monospace="0.0">SYN+HYPER+VAR+Abbr0.0920 -3.06%        P = 0.3 0.3784 -1.12%         P = 0.4 0.5171 +1.79%        P = 0.3</doubt><table caption="Table 3: Table showing effect of two passage extraction strategies A and B on system performance"></table><p>The general conclusion from these abbreviation expansion experiments is clear: knowledge of these synonymous instances is obviously beneficial, but a method that reduces the impact of their high ambi­guity is necessary. We discuss our proposed solution to this problem in Section 5.</p><p>Our final experiment (see Table 3) shows that the TREC's Passage MAP score can be increased by capturing the exact answer span in each rele­vant paragraph. Section 3 proposed two methods for achieving this: <i>Method A </i>finds the longest text span in paragraph that contains all query terms; <i>MethodB </i>splits the span and remove sentences if there is a dis­tance of one or more sentences between consecutive mentions of any of the query terms. Both reduction methods show improvements in Passage MAP, but at the expense of the other two metrics. This is to be expected, especially in the case of <i>Method </i><i>B,</i><i> </i>since splitting paragraphs means some relevant passages may get a lower rank or even drop out of the top 1000 passages.</p><p>Table 4 shows how our best run <b>(Best+B) </b>performs with respect to systems that partici­pated in the official TREC 2006 Genomics Track. TREC JMEDIAN is the median value for each MAP score reported at TREC. UIC_TREC<footnote anchor="8"/> was the top performing system submitted by the University of Illinois at Chicago, and UIC.SIGIR is the best post­submission Passage MAP score which was also pub­lished by the same group (Zhou et al., 2007). If our system had participated at TREC track we would have ranked 6th for Passage MAP, 3rd for Aspect MAP and 4th for Document MAP out of 92 submit­ted runs.</p><p>Table 4: Table showing performance of our best Pas­sage MAP scoring run Best+B with the top perform­ing TREC systems on the Genomics Track</p><footnote label="8">The official name for this run was UICGenRun3.</footnote></subsection></section><section number="5" title="Discussion and Conclusions"><p>The most successful systems at the TREC Genomics Track 2006 used a combination of expansion tech­niques from external resources such as publically-available and hand-crafted thesauri, in addition to lexical variant generation techniques similar to the one described in this paper. One of the principal contributions of this paper is our detailed analysis of what types of ontologically related terms (syn­onyms, hyponyms, hypernyms, lexical variants, ab­breviations) provide the most impact when used as expansion terms. In particular, we have focussed on abbreviation expansion, which has high potential for impact when passages rather than full documents are being retrieved. However, our experiments show that their high ambiguity can in some cases reduce retrieval effectiveness.</p><p>There are two possible solutions to the abbrevia­tion ambiguity problem: all abbreviations in the col­lection are identified in advance of indexing, and a unique identifier is assigned to each long-form and its corresponding abbreviated short-form. Hence, when the query is expanded, the unique identifier rather than the lexical form of the abbreviation is added to the query. Similarly, all abbreviations in the collection will be replaced by their identifier before passage indexing occurs.<page local="9" global="108"/> Another possible ap­proach would be to explicitly add the long-forms of abbreviations in a passage to its index entry. This is a document expansion rather than a query expan­sion strategy. We plan to investigate both of these methods in our future work.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Run</p></td><td class="cell"><p>Passage MAP</p></td><td class="cell"><p>Aspect MAP</p></td><td class="cell"><p>Document MAP</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Best</b></p><p><b>Best+A</b></p><p><b>Best+B</b></p></td><td class="cell"><p>0.0920</p><p>0.1100t     +19.6%     P&lt; 0.001 0.1175t     +27.7%     P&lt; 0.001</p></td><td class="cell"><p>0.3784</p><p>0.3673       -2.93%         P = 0.3 0.3518t     -7.03%     P = 0.004</p></td><td class="cell"><p>0.5171</p><p>0.5123     -0.93%       P = 0.3 0.5021     -2.90%     P = 0.08</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Run</p></td><td class="cell"><p>Passage</p></td><td class="cell"><p>Aspect</p></td><td class="cell"><p>Document</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>MAP</p></td><td class="cell"><p>MAP</p></td><td class="cell"><p>MAP</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>UICSIGIR</b></p></td><td class="cell"><p>0.1823</p></td><td class="cell"><p>0.3811</p></td><td class="cell"><p>0.5391</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>UICTREC</b></p></td><td class="cell"><p>0.1479</p></td><td class="cell"><p>0.3492</p></td><td class="cell"><p>0.5320</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Best+B</b></p></td><td class="cell"><p>0.1175</p></td><td class="cell"><p>0.3518</p></td><td class="cell"><p>0.5021</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>TREC.MEDIAN</b></p></td><td class="cell"><p>0.0345</p></td><td class="cell"><p>0.1581</p></td><td class="cell"><p>0.3083</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Another area for potential improvement that we wish to investigate further is paragraph reduc­tion. Passage MAP is severely affected by long­answer text spans. Paragraph reduction is simi­lar to answer extraction in factoid-based Question-Answering tasks. However, researchers have only recently begun to investigate answer extraction for more complex question types such as <i>Why </i>or <i>How </i>questions in an ad hoc retrieval setting (Allan, 2005). The Document Understanding Conference (DUC), which focusses on summarisation tasks, is also look­ing at complex questions; however, answers are typ­ically generated by collating information from mul­tiple documents (Dang, 2006).</p></section><references><p>J. Allan. 2005. Hard track overview in TREC 2005: High accuracy retrieval from documents. In <i>The Four­teenth Text REtrieval Conference (TREC 2005) Pro­ceedings.</i></p><p>S. Buttcher, C.L.A. Clarke, and G.V. Cormack. 2004. Domain-specific synonym expansion and validation for biomedical information retrieval. In <i>The Thirteen Text REtrieval Conference (TREC 2004) Proceedings.</i></p><p>H.T. Dang. 2006. Overview of duc 2006. In <i>The Doc­ument Understanding Conference Workshop Proceed­ings.</i></p><p>E. N. Efthimiadis. 1996. Query expansion. <i>Annual Re­view of Information Science and Technology, </i>31:121— 187.</p><p>W. Hersh, A. Cohen, P. Roberts, and H. Rekapalli. 2006.</p><p>Trec 2006 genomics track overview. (Voorhees and</p><p>Buckland, 2006).</p><p>Robert Krovetz and W. Bruce Croft. 1992. Lexical ambi­guity and information retrieval. <i>Information Systems, </i>10(2):115—141.</p><p>Yi Li. 2007. Probabilistic toponym resolution and ge­ographic indexing and querying. Masters thesis, The University ofMelbourne.</p><p>Jimmy Lin and Dina Demner-Fushman. 2006. The role of knowledge in conceptual retrieval: a study in the domain of clinical medicine. In <i>SIGIR '06:</i><i> Proceed­ings of the 29th annual international ACM SIGIR con­ference on Research and development in information retrieval, </i>pages 99—106.</p><p>S. E. Robertson, S. Walker, S. Jones, M. Hancock-Beaulieu, and M. Gatford. 1994. Okapi at TREC-3.</p><p>In <i>The Third Text Retrieval Conference (TREC 3) Pro­ceedings, </i>Gaithersburg, Maryland, November.</p><p>I. Ruthven and M. Lalmas. 2003. A survey on the use of relevance feedback for information access systems. <i>Knowledge Engineering Review</i>, 18(2):95—145.</p><p>Nicola Stokes, Yi Li, Alistair Moffat, and Jiawen Rong. 2008. An empirical study of the effects of NLP com­ponents on Geographic IR performance. <i>International Journal of Geographical Information Science. </i>To ap­pear.</p><p>D. Trieschnigg, W. Kraaij, and F. de Jong. 2006. The influence of basic tokenization on biomedical docu­ment retrieval. In <i>SIGIR 2007 Proceedings, </i>Amster­dam, The Netherlands, July.</p><p>E. M. Voorhees and Lori P. Buckland. 2006. <i>The Fif­teenth Text REtrieval Conference (TREC 2006) Pro­ceedings. </i>NIST, Gaithersburg, Maryland.</p><p>W. Zhou, V. I. Torvik, and N. R. Smalheiser. 2006a. ADAM: another database of abbreviations in MED-LINE. <i>Bioinformatics, </i>22(22):2813—2818.</p><doubt alpha="60.0" length="50" tooSmall="False" monospace="0.0">W.Zhou, C. Yu,V.Tovik, and N. Smalheiser. 2006b. A</doubt><p>concept-based framework for passage retrieval in ge-nomics. (Voorhees and Buckland, 2006).</p><p>Wei Zhou, Clement Yu, Neil Smalheiser, Vetle Torvik, and Jie Hong. 2007. Knowledge-intensive conceptual retrieval and passage extraction of biomedical litera­ture. In <i>SIGIR '07: Proceedings ofthe 30th annual international ACMSIGIR conference on Research and development in information retrieval, </i>pages 655—662, New York, NY, USA. ACM Press.</p></references></body></article>