<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="9"/><title>Answerfinder: Question Answering by Combining Lexical, Syntactic and Semantic Information</title><author surname="Molla" givenname="Diego"><org  name="Macquarie University" country="Australia" city="Sydney"/></author><author surname="Gardiner" givenname="Mary"><org  name="Mie University" country="Japan" city="Mie"/></author></firstpageheader><frontmatter><p><b>AnswerFinder — Question Answering by Combining Lexical, Syntactic and Semantic Information</b></p><p><b>Diego Molla </b>and <b>Mary Gardiner</b></p><p>Centre for Language Technology Division of Information and Communication Sciences Macquarie University Sydney, Australia diego@ics.mq.edu.au<b> </b>and gardiner@ics.mq.edu.au</p></frontmatter><abstract>We present a question answering system that combines information at the lexical, syntactic, and semantic levels, in the process to find and rank the candidate answer sentences. The can­didate exact answers are extracted from the candidate answer sentences by means of a com­bination of information-extraction techniques (named entity recognition) and patterns based on logical forms. The system participated in the question answering track of TREC 2004. </abstract></header><body><section number="1" title="Introduction"><p>Question answering is an area that is becom­ing increasingly active in research and is cur­rently being deployed into practical applica­tions. Research in question answering has re­cently been fostered by large-scale programs like AQUAINT<footnote anchor="1"/> and evaluation frameworks like TREC<footnote anchor="2"/>, NTCIR<footnote anchor="3"/>,and CLEF<footnote anchor="4"/>. Such research and the current need to cope with large vol­umes of text has led various companies to pro­duce practical question answering systems. For example, research groups from Microsoft, IBM, NTT, Oracle, and Sun have participated in the question answering track of TREC. In addition, there are several attempts to provide question-answering extensions to the current Web search engines, with demos available by MIT<footnote anchor="5"/>,LCC<footnote anchor="6"/>, and BrainBoost<footnote anchor="7"/>, among others.</p><p>AnswerFinder is an open-domain question an­swering system that combines information at the lexical, syntactic, and semantic levels in various stages to find the exact answer to the user question. This paper describes the An­swerFinder system as it stood at the time of the</p><footnote label="1"> www.ic-arda.org/InfoExploit/aquaint/index.html  2 trec.nist.gov</footnote><footnote label="3">research.nii.ac.jp/ntcir/index-en.html 4 clef.iei.pi.cnr.it</footnote><footnote label="5"> www.ai.mit.edu/projects/infolab/</footnote><footnote label="6"> www.languagecomputer.com/</footnote><footnote label="7"> www.brainboost.com/</footnote><p>TREC 2004 question answering track. Section 2 introduces TREC and the question answering track. Section 3 describes the architecture of the system. Section 4 details the function of each module within the AnswerFinder system. Section 5 gives the system performance on the TREC 2003 question set. Section 6 mentions re­lated work and Section 7 lists problems with the AnswerFinder system that should be addressed in the near future.</p></section><section number="2" title="The TREC 2004 Question Answering Track"><p>The Text REtrieval Conference (TREC) started in 1992 as part of the TIPSTER text program. A fundamental goal of the conference is to pro­vide an evaluation framework for the compar­ison of the results of independent information retrieval systems. The concept of information retrieval is to be understood in a broad sense, and this conference has developed various tracks that focus on specific areas of information re­trieval, such as ad-hoc (the name given to docu­ment retrieval), routing, speech, cross-language, web, video, and very large corpora (Voorhees, 2003).</p><p>The question answering track started in 1999 and ever since its creation it has been the most popular track. Every year the complexity and difficulty of the task increases. Thus, in 1999 the competing systems were asked to retrieve small snippets of text containing the answer. The questions were designed by the participants and the answer was guaranteed to be in the text corpus. In the 2004 competition, in contrast, the questions were extracted from logs of real questions, the answer is not guaranteed to be in the corpus, and the systems were asked to find the exact answers of factoid questions and list questions. The questions were grouped into targets, each target containing fact-based ques­tions and list questions (explicitly marked as such), plus a question asking to find any other information relevant to the target.<page local="2" global="10"/> The ques­tions were encoded in XML as shown in Fig­ure 1. In this example, the target is <i>Fred Durst</i>, so question with ID number 2.2 in the figure is asking <i>What record company is Fred Durst with?.</i></p><doubt alpha="54.8" length="42" tooSmall="False" monospace="0.0">&lt;target id = "2" text = "Fred Durst"&gt; &lt;qa&gt;</doubt><doubt alpha="48.3" length="29" tooSmall="False" monospace="0.0">&lt;q id = "2.1" type="FACTOID"&gt;</doubt><doubt alpha="64.4" length="45" tooSmall="False" monospace="0.0">What is the name of Durst's group? &lt;/q&gt; &lt;/qa&gt;</doubt><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">&lt;qa&gt;</doubt><doubt alpha="63.9" length="61" tooSmall="False" monospace="0.0">&lt;q id = "2.2" type="FACTOID"&gt; What record company is he with?</doubt><doubt alpha="30.0" length="10" tooSmall="False" monospace="0.0">&lt;/q&gt; &lt;/qa&gt;</doubt><doubt alpha="64.2" length="67" tooSmall="False" monospace="0.0">&lt;q id = "2.3" type="LIST"&gt; What are titles of the group's releases?</doubt><doubt alpha="60.8" length="51" tooSmall="False" monospace="0.0">&lt;q id = "2.4" type="FACTOID"&gt; Where was Durst born?</doubt><doubt alpha="44.4" length="27" tooSmall="False" monospace="0.0">&lt;q id = "2.5" type="OTHER"&gt;</doubt><doubt alpha="50.0" length="16" tooSmall="False" monospace="0.0">Other &lt;/q&gt; &lt;/qa&gt;</doubt><doubt alpha="66.7" length="9" tooSmall="False" monospace="0.0">&lt;/target&gt;</doubt><figure caption="Figure 1: A hand-made example of a group of questions using the TREC 2004 format"></figure><p>The corpus of supporting text was the AQUAINT corpus, which comprises over 1 mil­lion news articles taken from the New York Times, the Associated Press, and the Xin­hua News Agency newswires. This corpus is not large in comparison with the terabites of text available via the Internet, but it is still large enough to require the need to resort to shallow-processing preselection methods before performing a real attempt to find the answer.</p><p><b>3System</b><b> Overview</b></p><p>The question answering procedure used by An­swerFinder follows a pipeline structure that is typical of rule-based question answering sys­tems. The process is outlined in Figure 2 and is as follows:</p><p>1. All questions are normalised, so that <i>What record company is he with? </i>in Figure 1 be­comes <i>What record company is Fred Durst with?</i></p><p>2. The questions are classified into types based upon their expected answer. So the question <i>How far is it from Mars to Earth? </i>would be classified as a "Number" question as it expects a numeric value in response.</p><p>3. 100 candidate answer sentences are ex­tracted from the corpus.</p><p>4. The 100 sentences are re-scored based upon their word overlap, grammatical relations overlap, and flat logical form overlap with the question text.</p><p>5. Exact answers —fragments like <i>416 million miles</i>— are extracted from the candidate answer sentences.</p><p>6. The exact answer list is sorted, re-scored and filtered for duplicate exact answers.</p><p>7. A number of exact answers from the top of the list are selected, depending on the question type.</p><p>AnswerFinder uses the following knowledge sources to analyse the question and to select from among possible answers:</p><p><b>Named entity data </b>generated by the GATE system (Gaizauskas et al., 1996), marking pieces of text in the AQUAINT corpus as one of the types <i>Date, Location, Money, Organization</i>,and<i>Person. </i>These data are generated off-line before any question is processed. GATE's analysis was extended with a simple set of regular expressions that detect numbers as well.</p><p><b>The list of preselected documents</b> provided by the US National Insti­tute of Standards and Technology (NIST), containing for each target entity, the 1,000 top scoring documents for that entity. NIST co-sponsors the TREC conferences and it obtained the list of preselected documents by running the target query through the PRISE (Harman and Candela, 1990) document retrieval system.</p><page local="3" global="11"/><doubt alpha="81.2" length="16" tooSmall="True" monospace="0.0">Select Answer(s)</doubt><doubt alpha="77.8" length="9" tooSmall="True" monospace="0.0">Answer(s)</doubt><figure caption="Figure 2: System overview"></figure></section><section number="4" title="Modules"><subsection number="4.1" title="Question Normalisation"><p>AnswerFinder relies heavily on the common in­formation found between a question and the candidate answer sentence. Therefore questions like <i>What record company is he with? </i>need to undertake an anaphora resolution process to de­termine that <i>he </i>in fact refers to <i>Fred Durst.</i></p><p>Questions in the TREC 2004 competition co-referred with previous questions or with their target in a number of ways.</p><p>Questions might co-refer with their target pronominally:</p><p><b>Target: </b><i>Fred Durst</i> <b>Q </b>: <i>What record company is he with?</i></p><p>Questions might co-refer with their target us­ing a definite noun phrase:</p><p><b>Target </b>: <i>Club Med</i> <b>Q:</b><b> </b><i>How many Club Med vacation spots are there worldwide?</i></p><p>Questions might co-refer with another ques­tion:</p><p><b>Q2.1:   </b><i>What is the name of Durst's group?</i></p><p><b>Q</b>z3: <i>What are titles of the group's re­leases?</i></p><p>Finally, questions may relate to their target associatively, that is, there may not be a direct co-reference:</p><p><b>Target </b>: <i>Heaven's Gate</i> <b>Q </b>: <i>When did the mass suicide occur?</i></p><p>AnswerFinder normalises questions in the first case, where the question co-refers with the target pronominally. It performs a simple re­placing of pronouns in the question with the target text, forming a regular plural and pos­sessive where necessary, as shown in Table 1.</p><p>Finally, "other" type questions, which were of the generic form <i>other, </i>were transformed into <i>What is TARGET? </i>so that question 2.5 in Fig­ure 1 is transformed into <i>What is Fred Durst? </i>This was a crude attempt at doing something useful with the "other" type questions. Clearly a more detailed processing of these questions is required.</p></subsection><subsection number="4.2" title="Question Classification"><p>Particular question words signal particular named entity types required as a response. The example below requires a person's name in re­sponse to the question:</p><p><i>Who founded the Black Panthers orga­nization?</i></p><p>AnswerFinder uses a set of 29 regular expres­sions to determine what named entity type a question requires in its response from the list <i>person, date, location, money, number, city, or­ganization, percent, country, state, river, name, unknown. </i>The regular expressions were devel­oped with the question set from TREC 2002, and they produced an accuracy of 78.6% correct classifications. This figure is lower than the one reported by other systems like the ones by Pasca and Harabagiu (2001) or Zhang and Lee (2003), each of which reported an accuracy of 90% or over. The question classification module clearly needs further refinement, but an evaluation with the question set from TREC 2003 showed an ac­curacy of 77%, thus indicating that the regular expressions generalise well.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Rescore</p></td><td class="cell"><p></p></td><td class="cell"><p>Extract</p></td><td class="cell"><p></p></td><td class="cell"><p>Rescore</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b><i>-&gt;</i></b></p></td><td class="cell"><p>Exact</p></td><td class="cell"><p></p></td><td class="cell"><p>Exact</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Sentences</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Answers</p></td><td class="cell"><p></p></td><td class="cell"><p>Answers</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="4" global="12"/><p><i>What record company is </i><i>he </i><i>with? —&gt;   What record company is </i><i>Fred Durst</i> <i>How many of </i><i>its </i><i>members committed —&gt;   How many of </i><i>Heaven's Gate's </i><i>mem-</i> <i>suicide? bers committed suicide?</i> <i>In what countries are </i><i>they </i><i>found?</i><i> —&gt;   In what countries are </i><i>agoutis </i><i>found?</i></p><doubt alpha="80.0" length="5" tooSmall="False" monospace="0.0">with?</doubt><table caption="Table 1: Examples of pronoun resolution performed by AnswerFinder"></table></subsection><subsection number="4.3" title="Candidate Sentence Extraction"><p>Given the set of AQUAINT documents prese­lected by the NIST document retrieval system, AnswerFinder selects 100 sentences from these documents as candidate answer sentences.</p><p>Candidate sentences are selected in the fol­lowing way:</p><p>1. The 1,000 preselected documents provided by NIST for each target are split into sen­tences by means of a simple sentence split­ting process.</p><p>2. Each sentence is assigned a numeric score: 1 point for each distinct non-stopword over­lapping with the question string, and 10 points for the presence of one or more named entities of the right type. This way we reward heavily the presence of a string of the expected answer type.</p><p>3. The 100 top scoring sentences are returned as candidate answer sentences.</p><p>As an example of the scoring mechanism, con­sider this question/sentence pair:</p><p><b>Q:  </b><i>How far is it from </i><i>Mars </i><i>to </i><i>Earth?</i></p><p><b>A: </b><i>According to evidence from the SNC meteorite, which fell from </i><i>Mars </i><i>to </i><i>Earth </i><i>in ancient times, the water concentration in Martian mantle is es­timated to be </i><i>40 ppm, </i><i>far less than the terrestrial equivalents.</i></p><p>The question and sentence have 2 shared non-stopwords: <i>Mars </i>and <i>Earth. </i>Further, this sen­tence has a named entity of the required type (Number): <i>40 ppm, </i>making the total score for this sentence 12 points.</p></subsection><subsection number="4.4" title="Sentence Re-Scoring"><p>The goal of all the above modules is to reduce the corpus of text to a list of the 100 sentences with highest likelihood to contain an answer.</p><p>The sentence re-scoring module uses a combi­nation of lexical, syntactic, and semantic infor­mation to perform a more detailed analysis of these sentences:</p><p><b>lexical: </b>The   combined   word   overlap and named entity score.</p><p><b>syntactic: </b>The grammatical relation overlap score.</p><p><b>semantic: </b>Overlaps with flat logical form pat­terns.</p><p>We have seen the use of lexical information in Section 4.3. Below we will see the use of gram­matical relations and flat logical form patterns, and the final combinations used in TREC 2004.</p><subsubsection number="4.4.1" title="Grammatical Relation Overlap Score"><p>The grammatical relations were initially devised by Carroll et al. (1998) as a means to normalise the output of parsers for their comparative eval­uation. The set of grammatical relations repre­sent some of the common relations that exist between the words in a sentence, a selection of which is shown in Table 2. To build the gram­matical relations of questions and answer can­didate sentences, AnswerFinder processes the output of the Connexor Dependency Functional Grammar, which is a dependency-based robust parser with a wide-coverage grammar of English (Tapanainen and Jarvinen, 1997). Below is an example of the grammatical relations of a ques­tion and an answer candidate sentence.</p><p><b>Q </b>: <i>How far is it from Mars to Earth?</i></p><doubt alpha="57.1" length="14" tooSmall="False" monospace="0.0">(subj be it _)</doubt><p>(xcomp from be mars) <b>(ncmod earth from to)</b> <b>A:</b><b> </b><i>It is 416 million miles from Mars to Earth.</i></p><doubt alpha="62.5" length="16" tooSmall="False" monospace="0.0">(ncmod _ be far)</doubt><doubt alpha="64.7" length="17" tooSmall="False" monospace="0.0">(ncmod _ far how)</doubt><doubt alpha="66.7" length="36" tooSmall="False" monospace="0.0">(ncmod earth from to) (subj be it _)</doubt><p>(ncmod from be mars)<page local="5" global="13"/></p><doubt alpha="62.9" length="62" tooSmall="False" monospace="0.0">(xcomp _ be mile) (ncmod _ million 416) (ncmod _ mile million)</doubt><p>The score is the number of relations shared between question and sentence. In the example above, the overlap between the grammatical re­lations of question and candidate sentence is 2, corresponding to the two grammatical relations marked in boldface.</p></subsubsection><subsubsection number="4.4.2" title="Flat Logical Form Patterns"><p>In previous research we have developed a flat notation for the logical forms of sentences and a method to produce the logical forms from ar­bitrary sentences by traversing their syntactic structures (Molla, 2001; Mollâ and Hutchinson, 2002). These flat logical forms have been used to determine the likelihood that a sentence con­tains the answer by checking the semantic sim­ilarity of the question with the sentence. In a similar fashion to grammatical relations, the se­mantic similarity of two sentences is the number of logical terms shared between them. Thus if we have the following logical forms:</p><doubt alpha="62.9" length="132" tooSmall="False" monospace="0.0">Q:What is the population ofIceland?object(iceland, o6, [x6])object(population, o4, [x1])object(what, o1, [x1])prop(of, p5, [x1, x6])</doubt><doubt alpha="66.7" length="36" tooSmall="False" monospace="0.0">A:Iceland has a population of 270000</doubt><doubt alpha="23.8" length="21" tooSmall="False" monospace="0.0">dep(270000, d6, [x6])</doubt><doubt alpha="66.7" length="27" tooSmall="False" monospace="0.0">object (population,o4,[x4])</doubt><doubt alpha="65.2" length="23" tooSmall="False" monospace="0.0">object(iceland,o1,[x1])</doubt><doubt alpha="50.0" length="20" tooSmall="False" monospace="0.0">evt(have,e2,[x1,x4])</doubt><doubt alpha="47.4" length="19" tooSmall="False" monospace="0.0">prop(of,p5,[x4,x6])</doubt><p>The semantic similarity between the two sen­tences is 2, as the number of overlaps between the logical form of question and answer is 2 (overlap shown in boldface). Note that the com­putation of the overlap is complicated by the fact that logical terms include variables and it is necessary to keep the relation between the variables in the overlapping terms. Thus, in the example above, the variable <b>x1 </b>in the question terms corresponds with <b>x4 </b>in the answer candi­date sentence and therefore whenever <b>x1 </b>is used in the question, <b>x4 </b>must be used in the answer. A simple process of Prolog unification suffices to match the variables of the question terms with those of the sentence terms, by convert­ing the question term variables into real Prolog variables.</p><p>Since there are several ways to answer a ques­tion, for TREC 2004 we have developed a set of patterns to capture the expected logical form of sentences that contain the answer to questions. Below is the matching pattern associated with the template that we labelled as "what2" and one of its replacement patterns:</p><p><b>Template "what2" </b>: <b>Pattern:</b></p><doubt alpha="64.9" length="114" tooSmall="False" monospace="0.0">object(ObjX,VobjX,[VeX]), object (what,_,[Ve WHAT]), object(ObjY,VobjY,[VeWHAT]), prop (of, _, [VexistWHAT, VeX] )</doubt><p><b>Replacement 1:</b></p><p>dep(ANSWER,ANSW,[VeANSW]), object(ObjX,VobjX,[VeX]), object(ObjY,VobjY,[VeY])</p><doubt alpha="46.9" length="32" tooSmall="False" monospace="0.0">prop (of, _, [Ve Y, Ve ANS W] ),</doubt><doubt alpha="53.3" length="30" tooSmall="False" monospace="0.0">evt (have,_, [VeX, Ve WHAT] ),</doubt><p>Borrowing the notation of Prolog variables, the above template uses forms in uppercase or "_" to express the slots that can unify with log­ical form components. As the logical form of <i>What is the population of Iceland? </i>matches the dep(ANSWER,ANSW,[VeANSW]), pattern above (we use standard Prolog unifica­tion to perform the matching), then its logical form is transformed into:<page local="6" global="14"/></p><table caption="Table 2: Grammatical relations used in this paper" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Relation</i></p></td><td class="cell"><p><i>Description</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CONJ(type,head+)</p></td><td class="cell"><p>Conjunction</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MOD(type,head,dependent)</p></td><td class="cell"><p>Modifier</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CMOD(type,head,dependent)</p></td><td class="cell"><p>Clausal modifier</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NCMOD(type,head,dependent)</p></td><td class="cell"><p>Non-clausal modifier</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>DETMOD(type,head,dependent)</p></td><td class="cell"><p>Determiner</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SUBJ(head,dependent,initial_gr)</p></td><td class="cell"><p>Subject</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>OBJ(head,dependent,initial_gr)</p></td><td class="cell"><p>Object</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>DOB J(head, dependent, initiaLgr)</p></td><td class="cell"><p>Direct object</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>XCOMP(head,dependent)</p></td><td class="cell"><p>Clausal complement without an overt subject</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p><b>Q: </b><i>What is the population ofIceland?</i></p><doubt alpha="60.0" length="100" tooSmall="False" monospace="0.0">prop(of,_, [VeY,VeANSW]), ob ject(iceland,o6,[x6]), evt(have,_,[x6,xl]), object(population,o4,[VeY])</doubt><p>The semantic similarity between this logical form and the one of <i>Iceland has a population of 270000 </i>is now 5, since all five terms of the modified question logical form can be found in the logical form of the answer and all variables unify.</p><p>In addition to returning the overlap between a candidate sentence and a matching answer pattern, AnswerFinder uses the instantiation of the ANSWER variable to determine the answer: "270000" in the case of our example.</p><p>The introduction of flat logical form patterns parallels the use of patterns based on regular expressions, but using the logical level of a sen­tence instead of the surface level. This way it is hoped that less patterns are required to cover a broader range of sentences. In practice, however, the difficulty to read logical forms by humans slows down the production of patterns and replacements. As a result, a small set of 10 patterns were developed for our experiments in the questions from the TREC 2004 test set were covered by only 4 template patterns and there was an important number of questions that did not trigger any pattern.</p><doubt alpha="61.4" length="44" tooSmall="False" monospace="0.0">TREC 2004. As we can see in Table 3, most of</doubt><table caption="Table 3: Number of questions triggering each template; a question may trigger several tem­plates"></table><p>The patterns that were triggered most fre­quently were generic patterns that were intro­duced to maximise the coverage of the pattern set.  For example, the most frequent pattern, "what.generic", is defined so as to allow any noun to replace the word <i>what:</i></p><p><b>Template "what.generic" : Pattern:</b></p><doubt alpha="51.9" length="27" tooSmall="False" monospace="0.0">object ('what ',_, [XWho] )</doubt><p><b>Replacement:</b></p><p>object(_,ANSWER,[XWho])</p></subsubsection></subsection><subsection number="4.5" title="Exact Answer Extraction, Filtering and Scoring"><p>Having selected and re-ranked the 100 top-scoring candidate sentences, AnswerFinder then selects exact answer strings from within them. AnswerFinder combines the use of named enti­ties with that of logical form patterns:</p><p>1. For each candidate sentence, extract all named entities that match the question classification.</p><p>2. For each candidate sentence, extract AN­SWER values from any matching flat logi­cal form pattern.</p><p>Exact answers are scored as follows:</p><p>1. If the exact answer is a named entity, its score is the score of the candidate sentence it is found in.</p><p>2. If the exact answer is an ANSWER value from a flat logical form pattern, its score is the score of the candidate sentence it is found in.</p><p>3. If the exact answer is both a named entity and an ANSWER value from a flat logical form answer pattern, its score is twice the score of the candidate sentence it is found in.</p><p>If the same string is extracted from two an­swer sentences the score becomes the sum of the scores of the duplicate answers. This way an­swer redundancy is rewarded.</p></subsection><subsection number="4.6" title="Exact Answer Selection"><p>AnswerFinder selects the answers depending on the type of question:</p><p><b>Factoid questions </b>requiring exactly one an­swer: return the top scoring answer; or if there are no answers with a score more than 0, return "NIL" indicating that there were no answers.</p><table caption="Table 3: Number of questions triggering each template; a question may trigger several templates" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Template ID</i></p></td><td class="cell"><p><i>Num.</i></p></td><td class="cell"><p>Template ID</p></td><td class="cell"><p><i>Num.</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>howmanyl</p></td><td class="cell"><p>0</p></td><td class="cell"><p>howl</p></td><td class="cell"><p>1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>howmany2</p></td><td class="cell"><p>0</p></td><td class="cell"><p>who.generic</p></td><td class="cell"><p>39</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>what2</p></td><td class="cell"><p>3</p></td><td class="cell"><p>what_generic</p></td><td class="cell"><p>116</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>what3</p></td><td class="cell"><p>0</p></td><td class="cell"><p>what_noun</p></td><td class="cell"><p>69</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>what6</p></td><td class="cell"><p>1</p></td><td class="cell"><p><i>no match</i></p></td><td class="cell"><p>78</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>when!</p></td><td class="cell"><p>47</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="7" global="15"/><p><b>List questions and "other" questions</b> requiring a number of answers: return all exact answers within a threshold difference in score with the top score. If there are no exact answers with a score of more than 0, return the top scoring candidate sentence.</p><p><b>5Performance</b></p><p>After testing several combinations of lexical, syntactic, and semantic information (see the work by Mollaâ (2003) for the sort of analy­sis that we performed), AnswerFinder used two combinations of scores for the runs submitted</p><doubt alpha="46.2" length="13" tooSmall="False" monospace="0.0">to TREC 2004:</doubt><p><b>3gro+lfo </b>3 times the grammatical relation overlap score added to the flat logical form pattern overlap score. This combination was chosen because it gave the best results in our preliminary experiments with ques­tion sets taken from past TREC QA con­ferences.</p><p><b>lfo </b>The flat logical form pattern overlap score.</p><p>Although not explicitly expressed in the above combinations, lexical information is used implicitly because the scoring is based on the output of a preselection module that did use solely lexical information (word overlap and named entities), as we have seen in Section 4.3.</p><p>In a preliminary analysis of the system we used the answer patterns provided by Ken Litkowsky via NIST. These answer patterns cover all the answers found by the systems par­ticipating in TREC 2003. We tested our system with the TREC 2003 questions and checked the output of the sentence re-scoring module and the final output of the system. We found that the re-scoring module gave the highest score to a sentence containing the answer about 20% of times. In contrast, the final system returned a correct and exact answer about 5% of times.</p><p>The results of our participation in TREC 2004 are significantly better. In all runs, the accuracy of the factoid questions is 10%, the F-score of the list questions is 0.08, and the F-score of the "other" questions is 0.09. Since the system was not fine-tuned for the list or "other" questions only the results of the factoid ques­tions need to be considered. We believe that the reason for the better results of the factoid question with respect to our preliminary anal­ysis is that Litkowsky's patterns that we used did not cover all cases of good answers. In fact, we tried the patterns on the answers submitted to TREC 2004 and we obtained an accuracy of 8.59%, which is between the accuracy given by our preliminary experiments and the one given by the TREC human assessors. Comparatively with the other systems participating in TREC 2004, the results are below the median of the results returned by all the systems (which was 17%). Also, surprisingly, all of our runs had vir­tually the same results. These unusual results led us to suspect that a chain of bugs may have made the system ignore the information pro­vided by the logical form patterns. Currently we are analysing the results.</p></subsection></section><section number="6" title="Related Work"><p>AnswerFinder as it stood in TREC 2004 differs from previous versions in several aspects. First of all, now AnswerFinder uses the Named En­tity data that has been pre-calculated on the entire AQUAINT corpus. This way the system does not need to spend precious time during the on-line stage when the user is waiting for the an­swer of the question. Also, in contrast with An-swerFinder's participation in TREC 2003 where it focused on the extraction of passages contain­ing the answer, now AnswerFinder extracts ex­act answers and attempts to answer list and def­inition questions. In the process, AnswerFinder uses a set of templates based on patterns of log­ical forms.</p><p>The overall architecture of AnswerFinder is similar to that of other question answering sys­tems. The aim is to gradually reduce the amount of text to process through several levels of increasing complexity. We use an informa­tion retrieval system to preselect the documents and information from named entities and the expected answer type obtained from the ques­tion to reward the sentences that may contain the answer. One difference that sets our sys­tem apart from the majority is the use of log­ical forms in the process to further scope the sentences that are most likely to contain the an­swer. Other question answering systems use log­ical forms (Harabagiu et al., 2001, for example) that were developed independently from our re­search.</p><p>But the main difference with respect to other systemsistheuseofpatternsderivedfromlog-ical forms to determine the exact answer. A baseline method that would return the text tagged by the named entity recogniser has been used by various systems. Adding further complexity, systems like the one developed by Echi-habi et al. (2004) use patterns based on named entities and parts of speech.<page local="8" global="16"/> However, we are not aware of any other system besides An­swerFinder that tries to use logical form pat­terns.</p></section><section number="7" title="Conclusions and Further Work"><p>AnswerFinder is a question answering system that uses a combination of lexical, syntactic, and semantic information to find the answer to the user question. An early version of this sys­tem participated in the passages section of the 2003 TREC question answering track. After the equivalent of only 55 person-hours work, the system ranked above the median of the seven participating systems. For TREC 2004 we have included a named entity recogniser and a pro­cess to find exact answers that uses a combi­nation of patterns based on logical forms and named entities.</p><p>Current and future work focuses on the re­fining of the candidate sentence scoring, exact answer scoring, and pattern development. We also plan to work on a more detailed processing of list and definition questions.</p><p>For the refining of the sentence scoring, we are exploring the use of weighted measures for different types of terms in the flat logical forms. We are also exploring the integration of graph-based methods such as the ones developed by (Montes-y-Gomez et al., 2001).</p><p>For the exact answer scoring, we are de­veloping further logical form patterns to in­crease their coverage. We will also explore fuzzy matching methods so that every question will match at least one pattern.</p><p>To facilitate the discovery and development of logical form patterns, we are studying meth­ods to increase the readability of the flat logical forms by converting them into graph structures.</p></section><references><p>John Carroll, Ted Briscoe, and Antonio Sanfil-ippo. 1998. Parser evaluation: a survey and a new proposal. In <i>Proc. LREC98.</i></p><p>Abdessamad Echihabi, Ulf Hermjakob, Eduard Hovy, Daniel Marcu, Eric Melz, and Deepak Ravichandran. 2004. How to select an an­swer string? In Tomek Strzalkowski and Sanda Harabagiu, editors, <i>Advances in Tex­tual Question </i>Answering.Kluwer.</p><p>Robert Gaizauskas, Hamish Cunningham, Yorick Wilks,  Peter Rodgers,  and Kevin</p><p>Humphreys. 1996. GATE: an environment to support research and development in natural language engineering. In <i>Proceedings of the 8th IEEE International Conference on Tools with Artificial Intelligence, </i>Toulouse, France. Sanda Harabagiu, Dan Moldovan, Marius Pasca, Mihai Surdeanu, Rada Mihalcea, Rox-ana Glrju, Vasile Rus, Finley Lacatusu, and Razvan Bunescu. 2001. Answering complex, list and context questions with LCC's question-answering server. In Ellen M. Voorhees and Donna K. Harman, editors, <i>Proc.</i><i> TREC 2001, </i>number 500-250 in NIST Special Publication. NIST.</p><p>Donna K. Harman and Gerald Candela. 1990. Retrieving records from a gigabyte of text on a minicomputer using statistical ranking. <i>Journal of the American Society for Informa­tion Science, </i>41(8):581-589.</p><p>Diego Mollaâ and Ben Hutchinson. 2002. Dependency-based semantic interpretation foranswerextraction. In<i>Proc. 2002 Aus­tralasian NLP Workshop.</i></p><p>Diego Mollâa. 2001. Ontologically promiscuous flat logical forms for NLP. In Harry Bunt, Ielka van der Sluis, and Elias Thijsse, edi­tors, <i>Proceedings of IWCS-4, </i>pages 249-265. Tilburg University.</p><p>Diego Mollâa. 2003. Towards semantic-based overlap measures for question answering. In <i>Proc. ALTW03, </i>pages 130-137, Melbourne.</p><p>Manuel Montes-y-Gâomez, Alexander Gelbukh, and Ricardo Baeza-Yates. 2001. Flexi­ble comparison of conceptual graphs. In <i>Proc. DEXA-2001, </i>number 2113 in Lecture Notes in Computer Science, pages 102-111. Springer-Verlag.</p><p>Marius A. Pasca and Sanda M. Harabagiu. 2001. High performance question answering. In <i>Proc. SIGIR'01, </i>New Orleans, Luisiana,</p><p>USA. ACM.</p><p>Pasi Tapanainen and Timo Jarvinen. 1997. A non-projective dependency parser. In <i>Proc.</i></p><p><i>ANLP-97</i>.ACL.</p><p>EllenM.VoorheesandLoriP.Buckland, edi­tors. 2003. <i>The Twelfth Text REtrieval Con­ference (TREC 2003), </i>number 500-255 in</p><p>NIST Special Publication. NIST. Ellen M. Voorhees. 2003. Overview of TREC 2003. In Voorhees and Buckland (Voorhees and Buckland, 2003).</p><p>Dell Zhang and See Sun Lee. 2003. Question classification using support vector machines.</p><p>In <i>Proc. SIGIR 03</i>.ACM.</p></references></body></article>