<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="110"/><title>Towards semantic-based overlap measures for question-answering</title><author surname="Mollá" givenname="Diego"><org  name="Macquarie University" country="Australia" city="Sydney"/></author></firstpageheader><frontmatter><p><b>Towards Semantic-Based Overlap Measures for Question Answering</b></p><p><b>Diego Molla</b></p><p>Centre for Language Technology Macquarie University Sydney, NSW 2109 Tel. +61 2 9850 9531 Fax+61 2 9850 9551 diego@ics.mq.edu.au</p></frontmatter><abstract>In this paper we present an evalua­tion of overlap-based measures of sim­ilarity for sentences in the same lan­guage. The measures include syntactic and semantic information, and to that end they incorporate grammatical re­lations and flat logical forms. A full parser is required to build the above in­formation. Separate extrinsic evalua­tions within the context of question an­swering have been made with two dif­ferent parsers to test the impact of the parser and the overlap measures. </abstract></header><body><section number="1" title="Introduction"><p>Text-based Question Answering (QA) is a hot research topic and the increasing availability of electronic text will ensure that research in this area will continue for long. Much of the cur­rent research on QA focuses on the development of methodologies for processing relatively large volumes of text. For example, the competition-based QA track of the Text REtrieval Conference (TREC) (Voorhees, 2001) uses more than 3Gb of source text. Competing systems often exploit the data redundancy existing in the source text. Some of them even use the Web to increase the data re­dundancy (Brill et al., 2001; Clarke et al., 2001, for example). These systems typically trade ac­curacy for speed and avoid the use of intensive natural language processing techniques.</p><p>Most of the current QA systems are based on an architecture like that of Figure 1 (Hirschman and</p><p>Gaizauskas, 2001; Voorhees, 2001). In an off-line or indexing stage, an <b>indexing </b>module analyses the text documents and creates a set of document images that will be used by the subsequent QA modules. In an on-line stage, a <b>question analy­sis </b>module classifies the question and determines the type of the expected answer. The question analysis module would typically return a list of named-entity types that are compatible with the question (for example a <i>who </i>question typically indicates people or organisations). The module may also produce an image of the question. This image may be similar in format to the document images and can range from a simple bag ofwords (Cooper and Rüger, 2000, for example) to a fairly complex logical form (Harabagiu et al., 2001; El-worthy, 2000, for example). Once the question is analysed, a <b>document preselection </b>module iden­tifies the documents that are most likely to contain the answer. This module typically uses informa­tion retrieval techniques that rely on bag-of-words approaches and statistical information (Voorhees, 2001). A <b>filtering </b>module examines the resulting documents and selects or rewards the named enti­ties that are compatible with the question type. A <b>scoring </b>module then performs a more intensive analysis and ranks the preselected named entities (an possibly surrounding text) according to their likelihood to contain the answer. The scoring sys­tem relies on the output given by the question analysis module and possibly the images of the preselected documents that were created during the off-line stage. There may be feedback loops between the document preselection, filtering, and scoring modules to increase the likelihood of finding difficult answers (Harabagiu et al., 2001, for example).<page local="2" global="111"/></p><figure caption="Figure 1: Architecture of a generic question-answering system."></figure><p>The scoring methodology used can be as sim­ple as a word overlap or word frequency count, or as complex as an automatic proof system that op­erates on logical forms of both the questions and the answers. In some QA systems the scoring sys­tem relies heavily on the use of sentence patterns (Soubbotin, 2001, for example).</p><p>In the present study we have implemented a QA framework that uses simplifications of the modules described above. The emphasis in this study has been placed on the comparison of core methodologies for the scoring stage. To avoid the introduction of unwanted variables, we have avoided the use of methodologies that rely on world knowledge or domain knowledge. Thus, we have refrained from testing the use of exter­nal resources or inference systems.</p><p>With this QA framework we intend to assess the impact of syntactic and semantic information in a QA task. For that reason, we include infor­mation regarding word dependencies, grammati­cal relations, and logical forms in the procedure to measure the similarity between a question and an answer candidate. The results of the evaluations show both the impact of the scoring measures and the impact of the parsers used to extract the syn­tactic and semantic information. Section 2 de­scribes grammatical relations and their use in an overlap measure. Section 3 focuses on the overlap of flat logical forms. Section 4 introduces the QA system that was used for the evaluations, and Sec­tion 5 explains the methodology used in the au­tomatic evaluations. Finally, Section 6 discusses the results and Section 7 concludes and points to future lines of research.</p><figure caption="Figure 2: Hierarchy of grammatical relations."></figure></section><section number="2" title="Grammatical Relations"><p>We use the grammatical relations of Carroll et al. (1998), who devised them as a means to provide the canonical representation of the output of parsers for their evaluation. Figure 2 shows the hierarchical classification of the grammatical relations (Briscoe and Carroll, 2000). This hierarchy allows the mapping from the output of an arbitrary parser and therefore allow the evaluation of parsers with different output granularity.</p><p>Table 1 lists the grammatical relations used in this paper and the evaluation - For further detail about grammatical relations see (Briscoe and Car­roll, 2000). For example, the grammatical rela­tions for the sentence <i>The man that came ate ba­nanas and apples with a fork without asking </i>are:</p><doubt alpha="59.4" length="229" tooSmall="False" monospace="0.0">DETMOD ( _, man, the ) , CMOD(that,man,come), SUB J (come, man, _) , SUB J (eat, man, _) , DOB J ( eat, banana, _) , DOB J (eat, apple, _) CONJ(and,banana,apple), NCMOD(fork,eat,with), DETMOD (_, fork, a) , XCOMP(without,eat,ask)</doubt><p>Briscoe and Carroll's grammatical relations are not to be confused with the dependency arcs used in the theory of dependency grammar (Mel'cuk, 1988). To illustrate the difference, consider the sentence <i>The man that came ate bananas and</i> <i>apples with a fork.</i><page local="3" global="112"/><i> </i>Figure 3 shows the graph­ical representation of the structure returned by Conexor FDG, a dependency-based parsing sys­tem (Tapanainen and Jarvinen, 1997). In depen­dency grammar a unique head is assigned to each word, thus the head of <i>man </i>is <i>ate. </i>However <i>man </i>is the dependent of more than one gram­matical relation, namely SUB J (eat, man, _) and SUB J (come, man, _). Furthermore, in de­pendency grammar a word can have at most one dependent of each argument type, and so <i>ate </i>can have at most one object. But the same is not true for grammatical relations, and we get both OBJ (eat, banana, _) and OBJ (eat, apple, _). Thus, grammatical re­lations provide a sentence representation that is closer to the semantic contents of a sentence than the representation provided by dependency arcs.</p><p>Molla and Hutchinson (2003) used the gram­matical relations to compare the accuracy of two broad-coverage dependency-based parsers, Link Grammar (Sleator and Temperley, 1993) and Conexor Functional Dependency Grammar (Tapanainen and Jarvinen, 1997) — henceforth referred to as Conexor FDG. The evaluation used a subset of the original relations: SUBJ, OBJ, XCOMP, and MOD. This subset was used because of limitations of the output of the parsers and the algorithms for the automatic construction of the grammatical relations. Thus, the reduced gram­matical relations for the example <i>The man that came ate bananas and apples with a fork without asking </i>is:</p><doubt alpha="57.1" length="42" tooSmall="False" monospace="0.0">MOD(that,man,come), SUB J ( eat, man, _) ,</doubt><doubt alpha="59.5" length="111" tooSmall="False" monospace="0.0">SUBJ (come, man, _) , OBJ ( eat, banana, _) , OBJ ( eat, apple, _) , MOD(fork,eat,with), XCOMP(without,eat,ask)</doubt><p>The results of the evaluation on a corpus anno­tated with the correct grammatical relations (Car­roll et al., 1998) show significantly higher values of recall and precision for Conexor FDG with re­spect to Link Grammar (Table 2).</p><p>The grammatical relations can be used by the scoring module of our QA system. We only need to compute the overlap of grammatical re­lations between the question and the answer can­didate. In theory, we must use the hierarchical organisation of the grammatical relations to de­cide if two grammatical relations unify. For ex­ample, SUBJ (eat, man, _) should unify with lllll the    man that came ate bananas and apples with a fork<page local="4" global="113"/></p><table caption="Table 1: Grammatical relations used in this paper." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Relation</i></p></td><td class="cell"><p><i>Description</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CONJ(type,head+)</p></td><td class="cell"><p>Conjunction</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MOD(type,head,dependent)</p></td><td class="cell"><p>Modifier</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CMOD(type,head,dependent)</p></td><td class="cell"><p>Clausal modifier</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NCMOD(type,head,dependent)</p></td><td class="cell"><p>Non-clausal modifier</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>DETMOD(type,head,dependent)</p></td><td class="cell"><p>Determiner</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SUB J(head,dependent,initial _gr)</p></td><td class="cell"><p>Subject</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>OBJ(head,dependent,initial_gr)</p></td><td class="cell"><p>Object</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>DOBJ(head,dependent,initial_gr)</p></td><td class="cell"><p>Direct object</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>XCOMP(head,dependent)</p></td><td class="cell"><p>Clausal complement without an overt subject</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 2: Intrinsic evaluations of Link Grammar and Conexor FDG." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><i>Link Gram-</i></p></td><td class="cell"><p><i>Conexor</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><i>mar</i></p></td><td class="cell"><p><i>FDG</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Precision</i></p></td><td class="cell"><p>SUBJ</p></td><td class="cell"><p>50.3%</p></td><td class="cell"><p>73.6%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>OBJ</p></td><td class="cell"><p>48.5%</p></td><td class="cell"><p>84.8%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>XCOMP</p></td><td class="cell"><p>62.2%</p></td><td class="cell"><p>76.2%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>MOD</p></td><td class="cell"><p>57.2%</p></td><td class="cell"><p>63.7%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b><i>Average</i></b></p></td><td class="cell"><p><b><i>54.6%</i></b></p></td><td class="cell"><p><b><i>74.6%</i></b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Recall</i></p></td><td class="cell"><p>SUBJ</p></td><td class="cell"><p>39.1%</p></td><td class="cell"><p>64.5%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>OBJ</p></td><td class="cell"><p>50%</p></td><td class="cell"><p>53.4%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>XCOMP</p></td><td class="cell"><p>32.1%</p></td><td class="cell"><p>64.7%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>MOD</p></td><td class="cell"><p>53.7%</p></td><td class="cell"><p>56.2%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b><i>Average</i></b></p></td><td class="cell"><p><b><i>43.7%</i></b></p></td><td class="cell"><p><b><i>59.7%</i></b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><doubt alpha="57.1" length="7" tooSmall="False" monospace="0.0">main &lt;-</doubt><doubt alpha="42.9" length="7" tooSmall="False" monospace="0.0">^.&gt;det^</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">r</doubt><doubt alpha="61.1" length="18" tooSmall="False" monospace="0.0">-&gt; subj mod&lt; &gt;subj</doubt><doubt alpha="50.0" length="6" tooSmall="False" monospace="0.0">ins &lt;-</doubt><doubt alpha="40.0" length="5" tooSmall="False" monospace="0.0">&gt;--cc</doubt><doubt alpha="41.7" length="12" tooSmall="False" monospace="0.0">obj&lt;^^-cc&lt;-^</doubt><doubt alpha="64.7" length="17" tooSmall="False" monospace="0.0">y'pcomp&lt;-^ &gt;detNJ</doubt><figure caption="Figure 3: Dependency structure of a sample sentence."></figure><p>SUB_OR_DOBJ (eat, man). However, since the same parser was used for both the question and the answer, the granularity of grammatical rela­tions will be practically the same. Thus, each grammatical relation can be seen as an unstruc­tured token and the scoring module can simply count the number of common tokens, very much like counting the overlap of words. This was the approach used in our QA prototype.</p></section><section number="3" title="Flat Logical Forms"><p>Flat logical forms have been used in several NLP systems including question-answering sys­tems (Harabagiu et al., 2001; Lin, 2001; Molla et al., 2000, for example). The flat logical forms that we use in our QA system are borrowed from (Molla et al., 2000), who uses reification to flat­ten out nested expressions. For example, the flat logical form of <i>The cp command will quickly copy files </i>is:<footnote anchor="1"/></p><doubt alpha="52.1" length="146" tooSmall="False" monospace="0.0">object('cp',o2,[x2]), object('command',o3,[x3]),compoundJioun(x2 , x3 ) , propCquickly' ,p5,[e6]) , evt('copy',e6,[x3,x7]), object('file',o7,[x7])</doubt><p>Flat logical forms express the main predicate-argument dependencies between the entities in­troduced by the sentence in a form that is suit­able for computing semantic similarity. In par­ticular, we only need to find the common predi­cates between the question and the answer candi­date. The only additional complexity is the han­dling of variables in the terms of the question. It is therefore necessary to instantiate the question variables with constants found in the answer can­didate. For example, the logical form of <i>Which</i> <i>command copies files?</i><i> </i>is (the symbols in upper­case indicate variables):</p><footnote label="1">For illustration purposes, the logical forms used in this paper are slightly different from the ones shown in the liter­ature.</footnote><doubt alpha="50.7" length="75" tooSmall="False" monospace="0.0">objectCcommand' ,O1,[X1]) , evt('copy',E2, [X1,X2]), object('file',O2,[X2])</doubt><p>If this logical form is to match that of the sen­tence <i>The cp command will quickly copy files </i>above, the scoring module needs to instantiate the variable O1 in the question with the constant o3 in the answer candidate, X1 with x3 , and so on. In our implementation we have used Prolog unifi­cation.</p><p>Since there are several plausible combinations of variable instantiations, the scoring module finds the set of instantiations that provides the highest overlap of logical forms.</p><p>Table 3 shows the flat logical forms of ques­tions that differ solely in the argument positions, the flat logical form of an answer candidate, and the resulting overlaps.</p></section><section number="4" title="The Question Answering Framework"><p>In contrast with (Molla et al., 2000), the seman­tic interpreters used in our evaluations to compute the logical forms do not use any additional lexical, domain, or world knowledge. Furthermore, there is no disambiguation step and there is no anaphora resolution module. The resulting semantic inter­preters may therefore be less accurate, but the re­sulting QA systems are in a better position to be compared with the QA systems based on gram­matical relations described in Section 2. Once it is decided which methodology is better, it is con­ceivable to add the additional modules that further enhance the expressivity of the sentence image.</p><p>For the present evaluation we used the Reme­dia Publications Reading Comprehension corpus used by DeepRead (Hirschman et al., 1999). The corpus is aimed at testing the degree of reading comprehension by children, and the documents in this corpus are classified into several levels of reading proficiency.<page local="5" global="114"/> There are about 30 docu­ments for each of the levels 2, 3, 4, and 5, with a total of 117 documents. Each document includes a short piece of text and five questions (who-, <i>what-, when-, where-, </i>and <i>why-) </i>about the text.</p><doubt alpha="92.9" length="14" tooSmall="True" monospace="0.0">Sentence image</doubt><p>The corpus contains annotations of the coreffer-ence chains and the named entities. The answers are also marked-up in the text and a gold standard for every answer is available. These annotations make the corpus suitable for the development and test of QA systems.</p><p>To evaluate the impact of the parsers and the scoring modules we have developed a simple question answering system framework (Figure 4). Given that every document contains the questions that are to be asked about the document, our QA system does not need to include a preselection stage. Instead, every document is processed inde­pendently. The system has a pre-processing stage that segments the document into text and ques­tions. The text is split into sentences and each sentence is analysed independently.</p><p>Every question in the document is analysed to produce the question image. The question is also classified into one of the <i>who-, what-, when-, where-, </i>and <i>why- </i>categories. Depending on the question category, the question analysis module determines and returns the likely named entities of the expected answers.</p><p>Table 4 shows the named entities associated with each question type. The question classifier is extremely simple due to the fact that there are always five questions, and they have a very sim­ple pattern. Thus, the regular expressions shown in Table 4 suffice to identify the question types. The procedure to determine the expected answer type is therefore very simple but fairly effective for the corpus and the named-entity types used in the named-entity annotations. Since the focus of this work is on the comparison of the scoring modules we did not feel we needed to produce a more sophisticated question analysis module.</p><p>The named-entity annotations provided with the evaluation corpus relieves the system from us­ing a named-entity extraction component. The re­sulting system performs perfect named-entity ex­traction, which may artificially enhance the final quality of the answers returned. The results there­fore cannot be compared with the results given by QA systems such as those participating in the Question Answering track of TREC (Voorhees, 2001), but they are good for the purposes of com­parison that we pursue here.</p><p>The answer extraction module penalises all sentences that have no entities compatible with the answer type by giving them an initial score of -100. The only part of the QA system that varies across the comparisons is the scoring component.</p><table caption="Table 3: Question answering using flat logical forms. Overlap shown in bold." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Answer candidate</b></p></td><td class="cell"><p><b>Flat Logical Form</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>John saw Mary</i></p></td><td class="cell"><p>object('john',ol,[xl]), object('mary',o3,[x3]), evt('see',e2,[xl,x3])</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Question</b></p></td><td class="cell"><p><b>Flat Logical Form</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Did John see Mary? Did Mary see John?</i></p></td><td class="cell"><p><b>object('mary',0,[X]), evt('see',E,[Y,X]), object('john',02,[Y])</b></p><p>object('john',0<b>,[X]), evt('see',E,[Y,X]), </b>object('mary',02,[Y])</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 4: Question types and expected answer types." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Regex</i></p></td><td class="cell"><p><i>Expected Answer Type</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>"Who</p></td><td class="cell"><p>person, organization</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>"What</p></td><td class="cell"><p>any</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>"When</p></td><td class="cell"><p>date, time</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>"Where</p></td><td class="cell"><p>location</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>"Why</p></td><td class="cell"><p>any</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Figure 4: Architecture of the question-answering system." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Question</b></p></td><td class="cell"><p><b>Expected answer type</b></p></td><td class="cell"><p>F <b>iltering</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Analysis</b></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>Question image</b></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p><b>Scoring</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="6" global="115"/><p>The scoring methods described above can there­fore be compared free of interference from other modules.</p><p>Following the specifications of the QA track of TREC8 to TREC10, the system returns the five answer candidates with highest scores, ranked by scores in descending order.</p></section><section number="5" title="Evaluation Methodology"><p>All four combinations of parser (Link Grammar and Conexor FDG) and scoring module (gram­matical relations and flat logical forms) were used in the evaluations. All other modules in the QA system were left untouched.</p><p>The evaluation of the quality of the answers was done automatically. An answer candidate is judged correct if more than 80% of its words ap­pear in one of the correct answers provided by the corpus annotations. We have not evaluated the ac­curacy of this evaluation method, but we have no reasons to believe that the final conclusions of the comparison are significantly affected by this au­tomatic evaluation procedure, since all the exper­iments were evaluated with the same procedure.</p><p>The final measure is TREC QA's Mean Recip­rocal Rank (MRR). Thus, for a given question, if the first correct answer returned is in rank • , the question is scored as • •• , or • if no correct an­swer is found. The final score of the system is the mean of scores for the individual questions.</p><p>An initial inspection of the first results revealed that the overall scores were very low. As a result, several answer candidates would receive the same score. Table 5 shows an example of the answers returned for a question. We can observe that all five answer candidates were given the same score, but only the last candidate was correct. There is no way for the system to know which of the sentences with same score is best. To determine the impact of returning several sentences with the same score, two indices were computed in addi­tion to the MRR. The two indices correspond to the MRR that would result if the correct an­swer was chosen first or last among those of the same score. These indices represent the "best" and "worst" case, respectively, and ideally they would be almost equal.</p><doubt alpha="0.0" length="8" tooSmall="False" monospace="0.0">0.5 0.45</doubt><doubt alpha="0.0" length="8" tooSmall="False" monospace="0.0">0.4 0.35</doubt><doubt alpha="0.0" length="8" tooSmall="False" monospace="0.0">0.3 0.25</doubt><doubt alpha="0.0" length="8" tooSmall="False" monospace="0.0">0.2 0.15</doubt><doubt alpha="0.0" length="8" tooSmall="False" monospace="0.0">0.1 0.05</doubt><figure caption="Figure 5: Evaluation ofscoring measures."></figure></section><section number="6" title="Results and Discussion"><p>The vertical bars in Figure 5 show the variation between the best and worst MRR for the text of levels 3, 4, and 5 together. Five cases are displayed, corresponding with a baseline scoring based on stem overlap and all combinations of parser (either Conexor FDG or Link Grammar) and scoring measure (either overlap ofgrammati-cal relations or overlap of flat logical forms).</p><p>Overall, we can see that the flat logical forms give better results than the grammatical relations. This is not surprising, since the flat logical forms were designed for tasks that require the semantic comparison of sentences. In contrast, the gram­matical relations were designed for the compari­son of parsers.</p><p>Also, Conexor FDG produces better re­sults than Link Grammar. These results con­firm Molla and Hutchinson (2003)'s findings that Conexor FDG is slightly better than Link Gram­mar in a similar QA system that is based on a different corpus and evaluation methodology. Furthermore, the increase of performance with Conexor FDG is only marginal. This is in contrast with a direct comparison between Link Gram­mar and Conexor FDG which, as shown in Ta­ble 2 above, rated Conexor FDG much better than Link Grammar. The current experiment shows that, regardless of the method used (grammati­cal relations or flat logical forms), the MRR of the QA system that uses Conexor FDG is only slightly higher than the MRR of the QA sys­tem that uses Link Grammar. Thus, our results present further evidence that intrinsic evaluations<page local="7" global="116"/></p><figure caption="Figure 6: Combination ofoverlap measures."></figure><p>In the above formulas, • stands for word form overlap, • • is the overlap of dependencies, • • is the overlap of minimal logical forms, and •• • is the overlap of grammatical relations.</p><p>The dependencies used were extracted from the Conexor FDG parser, which is dependency-based. All the experiments about the combination of scoring systems were used with Conexor FDG. Similar results are expected with Link Grammar or other parsers. We are experimenting with the impact of other possible scoring combinations.</p><p>are of very limited value, as stated already by Galliers and Sparck Jones (1993). An extrinsic evaluation that shows the impact of the modules to evaluate within the context of an application (QA in this case) may give results that differ sub­stantially from those of an intrinsic evaluation.</p><p>What is surprising is the fact that, as Figure 5 shows, a measure based on simple stem overlap gives better results. Subsequent experiments in­dicate that a combination of the above scoring systems plus overlaps of word dependencies and word forms produce better results and they have a narrower difference between best and worst cases. For example, Figure 6 shows the results of two possible combinations:</p><p><b>Weight on grammatical relations:</b></p><doubt alpha="0.0" length="24" tooSmall="False" monospace="0.0">• •••• • ••    •••••9•••</doubt><p><b>Weight on flat logical forms:</b></p><doubt alpha="0.0" length="24" tooSmall="False" monospace="0.0">• •••• • ••    ••••9••••</doubt></section><section number="7" title="Conclusions and Further Research"><p>This research shows that the combined informa­tion on word dependencies, grammatical relations and flat logical forms improves the accuracy of the system with respect to the individual mea­sures, though the additional resources required to extract syntactic and semantic information may not justify the use of these measures against sim­ple word overlap.</p><p>Further research is necessary to determine the reason for this. For example, it may be that the very nature of the Reading Comprehension docu­ment set makes it unlikely to represent real-world text. The text is intentionally simple, the docu­ments are short and there is little text redundancy. Furthermore, the fact that the texts are of varied topics and that every document contains the ques­tions that apply to the document makes it highly unlikely that the questions associated to a specific document have eligible answer candidates outside the document. For all these reasons we are per­forming similar experiments with the corpus used in the QA track of TREC 10 and TREC 11. However, preliminary results support the results pre­sented in this paper.<page local="8" global="117"/></p><table caption="Table 5: Answers returned for the questionWhat does the Sears ad offer?" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Rank</i></p></td><td class="cell"><p><i>Sentence</i></p></td><td class="cell"><p><i>Score</i></p></td><td class="cell"><p><i>Overlap</i></p></td><td class="cell"><p><i>Correct</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p>1989   Remédia Publications,</p></td><td class="cell"><p>1</p></td><td class="cell"><p>compoundjioun (<b>vjx</b>2 , <b>vjx</b>3 )</p></td><td class="cell"><p>no</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Comprehension s-4</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2</p></td><td class="cell"><p>(North Redwood, Minn</p></td><td class="cell"><p>1</p></td><td class="cell"><p>compound Jioun ( <b>vjx</b>2 , <b>vjx</b>3 )</p></td><td class="cell"><p>no</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3</p></td><td class="cell"><p>Not long ago, he ran an ad in</p></td><td class="cell"><p>1</p></td><td class="cell"><p>object (ad, v_o7, [<b>vjx</b>7] )</p></td><td class="cell"><p>no</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>some newspapers In small towns</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>4</p></td><td class="cell"><p>The ad showed a drawing of</p></td><td class="cell"><p>1</p></td><td class="cell"><p>obj ect (ad, v_o2 , [v_x2] )</p></td><td class="cell"><p>no</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>lovely furniture</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>5</p></td><td class="cell"><p>The ad said the furniture was for</p></td><td class="cell"><p>1</p></td><td class="cell"><p>obj ect (ad, v_o2 , [v_x2] )</p></td><td class="cell"><p>yes</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>sale</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>It may well be that one needs to compute more complex overlap measures to leverage the addi­tional information. Additional further research includes the evaluation of weighted overlap mea­sures that consider the relative importance of spe­cific grammatical relations or logical form terms, and the determination of the optimal weights to be given.</p><p>Finally, perhaps the simple questions used in the Reading Comprehension corpus and in TREC do not require the use of much linguistic informa­tion. An evaluation framework with more com­plex questions is necessary to test this possibility.</p><p>We also plan to evaluate the impact of other NLP modules such as anaphora resolvers and dis-ambiguators, and the use of lexical resources (e.g. WordNet or localisations of WordNet) and do­main and world knowledge. The current QA sys­tem can be easily expanded to allow for all of these evaluations.</p></section><references><p>Eric Brill, Jimmy Lin, Michèle Banko, Susan Dumais, and Andrew Ng. 2001. Data-intensive question an­swering. In Voorhees and Harman (Voorhees and</p><p>Harman, 2001).</p><p>Ted Briscoe and John Carroll. 2000. Gram­matical relation annotation.   On-line document.</p><p>http://www.cogs.susx.ac.uk/lab/nlp/ carroll/grdescription/index.html.</p><p>John Carroll, Ted Briscoe, and Antonio Sanfilippo. 1998. Parser evaluation: a survey and a new pro­posal. In <i>Proc. LREC98.</i></p><doubt alpha="59.2" length="49" tooSmall="False" monospace="0.0">C.L.A. Clarke, G.V. Cormack, T.R. Lynam, C.M. Li,</doubt><p>and G.L. McLearn. 2001. Web reinforced question answering. In Voorhees and Harman (Voorhees and</p><p>Richard J. Cooper and Stefan M. Rüger. 2000. A sim­ple question answering system. In Voorhees and Harman (Voorhees and Harman, 2000).</p><p>David Elworthy. 2000. Question answering using a large NLP system. In Voorhees and Harman (Voorhees and Harman, 2000).</p><p>Julia R. Galliers and Karen Sparck Jones. 1993. Eval­uating natural language processing systems. Tech­nical Report TR-291, Computer Laboratory, Uni­versity of Cambridge.</p><p>Sanda Harabagiu, Dan Moldovan, Marius Pasca, Mi-hai Surdeanu, Rada Mihalcea, Roxana Girju, Vasile Rus, Finley Lacatuçu, and Razvan Bunescu. 2001. Answering complex, list and context questions with LCC's question-answering server. In Voorhees and Harman (Voorhees and Harman, 2001).</p><p>Lynette Hirschman and Rob Gaizauskas. 2001. Nat­ural language question answering: The view from here.   <i>Natural Language Engineering, </i>7(4):275300.</p><p>Lynette Hirschman, Marc Light, Eric Breck, and John D. Burger. 1999. Deep Read: A reading com­prehension system. In <i>Proc. ACL '99. </i>University of Maryland.</p><p>Jimmy J. Lin. 2001. Indexing and retrieving natural language using ternary expressions. Master's the­sis, MIT.</p><p>IgorMel'cuk. 1988. <i>Dependency Syntax: Theory and Practice. </i>State University of New York Press.</p><p>Diego Molla and Ben Hutchinson. 2003. Intrin­sic versus extrinsic evaluations of parsing systems.</p><p>In <i>Proc. European Association for Computational Linguistics (EACL), workshop on Evaluation Ini­tiatives in Natural Language Processing, </i>pages 43­50, Budapest, April. Association for Computational Linguistics, ACL.</p><p>Diego Molla, Rolf Schwitter, Michael Hess, and Rachel Fournier. 2000. Extrans, an answer extrac­tion system. <i>Traitment Automatique des Langues, </i>41(2):495-522.</p><p>Daniel D. Sleator and Davy Temperley. 1993. Parsing English with a link grammar. In <i>Proc. Third Inter­national Workshop on Parsing Technologies, </i>pages 277-292.</p><p>M. M. Soubbotin. 2001. Patterns of potential answer expression as clues to the right answers. In Voorhees and Harman (Voorhees and Harman, 2001).</p><p>Pasi Tapanainen and Timo Jarvinen. 1997. A non-projective dependency parser. In <i>Procs. ANLP-97.</i></p><p>ACL.</p><p>Ellen M. Voorhees and Donna K. Harman, editors.</p><p>2000. <i>The Ninth Text REtrieval Conference (TREC-</i><i>9) , </i>number 500-249 in NIST Special Publication.</p><p>NIST.</p><p>2001. <i>The Tenth Text REtrieval Conference (TREC-</i> <i>10) , </i>number 500-250 in NIST Special Publication.</p><p>Ellen M. Voorhees. 2001. The TREC question an­swering track. <i>Natural Language Engineering, </i>7(4):361-378.</p></references></body></article>