<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="76"/><title>Automatic Acquisition of Lexico-semantic Knowledge for QA</title><author surname="Plas" givenname="Lonneke van der"><org  name="University of Groningen" country="The Netherlands" city="Groningen"/></author><author surname="Bouma" givenname="Gosse"><org  name="University of Groningen" country="The Netherlands" city="Groningen"/></author></firstpageheader><frontmatter><p><b>Automatic Acquisition of Lexico-semantic Knowledge for QA</b></p><p><b>Lonneke van der Pias</b></p><p>Information Science University of Groningen Postbus 716, 9700 AS Groningen vdplas@let.rug.ni</p><p><b>Gosse Bouma</b></p><p>Information Science University of Groningen Postbus 716, 9700 AS Groningen gosse@let.rug.ni</p></frontmatter><abstract>We present an experiment for finding semantically similar words on the ba­sis of a parsed corpus of Dutch text and show that the acquired informa­tion correlates with relations found in Dutch EuroWordNet. Next, we demon­strate how the acquired knowledge can be used to boost the performance of an open-domain question answering sys­tem for Dutch. Automatically acquired lexico-semantic information is used to improve the recall of a method for ex­tracting function relations (such as <i>Wim Kok is the prime minister of the Nether­lands) </i>from corpora, and to improve the precision of our QA system on general wh-questions and definition questions. </abstract></header><body><section number="1" title="Introduction"><p>Lexico-semantic knowledge is increasingly im­portant in NLP, especially for applications such as Word Sense Disambiguation, Information Extrac­tion and Question Answering (QA). Although the coverage of handmade resources such as Wordnet (Fellbaum, 1998) in general is impressive, cov­erage problems remain for applications involv­ing specific domains or involving languages other than English.</p><p>We are interested in using lexico-semantic knowledge in an open-domain question answer­ing system for Dutch. Obtaining such knowledge from existing resources is possible, but only to a certain extent. The most important resource for our research is the Dutch part of EuroWordNet (Vossen, 1998), but its size is only half of that of the English WordNet. Many of the lexical items used in the CLEF QA<footnote anchor="1"/> corpora for Dutch, for instance, cannot be found in EuroWordNet. In addition, information about the classes to which named entities belong (i.e. <i>Narvik </i>isa <i>harbour) </i>has been shown to be useful for QA. However, such information is typically absent from hand-built resources. For these reasons, we are inter­ested in methods for acquiring lexico-semantic knowledge automatically from text corpora.</p><p>The remainder of the paper is organized as fol­lows. In the next section we briefly describe the question types for which we want to use lexico-semantic knowledge and in section 3, we describe related work. In section 4 we describe our ap­proach to finding distributionally similar words. Sections 5 and 6 decribe how the acquired knowl­edge is used for improving the performance of our QA system on specific question types, i.e. ques­tions asking for the name of persons which have a specific function in an organization (i.e. <i>Who is the secretary general of the UN?), </i>general wh-questions, and definition questions. We report the results of an evaluation on CLEF 2005 data in sec­tion 7. Section 8 contains our conclusions and suggestions for future research.</p></section><section number="2" title="Lexico-semantic Knowledge for QA"><p>We will now briefly describe the three question types whose performance we hope to improve us­ing automatically acquired lexical knowledge.</p><doubt alpha="65.2" length="23" tooSmall="False" monospace="0.0">'http://clef-qa.itc.it/</doubt><page local="2" global="77"/><p>Often questions are asked about the function of a particular person:</p><p>Who is the chair of Unilever?</p><p>Off-line methods (Fleischman et al, 2003) can be used to improve the performance of the system on such questions. In off-line QA plausible answers to highly likely questions are extracted before the actual question has been asked.</p><p>Bouma et al. (2005a) describe how syntactic patterns are used to extract answers for fre­quently occurring question types. The fol­lowing syntactic pattern could serve to extract <i>(Person,Role,Organization)-tuples </i>from the cor­pus:</p><doubt alpha="85.7" length="7" tooSmall="False" monospace="0.0">app mod</doubt><doubt alpha="64.3" length="28" tooSmall="False" monospace="0.0">name(PER) &lt;-noun-&gt; name(ORG)</doubt><p>Here, the <i>name(PER) </i>constituent provides the <i>Person </i>argument of the relation, the <i>noun </i>pro­vides the role, and the «ame(Ö^G)-constituent provides the name of the <i>Organization. </i>An im­portant source of noise in applying this pattern to the parsed corpus are cases where the <i>noun </i>is not indicating a role or a function:</p><p>colleague Henk ten Cate of Go Ahead</p><p>Here, the noun <i>colleague </i>does not represent a role within the organization <i>Go Ahead.</i></p><p>To remedy this problem, we collected a list of nouns denoting functions or roles from Dutch EWN, and restricted the search pattern to nouns occurring in this list:</p><p><i>name(PER) &lt;-function-&gt; name(ORG)</i></p><p>While this helps to improve precision, it also hurts recall, as many valid function words present in the corpus are not present in EWN. In section 5, we will report on an experiment where we expanded the list of function words extracted from EWN semi-automatically with distributionally similar words found in the corpus.</p><p>A second question type where the use of lexical knowledge is potentially useful are general wh-questions such as:</p><p>Which vulcano errupted in june 1991 ?</p><p>A QA system may find various named entities (such as <i>Fillipines </i>and <i>Pinatubo) </i>as potential an­swers to the question. Knowing that <i>Pinatubo </i>isa <i>vulcano </i>can help to identify the correct answer. Information about named entities is typically ab­sent in hand-made lexical resources. In section 6, we describe a method for acquiring such informa­tion automatically from a parsed corpus.</p><p>A final question type where lexical knowledge is useful are definition questions:</p><p>Who is Javier Solana?</p><p>For CLEF 2005, definition questions were re­stricted to persons and organizations and answers should provide "some fundamental information" to users who know nothing about the named en­tity. Determining which information should be used to provide an answer to such questions in general is hard. We tried an approach were we used automatically acquired isa-labels for named entities to find an appropriate category which needs to be included in the answer. In section 6, we describe how this information can be used to find answers to definition questions.</p></section><section number="3" title="Related Work"><p>Syntactic relations have been shown to provide information which can be used to acquire clus­ters of semantically similar words automatically (Lin, 1998a). As we have a fully parsed ver­sion of the Dutch CLEF QA corpus (78 mil­lion words, 4.1 million sentences) at our dis­posal, we were interested in applying this method to Dutch. In particular, we followed the strat­egy of Curran and Moens (2002) which evaluates various similarity measures and weight functions against various thesauri (MacQuarie (Bernard, 1990), Moby (Ward, 1996) and Roget (Roget, 1911)). We implemented most of the best per­forming similarity measures and weights accord­ing to the evaluation of Curran and Moens (2002) and evaluated their performance against Dutch EuroWordNet. Some results are given in sec­tion 4.</p><p>Automatically acquired clusters of seman­tically similar words can be used to ex­tend or enrich existing ontological resources. Alfonseca and Manandhar (2002), for instance, describe a method for expanding WordNet auto­matically.<page local="3" global="78"/> New concepts are placed in the Word-Net hierarchy according to their distributional similarity to words that are already in the hierar­chy. Their algorithm performs a top-down search and stops at the synset that is most similar to the new concept. In section 5, we are using a similar technique to expand the class of function words obtained from EuroWordNet.</p><p>Pasca(2004) and Pantel and Ravichandran (2004) present methods for acquiring class la­bels for instances (categorised named entities) from unstructured text. Pasca (2004) applies lexico-syntactic extraction patterns based on Part-of-Speech tags. Patterns were hand-built ini­tially, and extended automatically by scanning the corpus for the pairs of named entities and classes found with the initial patterns. Pat­terns which occur frequently in matching sen­tences can be added as additional extraction patterns. Pasca (2004) applies this information to websearch for example for processing list-type queries. For example, <i>SAS, SPSS, Minitab </i>and <i>BMDP </i>are returned in addition to the top documents for the query <i>statistical packages. </i>Pantel and Ravichandran (2004) propose an algo­rithm that takes a list of semantic classes in the form of clusters of words as input. Labels for these clusters are found by looking at four lexico-syntactic relationships <i>apposition (ayatol­lah Khomeini), nominal subject {Khomeini is an ayatollah), such as {Ayatollahs such as Khome­ini), </i>and <i>like {Ayatollahs like Khomeini). </i>Apart from judging the quality of their results manually, they conducted two QA experiments: answering definition questions and performing QA informa­tion retrieval (IR). They show that both tasks ben­efit from the use of automatically acquired class labels.</p></section><section number="4" title="Extracting semantically similar words"><p>An increasingly popular method for acquiring se­mantically similar words is to extract distribution-ally similar words from large corpora. The under­lying assumption of this approach is that seman­tically similar words are used in similar contexts. The context of a word <i>W </i>may be defined as the document in which <i>W </i>occurs or the <i>n </i>words sur­rounding <i>W </i>(n-grams, bag of words). Alternatively, the context may be defined syntactically. In that case, the words with which the target word is in a specific syntactic relation form the con­text of that word. Approaches which do not use syntax tend to find more associative relations be­tween words (i.e. between <i>patient </i>and <i>hospital), </i>whereas approaches using syntactic context tend to find concepts belonging to the same class (i.e. <i>doctor </i>and <i>surgeon). </i>As we are ultimately inter­ested in extending the coverage of a resource such as Dutch EuroWordNet, we focussed on the sec­ond approach.</p><p>Most research has been done using a lim­ited number of syntactic relations ((Lee, 1999), (Weeds, 2003)). However, (Lin, 1998a) shows that a system which uses a range of grammati­cal relations outperforms Hindle's (1990) results that were based on using information from just the subject and object relation. Apart from the subject and object relation we have used several other grammatical relations: adjective, coordina­tion, apposition and prepositional complement. Examples are given in table 1.</p><subsection number="4.1" title="Data collection"><p>As our data we used the Dutch CLEF QA cor­pus, which consists of 78 million words of Dutch newspaper text (Algemeen Dagblad and NRC Handelsblad 1994/1995). The corpus was parsed automatically using the Alpino parser (van der Beek et al., 2002; Malouf and van Noord, 2004). The result of parsing a sentence is a dependency graph according to the guidelines of the Corpus of Spoken Dutch (Moortgat et al, 2000).</p><p>From these dependency graphs, we extracted tuples consisting of the (non-pronominal) head of an NP (either a common noun or a proper name), the dependency relation, and either (1) the head of the dependency relation (for the object, subject, and apposition relation), (2) the head plus a preposition (for NPs occurring inside PPs which are prepositional complements), (3) the head of the dependent (for the adjective and apposition relation) or (4) the head of the other elements of a coordination (for the coordination relation). Examples are given in table 1. The number of tuples and the number of non-identical (Noun, Relation, OtherWord) triples (types) found are given in table 2.Note that<page local="4" global="79"/></p><p>Table 2. Number of tuples and non-identical de­pendency triples (types) extracted per dependency relation.</p><p>a single coordination can give rise to various dependency triples, as from a single coordination <i>likebier, wijn, en noten {beer, wine, and nuts) </i>we extract the triples <i>{bier, coord, wijn), {bier, coord, noten), {wijn, coord, bier), {wijn, coord, noten), {noten, coord, bier), </i>and <i>{noten, coord, wijn). </i>Similarly, from the apposition <i>premier Kok </i>we extract both <i>{premier, hd-app, Kok) </i>and <i>{Kok, app, premier).</i></p><p>For each noun that was seen at least 10 times in any dependency relation, we built a vector. Af­ter applying this cut-off, vectors are present for 83.479 nouns.</p></subsection><subsection number="4.2" title="Similarity measures and weights"><p>Various vector-based methods can be used to compute the distributional similarity between words. Curran and Moens (2002) report on a large-scale evaluation experiment, where they evaluated the performance of various commonly used methods. Van der Pias and Bouma (2005) present a similar experiment for Dutch, in which they tested most of the best performing measures according to Curran and Moens (2002). Point-wise Mutual Information (MI) and <i>Dice] </i>per­formed best in the experiments. We will now ex­plain this weight and similarity measure in further detail.</p><p>The information value of a cell in a word vector (which lists how often a word occurred in a specific grammatical relation to a specific word) is not equal for all cells. A large number of nouns can occur as the subject of the verb <i>hebben {have), </i>for instance, whereas only a few nouns may occur as the object of <i>uitpersen {squeeze). </i>Intuitively, the fact that two nouns both occur as subject of <i>hebben </i>tells us less about their semantic similarity than the fact that two nouns both occur as object of <i>uitpersen. </i>To account for this intuition, the fre­quency of occurrence in a vector can be replaced by a weighted score. The weighted score is an indication of the amount of information carried by that particular combination of a noun and its feature ( the grammatical relation, and the word heading the grammatical relation). For this ex­periment we used Pointwise Mutual Information (MI) (Church and Hanks, 1989).</p><doubt alpha="54.5" length="11" tooSmall="False" monospace="0.0">I(W,f)= log</doubt><doubt alpha="50.0" length="14" tooSmall="False" monospace="0.0">P{W,f)P{W)P{f)</doubt><p>To compute the similarity of two word vectors, we used a variant of the Dice-measure, which Curran and Moens (2002) refer to as <i>Dice]:</i></p><doubt alpha="80.0" length="5" tooSmall="False" monospace="0.0">Dice]</doubt><doubt alpha="50.0" length="22" tooSmall="False" monospace="0.0">2J2frmn(I(W1J),I(W2J))</doubt><doubt alpha="62.5" length="32" tooSmall="False" monospace="0.0">J2fI(Wi,f)+I(W2J)4.3 Performance</doubt><p>The Dutch version of the multilingual resource EuroWordNet (EWN) (Vossen, 1998) was used for evaluation. We randomly selected 1000 target words from Dutch EWN with a frequency of more than 10, according to the frequency information present in Dutch EWN. For each word we col­lected its 100 most similar words (nearest neigh­bours) according to the system under evaluation, and for each pair of words (target word + one of the most similar words) we calculated the seman­tic similarity according to Dutch EWN. A system scores well if the nearest neighbours found by the system also have a high semantic similarity ac­cording to EWN.</p><p>Lin (1998b) evaluates a number of measures for computing WordNet similarity. From the measures which are defined in terms of is-a re­lations only, the Wu and Palmer (1994) measure correlated best with human judgements. The<page local="5" global="80"/></p><table caption="Table 1. Types of dependency relations extracted." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>subject-verb</p></td><td class="cell"><p><i>cat.eat</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>verb-object</p></td><td class="cell"><p><i>feed-cat</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>adjective-noun</p></td><td class="cell"><p><i>black-cat</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>coordination</p></td><td class="cell"><p><i>cat-dog</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>apposition</p></td><td class="cell"><p><i>cat-Garfield</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>prep, complement</p></td><td class="cell"><p><i>go</i><i>+tO</i><i>-work</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>grammatical relation</p></td><td class="cell"><p>tuples</p></td><td class="cell"><p>types</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>subject</p></td><td class="cell"><p>5.639.140</p></td><td class="cell"><p>2.122.107</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>adjective</p></td><td class="cell"><p>3.262.403</p></td><td class="cell"><p>1.040.785</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>object</p></td><td class="cell"><p>2642.356</p></td><td class="cell"><p>993.913</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>coordination</p></td><td class="cell"><p>965.296</p></td><td class="cell"><p>2.465.098</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>prep, complement</p></td><td class="cell"><p>770.631</p></td><td class="cell"><p>389.139</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>apposition</p></td><td class="cell"><p>526.337</p></td><td class="cell"><p>602.970</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><doubt alpha="85.7" length="7" tooSmall="False" monospace="0.0">app mod</doubt><p>Table 3. Average EWN similarity at <i>k</i><i> </i>candidates when combining dependency relations based on Dice|+ MI.</p><p>Wu/Palmer measure for computing the semantic similarity between two words Wl and W2 in a word net, whose most-specific common ancestor is W3, is defined as follows:</p><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">Sim</doubt><doubt alpha="20.0" length="5" tooSmall="False" monospace="0.0">2(D3)</doubt><doubt alpha="26.7" length="15" tooSmall="False" monospace="0.0">Dl + D2 + 2(D3)</doubt><p>where, Dl (D2) is the distance from Wl (W2) to the lowest common ancestor of Wl and W2, W3. D3 is the distance of that ancestor to the root node.</p><p>Table 3 reports average EWN similarity for the 1, 5, 10, 20, 50, and 100 most similar words for the 1000 words in ours test set. If a word is am­biguous according to EWN (i.e. is a member of several synsets), the highest similarity score is used. The EWN similarity of a set of word pairs is defined as the average of the similarity be­tween the pairs. The baseline for this task is 0.26, which is the score obtained by picking 100 ran­dom words as nearest neighbours of a given target word, van der Pias and Bouma (2005) show that the system using data obtained from all syntactic relations outperforms systems using only a sub­set of the syntactic relations. Furthermore, they show that Dice|+ MI outperforms various other combinations of weight functions and similarity measures.</p></subsection></section><section number="5" title="Using automatically acquired role and function words"><p>In section 2, we explained that for QA we are in­terested in extracting, off-line, all instances of the following pattern in our corpus:</p><p><i>name(PER) *-function-&gt; name(ORG)</i></p><p>To obtain a list of words describing a role or func­tion, we extracted from Dutch EWN all words un­der the node <i>leider {leader) </i>(255 in total). The majority of hyperonyms of this node seemed to indicate function words we were interested in (i.e. it contained (the Dutch equivalents of) <i>king, queen,president, director, chair, </i>etc.), while other potential candidates (such as <i>beroep {profession) </i>seemed less suitable. However, the coverage of this list, when tested on a newspaper corpus, is far from complete. On the one hand, the list con­tains a fair amount of archaic items, while on the other hand, many functions that occur frequently in newspaper text are missing (i.e. Dutch equiv­alents of <i>banker, boss, national team coach, cap­tain, secretary-general, </i>etc.).</p><p>To improve recall, we extended the list of function words obtained from EWN semi-automatically with distributionally similar words. In particular, for each of the 255 words in the EWN list, we retrieved its 100 most distribution-ally similar words. We gave each retrieved word a score that corresponds to its reverse rank (1st word: 100, 2nd: 99, 3rd: 98 etc.). The overall score for a word was the sum of the scores it ob­tained for the individual key words. Thus, words that are semantically similar to several words in the original list will obtain a higher score than words that were returned only once or twice. Words that were present already in the EWN-list were filtered.</p><p>An informal evaluation of the result learned that many false positives in the expanded list were either named entities or nouns referring to groups of people <i>{board, committee, ,...). </i>The distinc­tion between groups and functions of individuals is hard to make on the basis of distributional data. For instance, both a <i>board </i>and a <i>director </i>can take decisions, report results, be criticized, etc. We tried to filter both proper names and groups auto­matically, by discarding noun stems that start with a capital, and noun stems which are listed under the node <i>groep {group) </i>in EWN.</p><p>Finally, we selected the top-1000 of the filtered list, and validated it manually. The list contained 644 valid role or function nouns, which are absent in EWN. A substantial number of the errors are nouns which refer to a group but which are not listed as such in EWN.</p><p>The 644 valid nouns were merged with the original EWN list, to form a list of 899 function or role nouns. Next, the off-line extraction pro­cess was executed using both the original EWN list and the expanded list.<page local="6" global="81"/> The effect on recall is illustrated in table 4. The number of extracted tuples increases with 125%, while the number of unique tuples increases with 181%. The effect of this increase on the performance of our QA sys­tem is described in section 7.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>hline <i>k=</i></p></td><td class="cell"><p>1</p></td><td class="cell"><p>5</p></td><td class="cell"><p><i>m </i><i>Su</i></p><p>10</p></td><td class="cell"><p>nilarit 20</p></td><td class="cell"><p>y at 50</p></td><td class="cell"><p>100</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>system</p></td><td class="cell"><p>.60</p></td><td class="cell"><p>.54</p></td><td class="cell"><p>.52</p></td><td class="cell"><p>.49</p></td><td class="cell"><p>.46</p></td><td class="cell"><p>.44</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><section number="6" title="Using automatically acquired instances"><p>Both Pasca (2004) and Pantel and Ravichandran (2004) describe methods for acquiring labels for named entities from large text corpora and evalu­ate the results in the context of web search and question answering. Pantel and Ravichandran (2004) use the apposition relation to find poten­tial labels for named entities. As we already had extracted all appositions from the CLEF corpus as part of the vector-based method for finding se-mantically similar words, we decided to use this information for two other QA tasks as well.</p><p>As can be seen in table 2, we extracted 602K apposition relations (30IK regardless of direc­tion), from a total of over 526K appositions tu­ples found in the corpus. This database con­tains, for instance, 112 appositions with names of <i>ferry boats {Estonia, Anna Maria Lauro, Sally Star </i>etc.) and no less than 2951 appositions with names of national team coaches <i>{Bobby Robson, Jack Charlton, Menotti, Berti Vogts </i>etc.). The class labels extracted for each named entity may contain a certain amount of noise. However, by focussing on the most frequent label for a named entity, most of the noise can be discarded. For instance, <i>Guus Hiddink </i>occurs 197 times in the extracted apposition tuples, 170 times as <i>bond­scoach {national team chef, </i>and not more than 5 times with various other labels <i>{coach, colleague, guest, newcomer,</i>...). Regarding the ambiguity of the classified named entities we can say that on average a named entity has 1.7 labels. The distri­bution is skewed: 80 % has only 1 label and for example the most ambiguous named entity, <i>the</i> <i>Netherlands, </i>has 515 labels in total.</p><p>We used the extracted class labels to improve the performance of our QA system on general WH-questions such as:</p><p>Which ferry sank southeast of the island Utö?</p><p>Question analysis and classification tells us that this is a question of type which ( ferry). Can­didate answers that are selected by our system are: <i>Tallinn, Estonia, Raimo Tiilikainen </i>etc. The QA system uses various strategies to rank poten­tial answers, i.e. the score assigned to the pas­sage by Information Retrieval(IR), the presence of named entities from the question in the sen­tence in which the answer is found, the syntactic similarity between question and answer sentence, the frequency of the answer in the set of poten­tial answers etc. Still, selecting the correct named entity for answers to general WH-questions poses considerable problems for our system.</p><p>To improve the performance of the system on these questions, we incorporated an additional strategy for selecting the correct answer. Poten­tial answers which have been assigned the class corresponding to the question stem (i.e. ferry in this case) are ranked higher than potential an­swers for which this class label cannot be found in the database of ISA-relations. Since <i>Estonia </i>is the only potential answer which ISA ferry, accord­ing to our database, this answer is selected. Note that in answering WH-questions we do not select only the most frequent label assigned to a named entity, but simply check whether the named en­tity occurs at least once with the appropriate class label.</p><p>A second question type where the acquired class labels are relevant are definition question. The CLEF 2005 QA test set contains no less than 60 questions of the form:</p><p>What is Sabena?</p><p>The named entity <i>Sabena </i>occurs frequently in the corpus, but often with class labels assigned to it, which are not suitable for inclusion in a def­inition <i>{possibility, partner, company,,.... </i>By fo­cussing on the most frequent class label assigned to a named entity <i>{airline company </i>in this case), a more appropriate label for a definition can be found.<page local="7" global="82"/> Frequency is important but often the class label by itself is not sufficient for an adequate def­inition. Therefore we expand the class label with modifiers which typically need to be included in a definition.</p><table caption="Table 4. Coverage of function table with (EWN+) and without (EWN) expansion." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>EWN</p></td><td class="cell"><p>EWN+</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>tuples</p></td><td class="cell"><p>unique</p></td><td class="cell"><p>tuples</p></td><td class="cell"><p>unique</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>34191</p></td><td class="cell"><p>16530</p></td><td class="cell"><p>77028</p></td><td class="cell"><p>46589</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>More in particular, our strategy for answering definition questions consisted of two phases:</p><p>• Phase 1 : The most frequent class found for a named entity is taken.</p><p>• Phase 2: The sentences which mention the named entity and the class are selected, and searched for additional information which might be relevant. Snippets of information that are in a adjectival relation or a preposi­tional complement to the class label are se­lected.</p><p>For the example above, our system produces <i>Belgian airline company </i>as answer.</p><p>However, deciding beforehand what informa­tion is relevant is not trivial. As explained we de­cided to only expand the label with adjectival and pp modifiers that are adjacent to the class label in the corresponding sentence. This is the reason for a number of answers being inexact. Given the constituent <i>the museum Hermitage in St Peters­burg, </i>this strategy fails to include <i>in St Peters­burg, </i>for instance. We did not include relative clause modifiers, as these tend to contain infor­mation which is not appropriate for a definition. However, for the question, <i>Who is is Iqbal Masih, </i>this leads the system to answer <i>twelve year old boy, </i>extracted from the constituent <i>twelve year old boy, who fought against child labour and was shot Sunday in hist home town Muritke. </i>Here, at least the first conjunct of the relative clause should have been included. Similarly, we did not include purpose clauses, which leads the system to respond <i>large scale American attempt </i>to the question <i>what was the Manhattan project, </i>instead of <i>large scale American attempt to develop the first (that is, before the Germans) atomic bomb.</i></p></section><section number="7" title="Evaluation"><p>We compared the performance of two versions of our QA system on the Dutch questions from CLEF 2005. As no official results for CLEF 2005 were known to us at the time of the experiment, <footnote anchor="2"/> answers were judged for correctness by our­selves and two additional project members. An­swers were judged correct if at least three of the four judges considered them correct. Note that in CLEF, systems must return only a single, exact, answer.</p><p>In table 5 the performance of the baseline and improved system is shown. In the first column the question type is given (question types not rel­evant for this paper are left out). In the second and fourth column the number of questions clas­sified as being of the corresponding question type is shown. In colums 3 and 5 the corresponding CLEF score is given.</p><p>The baseline of our QA system, was the Joost QA system, without a special question type for function questions, and without access to isa-relations. The baseline treats function questions as person questions, i.e. as questions which re­quire a named entity of type <i>person </i>as an an­swer. General wh-questions and definition ques­tions are answered by selecting the most highly ranked answer from the list of relevant paragraphs returned by the IR component. Answers to defi­nition questions are basically selected by means of the same strategy as described for the im­proved system above, except that answers must now be selected from the documents returned by IR, rather than from sentences known to contain a relevant class label.</p><p>The improved system makes use of the ques­tion type <i>function </i>and the related table in which information about functions is stored. Further­more it uses isa-relations in answering general WH questions and definition questions.</p><p>The overall effect of these additions is an im­provement in (estimated) CLEF score of 8% and an error reduction of 16%.</p><p>Adding a question class for functions, and a related table with (off-line extracted) answers to such questions has the effect that 19 person ques­tions and one general wh-question in the base­line system are now classified as function ques­tions. The effect on accuracy of this change seems small (as person questions are already answered relatively well), but is nevertheless positive. Of the 20 questions that are classified as function<page local="8" global="83"/></p><footnote label="2">see Bouma et al. (2005b) for official results</footnote><p>Table 5. Overall performance of the baseline and improved QA system on the CLEF 2005 Dutch QA test set.</p><p>questions in the improved system, 4 involve the question stems <i>weduwe (widdow), adviseur (advi­sor), secretaris-generaal (secretary-general) </i>and <i>vriendin (girl friend), </i>which were present in our extended list of function nouns only.</p><p>Adding isa-relations as an additional knowl­edge source for answering WH-questions im­proves the CLEF score of 36 WH-questions with 15 % and gives an error reduction of 22%. Us­ing the same information to provide answers to definition questions improves the CLEF score on 60 definition with almost 15%, which is an error reduction of 22%.</p></section><section number="8" title="Conclusions and future work"><p>We have demonstrated that lexico-semantic knowledge can be acquired from syntactically parsed corpora, and that the inclusion of such knowledge in a QA system has a positive effect on the overall performance of the QA system. Firstly, the use of off-line techniques in general has a positive effect on the accuracy of QA. Here, we have demonstrated that the resources required to do off-line extraction accurately can be ac­quired semi-automatically by expanding a given list of relevant function words. Secondly, the per­formance of the system on general WH-questions and definition questions was shown to improve considerably if it has access to automatically ac­quired class labels.</p><p>The research reported here can be extended in several ways. For instance, while we used a considerable number of grammatical relations for finding semantically similar words, we did not use predicative complements. Sentences contain­ing such a complement (i.e. <i>Garfield is a cat) </i>do seem to provide useful information for learning semantic similarity. In addition, this relation may be used to expand the number categorised named entities.</p><p>Alternative ways of exploiting the class labels in QA can be explored as well. Pantel and Ravichandran (2004), for instance, use class labels to index the document collection. I.e. every paragraph which mentions a named entity known to be a <i>ferry, </i>is labeled with this class as well. This strategy allows the IR component to make use of class information. Pantel and Ravichandran (2004) show that this improves the precision of IR considerably. In future work, we would like to explore this possibility as well.</p></section><section title="Acknowledgements"><p>This research was carried out as part of the re­search program for <i>Interactive Multimedia Infor­mation Extraction, </i>imix, financed by nwo, the Dutch Organisation for Scientific Research.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>questionJype</p></td><td class="cell"><p>baseline</p></td><td class="cell"><p>improved</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>#q</p></td><td class="cell"><p>score</p></td><td class="cell"><p>#q</p></td><td class="cell"><p>score</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>WH-questions</p></td><td class="cell"><p>36</p></td><td class="cell"><p>0.31</p></td><td class="cell"><p>35</p></td><td class="cell"><p>0.46</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>definition</p></td><td class="cell"><p>60</p></td><td class="cell"><p>0.53</p></td><td class="cell"><p>60</p></td><td class="cell"><p>0.68</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>person</p></td><td class="cell"><p>26</p></td><td class="cell"><p>0.69</p></td><td class="cell"><p>7</p></td><td class="cell"><p>0.71</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>function</p></td><td class="cell"><p>0</p></td><td class="cell"><p>0.00</p></td><td class="cell"><p>20</p></td><td class="cell"><p>0.75</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>total</p></td><td class="cell"><p><b>200</b></p></td><td class="cell"><p><b>0.49</b></p></td><td class="cell"><p><b>200</b></p></td><td class="cell"><p><b>0.57</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><references><p>E. Alfonseca and S. Manandhar. 2002. Improv­ing an ontology refinement method with hyponymy patterns. In <i>Proceedings of the international con­ference on Language Resources and Evaluation (LREC-2002).</i></p><p>J.R.L. Bernard. 1990. The Macquairie encyclopedic thesaurus. The Macquairie Library, Sydney, Aus­tralia.</p><p>Gosse Bouma, Jori Mur, and Gertjan van Noord. 2005a. Reasoning over dependency relations for QA. In <i>Proceedings of the IJCAI workshop on Knowledge and Reasoning for Answering Ques­tions (KRAQ), </i>pages 15-21, Edinburgh.</p><p>Gosse Bouma, Jori Mur, Gertjan van Noord, Lonneke van der Pias, and Jörg Tiedemann. 2005b. Question answering for Dutch using dependency relations. In <i>Proceedings of CLEF 2005. </i>To appear.</p><p>K.W. Church and P. Hanks. 1989. Word association norms, mutual information and lexicography. <i>Pro­ceedings of the 27th annual conference of the Asso­ciation of Computational Linguistics, </i>pages 76-82.</p><p>J.R. Curran and M. Moens. 2002. Improvements in automatic thesaurus extraction. In <i>Proceedings of the Workshop on Unsupervised Lexical Acquisition, </i>pages <i>59-67.</i></p><page local="9" global="84"/><p>C. Fellbaum. 1998. Wordnet, an electronic lexical database. MIT Press.</p><p>Michael Fleischman, Eduard Hovy, and Abdessamad Echihabi. 2003. Offline strategies for online ques­tion answering: Answering questions before they are asked. In <i>Proc. 41st Annual Meeting of the As­sociation for Computational Linguistics, </i>pages 1-7, Sapporo, Japan.</p><p>D. Hindle. 1990. Noun classification from predicate-argument structures. In <i>Proceedings of ACL-90, </i>pages 268—275.</p><p>L.Lee. 1999. Measures of distributional similarity. In <i>3 7th Annual Meeting of the Association for Compu­tational Linguistics, </i>pages 25-32.</p><p>D. Lin. 1998a. Automatic retrieval and clustering of similar words. In <i>COLING-ACL, </i>pages 768-774.</p><p>Dekang Lin. 1998b. An information-theoretic def­inition of similarity. In <i>Proc. 15th International Conf. on Machine Learning, </i>pages 296-304. Mor­gan Kaufmann, San Francisco, CA.</p><p>Robert Malouf and Gertjan van Noord. 2004. Wide coverage parsing with stochastic attribute value grammars. In <i>IJCNLP-04 Workshop Beyond Shal­low Analyses - Formalisms andstati stical modeling for deep analyses, </i>Hainan.</p><p>Michael Moortgat, Ineke Schuurman, and Ton van der Wouden. 2000. CGN syntactische annotatie. Inter­nal Project Report Corpus Gesproken Nederlands, see http://lands. let.kun.nl/cgn.</p><p>Patrick Pantel and Deepak Ravichandran. 2004. Automatically labeling semantic classes. In Daniel Marcu Susan Dumais and Salim Roukos, ed­itors, <i>HLT-NAACL 2004: Main Proceedings, </i>pages 321-328, Boston, Massachusetts, USA, May 2 -May 7. Association for Computational Linguistics.</p><p>M. Pasca. 2004. Acquisition of categorized named entities for web search. In <i>Proceedings of the Thir­teenth ACM conference on Information and knowl­edge management, </i>pages 137 - 145.</p><p>P. Roget. 1911. Thesaurus of English words and phrases.</p><p>Leonoor van der Beek, Gosse Bouma, and Gertjan van Noord. 2002. Een brede computationele grammat-ica voor het Nederlands. <i>Nederlandse Taalkunde, </i>7(4):353-374.</p><p>Lonneke van der Pias and Gosse Bouma. 2005. Syntactic contexts for finding semantically similar words. In <i>Proceedings of CLIN 2004, </i>Leiden Uni­versity. To Appear.</p><p>P. Vossen. 1998. Eurowordnet a multilingual database with lexical semantic networks.</p><p>G.Ward. 1996. Moby thesaurus. Moby Project.</p><p>J. Weeds. 2003. <i>Measures and Applications of Lexical Distributional Similarity. </i>Ph.D. thesis, University of Sussex.</p><p>Z. Wu and M. Palmer. 1994. Verb semantics and lex­ical selection. In <i>The 23rd Annual Meeting of the Association for Computational Linguistics, </i>pages 133-138.</p></references></body></article>