<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="543"/><title>Investigation of Question Classifier in Question Answering</title><author surname="Thint" givenname="Marcus"><org  name="British Telecom Group" city="England"/></author><author surname="Huang" givenname="Zhiheng"><org  name="Ghent University" country="Belgium" city="Ghent"/></author><author surname="Celikyilmaz" givenname="Asli"><org  name="University of California at Berkeley" country="USA" city="Berkeley"/></author></firstpageheader><frontmatter><p><b>Investigation of Question Classifier in Question Answering</b></p><p><b>Marcus Thint</b></p><p>Intelligent Systems Research Center British Telecom Group Chief Technology Office</p><p><b>Zhiheng Huang</b></p><p>EECS Department University of California</p><p>at Berkeley CA 94720-1776, USA</p><p>zhiheng@cs.berkeley.edu</p><p>marcus.2.thintdbt.com</p><p><b>Asli Celikyilmaz</b></p><p>asli@cs.berkeley.edu</p></frontmatter><abstract>In this paper, we investigate how an ac­curate question classifier contributes to a question answering system. We first present a Maximum Entropy (ME) based question classifier which makes use of head word features and their WordNet hy-pernyms. We show that our question clas­sifier can achieve the state of the art per­formance in the standard UIUC question dataset. We then investigate quantitatively the contribution of this question classifier to a feature driven question answering sys­tem. With our accurate question classifier and some standard question answer fea­tures, our question answering system per­forms close to the state of the art using TREC corpus. </abstract></header><body><section number="1" title="Introduction"><p>Question answering has drawn significant atten­tion from the last decade (Prager, 2006). It at­tempts to answer the question posed in natural language by providing the answer phrase rather than the whole documents. An important step in question answering (QA) is to classify the ques­tion to the anticipated type of the answer. For example, the question of <i>Who discovered x-rays </i>should be classified into the type of human (indi­vidual). This information would narrow down the search space to identify the correct answer string. In addition, this information can suggest different strategies to search and verify a candidate answer. In fact, the combination of question classification and the named entity recognition is a key approach in modern question answering systems (Voorhees and Dang, 2005).</p><p>The question classification is by no means triv­ial: Simply using question wh-words can not achieve satisfactory results.   The difficulty lies in classifying the <i>what </i>and <i>which </i>type questions. Considering the example <i>What is the capital of Yu­goslavia, </i>it is of location (city) type, while <i>What is the pH scale </i>is of definition type. As with the previous work of (Li and Roth, 2002; Li and Roth, 2006; Krishnan et al., 2005; Moschitti et al., 2007), we propose a feature driven statistical question classifier (Huang et al., 2008). In partic­ular, we propose head word feature and augment semantic features of such head words using Word-Net. In addition, Lesk's word sense disambigua­tion (WSD) algorithm is adapted and the depth of hypernym feature is optimized. With further aug­ment of other standard features such as unigrams, we can obtain accuracy of 89.0% using ME model for 50 fine classes over UIUC dataset.</p><p>In addition to building an accurate question classifier, we investigate the contribution of this question classifier to a feature driven question an­swering rank model. It is worth noting that, most of the features we used in question answering rank model, depend on the question type information. For instance, if a question is classified as a type of <i>sport, </i>we then only care about whether there are sport entities existing in the candidate sentences. It is expected that a fine grained named entity rec­ognizer (NER) should make good use of the accu­rate question type information. However, due to the lack of a fine grained NER tool at hand, we employ the Stanford NER package (Finkel et al., 2005) which identifies only four types of named entities. Even with such a coarse named entity recognizer, the experiments show that the question classifier plays an important role in determining the performance of a question answering system.</p><p>The rest of the paper is organized as follow­ing. Section 2 reviews the maximum entropy model which are used in both question classifica­tion and question answering ranking. Section 3 presents the features used in question classifica­tion. Section 4 presents the question classification</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">543</doubt><p><i>Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, </i>pages 543-550, Singapore, 6-7 August 2009. ©2009 ACL and AFNLP<page local="2" global="544"/></p><p>accuracy over UIUC question dataset. Section 5 presents the question answer features. Section 6 illustrates the results based on TREC question an­swer dataset. And Section 7 draws the conclusion.</p></section><section number="2" title="Maximum Entropy Models"><p>Maximum entropy (ME) models (Berger et al., 1996; Manning and Klein, 2003), also known as log-linear and exponential learning models, pro­vide a general purpose machine learning technique for classification and prediction which has been successfully applied to natural language process­ing including part of speech tagging, named entity recognition etc. Maximum entropy models can in­tegrate features from many heterogeneous infor­mation sources for classification. Each feature corresponds to a constraint on the model. Given a training set of <b>(C, </b><b><i>D), </i></b>where <b><i>C </i></b>is a set of class labels and <b><i>D </i></b>is a set of feature represented data points, the maximal entropy model attempts to maximize the log likelihood</p><p>Table 1: 6 coarse and 50 fine Question types de­fined in UIUC question dataset.</p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">E</doubt><doubt alpha="45.5" length="11" tooSmall="False" monospace="0.0">(c,d)e(c,D)</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">log</doubt><p>where <i>fi(c,d)</i><i> </i>are feature indicator functions. We use ME models for both question classification and question answer ranking. In question answer context, such function, for instance, could be the presence or absence of dictionary entities (as pre­sented in Section 5.2) associated with a particular class type (either <i>true or false, </i>indicating a sen­tence can or cannot answer the question). Aj are the parameters need to be estimated which reflects the importance of <i>fi(c,</i><i> </i><i>d)</i><i> </i>in prediction.</p></section><section number="3" title="Question Classification Features"><p>Li and Roth (2002) have developed a machine learning approach which uses the SNoW learning architecture. They have compiled the UIUC ques­tion classification dataset<footnote anchor="1"/> which consists of 5500 training and 500 test questions.<footnote anchor="2"/> All questions in the dataset have been manually labeled according to the coarse and fine grained categories as shown in Table 1, with coarse classes (in bold) followed by their fine classes.</p><p>The UIUC dataset has laid a platform for the follow-up research including (Hacioglu and Ward, 2003; Zhang and Lee, 2003; Li and Roth, 2006;</p><footnote label="1">Available at http://l  2r.cs.uiuc.edu/~cogcomp/Data/QA/QC . 2 Test questions are from TREC 10.</footnote><p>Krishnan et al., 2005; Moschitti et al., 2007). In contrast to Li and Roth (2006)'s approach which makes use of a very rich feature set, we propose to use a compact yet effective feature set. The fea­tures are briefly described as following. More de­tailed information can be found at (Huang et al., 2008).</p><p><b>Question wh-word </b>The wh-word feature is the question wh-word in given questions. For ex­ample, the wh-word of question <i>What is the population of China </i>is <i>what.</i></p><p><b>Head Word </b><i>head word </i>is defined as one <i>single </i>word specifying the object that the question seeks. For example the head word of <i>What is a group of turkeys called, </i>is <i>turkeys. </i>This is different to previous work including (Li and Roth, 2002; Krishnan et al., 2005) which has suggested a contiguous span of words <i>{a</i><i> group of turkeys </i>in this example). The single word definition effectively avoids the noisy information brought by non-head word of the span <i>{group </i>in this case). A syntac­tic parser (Petrov and Klein, 2007) and the Collins rules (Collins, 1999) are modified to extract such head words.</p><p><b>WordNet Hypernym </b>WordNet hypernyms are extracted for the head word of a given ques­tion. The classic Lesk algorithm (Lesk, 1986) is used to compute the most probable sense for a head word in the question context, and then the hypernyms are extracted based on that sense. The depth of hypernyms is set to</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">544</doubt><p><i>J2c' </i><b>exPEj Ai/i(c»</b><i>dY</i><page local="3" global="545"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(1)</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ABBR</b></p></td><td class="cell"><p>letter</p></td><td class="cell"><p>desc</p></td><td class="cell"><p><b>NUM</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>abb</p></td><td class="cell"><p>other</p></td><td class="cell"><p>manner</p></td><td class="cell"><p>code</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>exp</p></td><td class="cell"><p>plant</p></td><td class="cell"><p>reason</p></td><td class="cell"><p>count</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ENTITY</b></p></td><td class="cell"><p>product</p></td><td class="cell"><p><b>HUMAN</b></p></td><td class="cell"><p>date</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>animal</p></td><td class="cell"><p>religion</p></td><td class="cell"><p>group</p></td><td class="cell"><p>distance</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>body</p></td><td class="cell"><p>sport</p></td><td class="cell"><p>individual</p></td><td class="cell"><p>money</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>color</p></td><td class="cell"><p>substance</p></td><td class="cell"><p>title</p></td><td class="cell"><p>order</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>creative</p></td><td class="cell"><p>symbol</p></td><td class="cell"><p>desc</p></td><td class="cell"><p>other</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>currency</p></td><td class="cell"><p>technique</p></td><td class="cell"><p><b>LOC</b></p></td><td class="cell"><p>period</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>dis.med.</p></td><td class="cell"><p>term</p></td><td class="cell"><p>city</p></td><td class="cell"><p>percent</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>event</p></td><td class="cell"><p>vehicle</p></td><td class="cell"><p>country</p></td><td class="cell"><p>speed</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>food</p></td><td class="cell"><p>word</p></td><td class="cell"><p>mountain</p></td><td class="cell"><p>temp</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>instrument</p></td><td class="cell"><p><b>DESC</b></p></td><td class="cell"><p>other</p></td><td class="cell"><p>size</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>lang</p></td><td class="cell"><p>definition</p></td><td class="cell"><p>state</p></td><td class="cell"><p>weight</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>six with trial and error.<footnote anchor="3"/> Hypernyms features capture the general terms of extracted head word. For instance, the head word of ques­tion <i>What is the proper name for a female walrus </i>is extracted as <i>walrus </i>and its direct hypernyms such as <i>mammal </i>and <i>animal </i>are extracted as informative features to predict the correct question type of ENTY:animal.</p><p><b>Unigram words </b>Bag of words features. Such features provide useful question context in­formation.</p><p><b>Word shape </b>Five word shape features, namely all upper case, all lower case, mixed case, all digits, and other are used to serve as a coarse named entity recognizer.</p></section><section number="4" title="Question Classification Experiments"><p>We train a Maximum Entropy model using the UIUC 5500 training questions and test over the 500 test questions. Tables 2 shows the accuracy of 6 coarse class and 50 fine grained class, with fea­tures being fed incrementally. The question classi­fication performance is measured by accuracy, i.e., the proportion of the correctly classified questions among all test questions. The baseline using the</p><p>Table 2: Question classification accuracy using in­cremental feature sets for 6 and 50 classes over UIUC split._ wh-head word results in 46.0% and 46.8% respec­tively for 6 coarse and 50 fine class classification. The incremental use of head word boosts the accu­racy significantly to 92.2% and 82.0% for 6 and 50 classes. This reflects the informativeness of such feature. The inclusion of hypernym feature within 6 depths boosts 3.6% for 50 classes, while result­ing in slight loss for 6 coarse classes. The further use of unigram feature leads to 2.8% gain in 50 classes. Finally, the use of word shape leads to 0.6% accuracy increase for 50 classes. The best accuracies achieved are 93.6% and 89.0% for 6 and 50 classes respectively.</p><footnote label="3">We performed 10 cross validation experiment over train­ing data and tried various depths of 1, 3, 6, 9 and oo, with oo signifies that no depth constraint is imposed.</footnote><p>The individual feature contributions were dis­cussed in greater detail in (Huang et al., 2008). Also, The SVM (rathern than ME model) was em­ployed using the same feature set and the results were very close (93.4% for 6 class and 89.2% for 50 class). Table 3 shows the feature ablation ex­periment<footnote anchor="4"/> which is missing in that paper. The experiment shows that the proposed head word and its hypernym features play an essential role in building an accurate question classifier.</p><p>Table 3: Question classification accuracy by re­moving one feature at a time for 6 and 50 classes over UIUC split._</p><p>Our best result feature space only consists of 13'697 binary features and each question has 10 to 30 active features. Compared to the over feature size of 200'000 in Li and Roth (2002), our feature space is much more compact, yet turned out to be more informative as suggested by the experiments. Table 4 shows the summary of the classification accuracy of all question classifiers which were ap­plied to UIUC dataset.<footnote anchor="5"/> Our results are summa­rized in the last row.</p><p>In addition, we have performed the 10 cross validation experiment over the 5500 UIUC train­ing corpus using our best model. The result is 89.05il.25 and83.73±1.61 for 6 and50 classes,<footnote anchor="6"/>which outperforms the best result of 86.<b>1 ± 1.1 </b>for 6 classes as reported in (Moschitti et al., 2007).</p></section><section number="5" title="Question Answer Features"><p>For a pair of a question and a candidate sentence, we extract binary features which include CoNLL named entities presence feature (NE), dictionary entities presence feature (DIC), numerical entities presence feature (NUM), question specific feature (SPE), and dependency validity feature (DEP).<page local="4" global="546"/></p><footnote label="4">Remove one feature at a time from the entire feature set.</footnote><footnote label="5">Note (1) that SNoW accuracy without the related word dictionary was not reported. With the semantically related word dictionary, it achieved 91%. Note (2) that SNoW with a semantically related word dictionary achieved 84.2% but the other algorithms did not use it.</footnote><footnote label="6">These results are worse than the result over UIUC split; as the UIUC test data includes a larger percentage of easily classified question types.</footnote><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">545</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>6 class</p></td><td class="cell"><p>50 class</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>overall</p></td><td class="cell"><p>93.6</p></td><td class="cell"><p>89.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>- wh-word</p></td><td class="cell"><p>93.6</p></td><td class="cell"><p>89.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>- head word</p></td><td class="cell"><p>92.8</p></td><td class="cell"><p>88.2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>- hypernym</p></td><td class="cell"><p>90.8</p></td><td class="cell"><p>84.2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>- unigram</p></td><td class="cell"><p>93.6</p></td><td class="cell"><p>86.8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>- word shape</p></td><td class="cell"><p>93.0</p></td><td class="cell"><p>88.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>6 class</p></td><td class="cell"><p>50 class</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>wh-word</p></td><td class="cell"><p>46.0</p></td><td class="cell"><p>46.8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>+ head word</p></td><td class="cell"><p>92.2</p></td><td class="cell"><p>82.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>+ hypernym</p></td><td class="cell"><p>91.8</p></td><td class="cell"><p>85.6</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>+ unigram</p></td><td class="cell"><p>93.0</p></td><td class="cell"><p>88.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>+ word shape</p></td><td class="cell"><p>93.6</p></td><td class="cell"><p>89.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 4: Accuracy of all question classifiers which were applied to UIUC dataset."></table><subsection number="5.1" title="CoNLL named entities presence feature"><p>We use Stanford named entity recognizer (NER) (Finkel et al., 2005) to identify CoNLL style NEs<footnote anchor="7"/>as possible answer strings in a candidate sentence for a given type of question. In particular, if the question is ABBR type, we tag CoNLL LOC, ORG and MISC entities as candidate answers; If the question is HUMAN type, we tag CoNLL PER and ORG entities; And if the question is LOC type, we tag CoNLL LOC and MISC entities. For other types of questions, we assume there is no candidate CoNLL NEs to tag. We create a binary feature <b>NE </b>to indicate the presence or absence of tagged CoNLL entities. Further more, we cre­ate four binary features <b>NE-PER, NE-LOC, NE-ORG, </b>and <b>NE-MISC </b>to indicate the presence of tagged CoNLL PER, LOC, ORG and MISC enti­ties.</p></subsection><subsection number="5.2" title="Dictionary entities presence feature"><p>As four types of CoNLL named entities are not enough to cover 50 question types, we include the 101 dictionary files compiled in the Ephyra project (Schlaefer et al., 2007). These dictionary files con­tain names for specific semantic types. For exam­ple, the <i>actor </i>dictionary comprises a list of actor names such as <i>Tom Hanks </i>and <i>Kevin Spacey. </i>For each question, if the head word of such question (see Section 3) matches the name of a dictionary file, then each noun phrase in a candidate sentence is looked up to check its presence in the dictio­nary. If so, a binary <b>DIC </b>feature is created. For example, for the question <i>What rank did Chester</i> <i>Nimitz reach, </i>as there is a <i>military rank </i>dictionary matches the head word <i>rank, </i>then all the noun phrases in a candidate sentence are looked up in the <i>military rank </i>dictionary. As a result, a sen­tence contains word <i>Admiral </i>will result in the DIC feature being activated, as such word is present in the <i>military rank </i>dictionary.</p><footnote label="7">Person (PER), location (LOC), organization (ORG), and miscellaneous (MISC).</footnote><p>Note that an implementation tip is to allow the proximity match in the dictionary look up. Con­sider the question <i>What film introduced Jar Jar Binks. </i>As there is a match between the ques­tion head word <i>film </i>and the dictionary named <i>film, </i>each noun phrase in the candidate sentence is checked. However, no dictionary entities have been found from the candidate sentence <i>Best plays Jar Jar Binks, a floppy-eared, two-legged creature in "Star Wars: Episode I - The Phantom Men­ace", </i>although there is movie entitled <i>Star Wars Episode I: The Phantom Menace </i>in the dictionary. Notice that <i>Star Wars: Episode I - The Phantom Menace </i>in the sentence and the dictionary entity <i>Star Wars Episode I: The Phantom Menace </i>do not have exactly identical spelling. The use of prox­imity look up which allows edit distance being less than 10% error can resolve this.</p></subsection><subsection number="5.3" title="Numerical entities presence feature"><p>There are so far no match for question types of NUM (as shown in Table 1) including NUM:count and NUM:date etc. These types of questions seek the numerical answers such as the amount of money and the duration of period. It is natural to compile regular expression patterns to match such entities. For example, for a NUM:money typed question <i>What is Rohm and Haas's annual rev­enue, </i>we compile NUM:money regular expression pattern which matches the strings of number fol­lowed by a currency sign ($ and <i>dollars </i>etc). Such pattern is able to identify <i>4 billion $ </i>as a candidate answer in the candidate sentence <i>Rohm and Haas, with 4 billion $ in annual sales... </i>There are 13 pat­terns compiled to cover all numerical types. We create a binary feature <b>NUM </b>to indicate the pres­ence of possible numerical answers in a sentence.</p></subsection><subsection number="5.4" title="Specific features"><p>Specific features are question dependent. For ex­ample, for question <i>When was James Dean born, </i>any candidate sentence matches the pattern <i>James Dean (number - number) </i>is likely to answer such question. We create a binary feature <b>SPE </b>to indi­cate the presence of such match between a question and a candidate sentence.<page local="5" global="547"/> We list all question and sentence match patterns which are used in our experiments as following:</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">546</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Algorithm</p></td><td class="cell"><p>6 class</p></td><td class="cell"><p>50 class</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Li and Roth, SNoW</p></td><td class="cell"><p>_(i)</p></td><td class="cell"><p>78.8(<footnote anchor="2"/>)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Hacioglu et al., SVM+ECOC</p></td><td class="cell"><p><b>—</b></p></td><td class="cell"><p>80.2-82</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Zhang &amp; Lee, Linear SVM</p></td><td class="cell"><p>87.4</p></td><td class="cell"><p>79.2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Zhang &amp; Lee, Tree SVM</p></td><td class="cell"><p>90.0</p></td><td class="cell"><p><b>—</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Krishnan et al., SVM+CRF</p></td><td class="cell"><p>93.4</p></td><td class="cell"><p>86.2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Moschitti et al., Kernel</p></td><td class="cell"><p>91.8</p></td><td class="cell"><p>-</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Maximum Entropy Model</p></td><td class="cell"><p>93.6</p></td><td class="cell"><p>89.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p><b>when born feature 1 </b>The question begins with <i>when is/was </i>and follows by a person name and then follows by key word <i>born; </i>The candidate sentence contains such per­son name which follows by the pattern of <i>(number -number).</i></p><p><b>when born feature 2 </b>The question begins with <i>when is/was </i>and follows by a person name and then follows by key word <i>born; </i>The candidate sentence contains such per­son name, a NUM:date entity, and a key word <i>born.</i></p><p><b>where born feature 1 </b>The question begins with <i>where is/was </i>and follows by a person name and then follows by key word <i>born; </i>The candidate sentence contains such person name, a NER LOC entity, and a key word <i>born.</i></p><p><b>when die feature 1 </b>The question begins with <i>when did </i>and follows by a person name and then follows by key word <i>die; </i>The candidate sentence contains such person name which follows by the pattern of <i>(number - number).</i></p><p><b>when die feature 2 </b>The question begins with <i>when did </i>and follows by a person name and then follows by key word <i>die; </i>The candidate sentence contains such person name, a NUM:date entity, and a key word <i>died.</i></p><p><b>how many feature </b>The question begins with <i>how many </i>and follows by a noun; The candidate sentence contains a number and then follows by such noun.</p><p><b>cooccurrent Feature </b>This feature takes two phrase argu­ments, if the question contains the first phrase and the candidate sentence contains the second, such feature would be activated.</p><p>Note that the construction of specific features require the access to aforementioned extracted named entities. For example, the <b>when born fea­ture 2 </b>pattern needs the information whether a candidate sentence contains a NUM:date entity and <b>where born feature 1 </b>pattern needs the in­formation whether a candidate sentence contains a NER LOC entity. Note also that the patterns of <b>when born feature </b>and <b>when die feature </b>have similar structure and thus can be simplified in im­plementation. <b>How many feature </b>can be used to identify the sentence <i>Amtrak annually serves about 21 million passengers </i>for question <i>How many passengers does Amtrak serve annually. </i>The <b>cooccurrent feature </b>is the most general one. An example of cooccurrent feature would take the arguments of <i>marry </i>and <i>husband, </i>or <i>marry </i>and <i>wife. </i>Such feature would be activated for ques­tion <i>Whom did Eileen Marie Collins marry </i>and candidate sentence <i>... were Collins' husband, Pat Youngs, an airline pilot... </i>It is worth noting that the two arguments are not necessarily differ­ent. For example, they could be both <i>established, </i>which makes such feature activated for question <i>When was the IFC established </i>and candidate sen­tence <i>IFC was established in 1956 as a member of the World Bank Group.</i><i> </i>The reason why we use the cooccurrence of the word <i>established </i>is due to its main verb role, which may carry more information than other words.</p></subsection><subsection number="5.5" title="Dependency validity features"><p>Like (Cui et al., 2004), we extract the dependency path from the question word to the common word (existing in both question and sentence), and the path from candidate answer (such as CoNLL NE and numerical entity) to the common word for each pair of question and candidate sentence using Stanford dependency parser (Klein and Manning, 2003; Marneffe et al., 2006). For example, for question <i>When did James Dean die </i>and candidate sentence <i>In 1955, actor James Dean was killed in a two-car collision near Cholame, Calif.</i>, we ex­tract the pathes of <i>When:advmod:nsubj:Dean </i>and <i>1955:prep-in:nsubjpass:Dean </i>for question and sentence respectively, where <i>advmod </i>and <i>nsubj </i>etc. are grammatical relations. We propose the dependency validity feature <b>(DEP) </b>as following. For all paired paths between a question and a can­didate sentence, if at least one pair of path in which all pairs of grammatical relations have been seen in the training, then the DEP feature is set to be true, false otherwise. That is, the true validity fea­ture indicates that at least one pair of path between the question and candidate sentence is possible to be a true pair (ie, the candidate noun phrase in the sentence path is the true answer).</p></subsection></section><section number="6" title="Question Answer Experiments"><p>Recall that most of the question answer features depend on the question classifier. For instance, the NE feature checks the presence or absence of CoNLL style named entities subject to the clas­sified question type. In this section, we evaluate how the quality of question classifiers affects the question answering performance.</p><subsection number="6.1" title="Experiment setup"><p>We use TREC99-03 factoid questions for training and TREC04 factoid questions for testing. To fa­cilitate the comparison to others work (Cui et al., 2004; Shen and Klakow, 2006), we first retrieve all relevant documents which are compiled by Ken Litkowski<footnote anchor="8"/> to create training and test datasets. We then apply key word search for each question and retrieve the top 20 relevant sentences.<page local="6" global="548"/> We create a feature represented data point using each pair of question and candidate sentence and label it either <i>true </i>or <i>false </i>depending on whether the sentence can answer the given question or not. The labeling is conducted by matching the gold factoid answer pattern against the candidate sentence.</p><footnote label="8">Available at http://trec.nist.gov/data/qa.html .</footnote><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">547</doubt><p>There are two extra steps performed for train­ing set but not for test data. In order to construct a high quality training set, we manually check the correctness of the training data points and remove the false positive ones which cannot support the question although there is a match to gold answer. In addition, in order to keep the training data well balanced, we keep maximum four false data points (question answer pair) for each question but no limit over the true label data points. In doing so, we use 1458 questions to compile 8712 training data points and among them 1752 have true labels. Similarly, we use 202 questions to compile 4008 test data points and among them 617 have true la­bels.</p><p>We use the training data to train a maximum entropy model and use such model to rank test data set. Compared with a classification task (such as the question classifier), the ranking process re­quires one extra step: For data points which share the same question, the probabilities of being pre­dicted as true label are used to rank the data points. In align with the previous work, performance is evaluated using mean reciprocal rank (MRR), top 1 prediction accuracy (topi) and top 5 prediction accuracy (top5). For the test data set, 157 among the 202 questions have correct answers found in retrieved sentences. This leads to the upper bound of MRR score being 77.8%.</p><p>To evaluate how the quality of question clas­sifiers affects the question answering, we have created three question classifiers: QC1, QC2 and QC3. The features which are used to train these question classifiers and their performance are shown in Table 5. Note that QC3 is the best question classifier we obtained in Section 4.</p><table caption="Table 5: Features used to train and the perfor­mance of three question classifiers."></table></subsection><subsection number="6.2" title="Experiment results"><p>The first experiment is to evaluate the individ­ual contribution of various features derived using three question classifiers. Table 6 shows the base­line result and results using DIC, NE, NE-4, REG, SPE, and DEP features. The baseline is the key word search without the use of maximum entropy model. As can be seen, the question classifiers do not affect the DIC feature at all, as DIC fea­ture does not depend on question classifiers. Bet­ter question classifier boosts considerable gain for NE, NE-4 and REG in their contribution to ques­tion answering. For example, the best question classifier QC3 outperforms the worst one (QC1) by 1.5%, 2.0%, and 2.0% MRR scores for NE, NE-4 and REG respectively. However, it is sur­prising that the MRR and top5 contribution of NE and NE-4 decreases if QC1 is replaced by QC2, al­though the topi score results in performance gain slightly. This unexpected results can be partially explained as follows. For some questions, even QC2 produces correct predictions, the errors of NE and NE-4 features may cause over-confident scores for certain candidate sentences. As SPE and DEP are not directly dependent on question clas­sifier, their individual contribution only changes slightly or remains the same for different ques­tion classifiers. If the best question classifier is used, the most important features are SPE and REG, which can individually boost the MRR score over 54%, while the others result in less significant gains.</p><p>We now incrementally use various features and the results are show in Table 6 as well. As can be seen, the more features and the better question classifier are used, the higher performance the ME model has. The inclusion of REG and SPE results in significant boost for the performance. For ex­ample, if the best question classifier QC3 is used, the REG results in 6.9% and 8% gain for MRR and topi scores respectively. This is due to a large portion of NUM type questions in test dataset. The SPE feature contributes significantly to the per­formance due to its high precision in answering birth/death time/location questions. NE and NE-4 result in reasonable gains while DEP feature con­tributes little. However, this does not mean that DEP is not important, as once the model reaches a high MRR score, it becomes hard to improve.</p><table caption="Table 6 clearly shows that the question type classifier plays an essential role in a high perfor-"></table><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">548</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Name</p></td><td class="cell"><p>features</p></td><td class="cell"><p>6 class</p></td><td class="cell"><p>50 class</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>QC1</p></td><td class="cell"><p>wh-word</p></td><td class="cell"><p>46.0</p></td><td class="cell"><p>46.8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>QC2</p></td><td class="cell"><p>wh-word+ head</p></td><td class="cell"><p>92.2</p></td><td class="cell"><p>82.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>QC3</p></td><td class="cell"><p>All</p></td><td class="cell"><p>93.6</p></td><td class="cell"><p>89.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="7" global="549"/><p>Table <u>6: Performance of individual and incremental feature sets for three question classifiers.</u></p><p>mance question answer system. Assume all the features are used, the better question classifier sig­nificantly boosts the overall performance. For ex­ample, the best question classifier QC3 outper­forms the worst QC1 by 4.9%, 6.5%, and 2.0% for MRR, top1 and top5 scores respectively. Even compared to a good question classifier QC2, the for MRR, top1 and top5 scores respectively. One can imagine that if a fine grained NER is available (rather than the current four type coarse NER), the potential gain is much significant.</p><doubt alpha="50.0" length="46" tooSmall="False" monospace="0.0">gain of using QC3 is still 3.4%, 4.0% and 2.0%</doubt><p>The reason that the question classifier affects the question answering performance is straightfor­ward. As a upstream source, the incorrect classi­fication of question type would confuse the down­stream answer search process. For example, for question <i>What is Rohm and Haas's annual rev­enue, </i>our best question classifier is able to clas­sify it into the correct type of NUM:money and thus would put <i>$ 4 billion </i>as a candidate answer. However, the inferior question classifiers misclas-sify it into HUM:ind type and thereby could not return a correct answer. Figure 1 shows the indi­vidual MRR scores for the 42 questions (among the 202 test questions) which have different pre­dicted question types using QC3 and QC2. For al­most all test questions, the accurate question clas­sifier QC3 achieves higher MRR scores compared to QC2.</p><table caption="Table 7 shows performance of various question answer systems including (Tanev et al., 2004; Wu"></table><doubt alpha="50.0" length="48" tooSmall="False" monospace="0.0">et al., 2005; Cui et al., 2004; Shen and Klakow,</doubt></subsection></section><section number="20" title="25 question id"><p>Figure 1: Individual MRR scores for questions which have different predicted question types us­ing QC3 and QC2.</p><p>2006) and this work which were applied to the same training and test datasets. Among all the sys­tems, our model can achieve the best MRR score of 66.3%, which is close to the state of the art of 67.0%. Considering the question answer features used in this paper are quite standard, the boost is mainly due to our accurate question classifier.</p><table caption="Table 7: Various system performance comparison."></table><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">2</doubt><doubt alpha="0.0" length="1" tooSmall="True" monospace="0.0">0</doubt><doubt alpha="0.0" length="1" tooSmall="True" monospace="0.0">5</doubt><doubt alpha="0.0" length="2" tooSmall="True" monospace="0.0">10</doubt><doubt alpha="0.0" length="2" tooSmall="True" monospace="0.0">15</doubt><doubt alpha="0.0" length="2" tooSmall="True" monospace="0.0">30</doubt><doubt alpha="0.0" length="2" tooSmall="True" monospace="0.0">35</doubt><doubt alpha="0.0" length="2" tooSmall="True" monospace="0.0">40</doubt><doubt alpha="0.0" length="2" tooSmall="True" monospace="0.0">45</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">549</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Individual</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Feature</p></td><td class="cell"><p>MRR</p></td><td class="cell"><p><b>1 </b>Top1</p></td><td class="cell"><p>Top5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>QC1</p></td><td class="cell"><p>QC2</p></td><td class="cell"><p>QC3</p></td><td class="cell"><p>QC1</p></td><td class="cell"><p>QC2</p></td><td class="cell"><p>QC3</p></td><td class="cell"><p>QC1</p></td><td class="cell"><p>QC2</p></td><td class="cell"><p>QC3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Baseline</p></td><td class="cell"><p>49.9</p></td><td class="cell"><p>49.9</p></td><td class="cell"><p>49.9</p></td><td class="cell"><p>40.1</p></td><td class="cell"><p>40.1</p></td><td class="cell"><p>40.1</p></td><td class="cell"><p>59.4</p></td><td class="cell"><p>59.4</p></td><td class="cell"><p>59.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>DIC</p></td><td class="cell"><p>49.5</p></td><td class="cell"><p>49.5</p></td><td class="cell"><p>49.5</p></td><td class="cell"><p>42.6</p></td><td class="cell"><p>42.6</p></td><td class="cell"><p>42.6</p></td><td class="cell"><p>60.4</p></td><td class="cell"><p>60.4</p></td><td class="cell"><p>60.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NE</p></td><td class="cell"><p>48.5</p></td><td class="cell"><p>47.5</p></td><td class="cell"><p>50.0</p></td><td class="cell"><p>40.6</p></td><td class="cell"><p>40.6</p></td><td class="cell"><p>42.6</p></td><td class="cell"><p>61.9</p></td><td class="cell"><p>60.9</p></td><td class="cell"><p>63.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NE-4</p></td><td class="cell"><p>49.5</p></td><td class="cell"><p>48.5</p></td><td class="cell"><p>51.5</p></td><td class="cell"><p>41.6</p></td><td class="cell"><p>42.1</p></td><td class="cell"><p>44.6</p></td><td class="cell"><p>62.4</p></td><td class="cell"><p>61.9</p></td><td class="cell"><p>64.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>REG</p></td><td class="cell"><p>52.0</p></td><td class="cell"><p>54.0</p></td><td class="cell"><p>54.0</p></td><td class="cell"><p>44.1</p></td><td class="cell"><p>47.0</p></td><td class="cell"><p>47.5</p></td><td class="cell"><p>64.4</p></td><td class="cell"><p>65.3</p></td><td class="cell"><p>65.3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SPE</p></td><td class="cell"><p>55.0</p></td><td class="cell"><p>55.0</p></td><td class="cell"><p>55.0</p></td><td class="cell"><p>48.5</p></td><td class="cell"><p>48.5</p></td><td class="cell"><p>48.5</p></td><td class="cell"><p>64.4</p></td><td class="cell"><p>64.4</p></td><td class="cell"><p>64.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>DEP</p></td><td class="cell"><p>51.0</p></td><td class="cell"><p>51.5</p></td><td class="cell"><p>52.0</p></td><td class="cell"><p>43.6</p></td><td class="cell"><p>44.1</p></td><td class="cell"><p>44.6</p></td><td class="cell"><p>65.3</p></td><td class="cell"><p>65.8</p></td><td class="cell"><p>65.8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Incremental</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Baseline</p></td><td class="cell"><p>49.9</p></td><td class="cell"><p>49.9</p></td><td class="cell"><p>49.9</p></td><td class="cell"><p>40.1</p></td><td class="cell"><p>40.1</p></td><td class="cell"><p>40.1</p></td><td class="cell"><p>59.4</p></td><td class="cell"><p>59.4</p></td><td class="cell"><p>59.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>+DIC</p></td><td class="cell"><p>49.5</p></td><td class="cell"><p>49.5</p></td><td class="cell"><p>49.5</p></td><td class="cell"><p>42.6</p></td><td class="cell"><p>42.6</p></td><td class="cell"><p>42.6</p></td><td class="cell"><p>60.4</p></td><td class="cell"><p>60.4</p></td><td class="cell"><p>60.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>+NE</p></td><td class="cell"><p>50.0</p></td><td class="cell"><p>48.5</p></td><td class="cell"><p>51.0</p></td><td class="cell"><p>43.1</p></td><td class="cell"><p>42.1</p></td><td class="cell"><p>44.6</p></td><td class="cell"><p>62.9</p></td><td class="cell"><p>61.4</p></td><td class="cell"><p>64.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>+NE-4</p></td><td class="cell"><p>51.5</p></td><td class="cell"><p>50.0</p></td><td class="cell"><p>53.0</p></td><td class="cell"><p>44.1</p></td><td class="cell"><p>43.6</p></td><td class="cell"><p>46.0</p></td><td class="cell"><p>63.4</p></td><td class="cell"><p>62.9</p></td><td class="cell"><p>65.8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>+REG</p></td><td class="cell"><p>55.0</p></td><td class="cell"><p>56.9</p></td><td class="cell"><p>59.9</p></td><td class="cell"><p>48.0</p></td><td class="cell"><p>51.0</p></td><td class="cell"><p>54.0</p></td><td class="cell"><p>68.3</p></td><td class="cell"><p>68.8</p></td><td class="cell"><p>71.8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>+SPE</p></td><td class="cell"><p>60.4</p></td><td class="cell"><p>62.4</p></td><td class="cell"><p>65.3</p></td><td class="cell"><p>55.4</p></td><td class="cell"><p>58.4</p></td><td class="cell"><p>61.4</p></td><td class="cell"><p>70.8</p></td><td class="cell"><p>70.8</p></td><td class="cell"><p>73.8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>+DEP</p></td><td class="cell"><p>61.4</p></td><td class="cell"><p>62.9</p></td><td class="cell"><p><b>66.3</b></p></td><td class="cell"><p>55.9</p></td><td class="cell"><p>58.4</p></td><td class="cell"><p><b>62.4</b></p></td><td class="cell"><p>71.8</p></td><td class="cell"><p>71.8</p></td><td class="cell"><p><b>73.8</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>System</p></td><td class="cell"><p>MRR</p></td><td class="cell"><p>Top1</p></td><td class="cell"><p>Top5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Tanev et al. 2004</p></td><td class="cell"><p>57.0</p></td><td class="cell"><p>49.0</p></td><td class="cell"><p>67.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Cui et al. 2004</p></td><td class="cell"><p>60.0</p></td><td class="cell"><p>53.0</p></td><td class="cell"><p>70.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Shen and Klakow, 2006</p></td><td class="cell"><p>67.0</p></td><td class="cell"><p>62.0</p></td><td class="cell"><p>74.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>This work</p></td><td class="cell"><p>66.3</p></td><td class="cell"><p>62.4</p></td><td class="cell"><p>73.8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="8" global="550"/></section><section number="7" title="Conclusion"><p>In this paper, we have presented a question clas­sifier which makes use of a compact yet effi­cient feature set. The question classifier outper­forms previous question classifiers over the stan­dard UIUC question dataset. We further investi­gated quantitatively how the quality of question classifier impacts the performance of question an­swer system. The experiments showed that an ac­curate question classifier plays an essential role in question answering system. With our accurate question classifier and some standard question an­swer features, our question answering system per­forms close to the state of the art.</p></section><section title="Acknowledgments"><p>We wish to thank the three anonymous review­ers for their invaluable comments. This re­search was supported by British Telecom grant CT1080028046 and BISC Program of UC Berke­ley.</p></section><references><p>A. L. Berger, S. A. D. Pietra, and V. J. D. Pietra. 1996. A maximum entropy approach to natural language processing. <i>Computational Linguistics, </i>22(1):39-71.</p><p>M. Collins. 1999. Head-driven statistical models for natural language parsing. <i>PhD thesis, </i>University of Pennsylvania.</p><p>H. Cui, K Li, R. Sun, T. Chua, and M. Kan. 2004. Na­tional university of singapore at the trec-13 question answering. In <i>Proc. of TREC 2004, NIST.</i></p><p>J. Finkel, T. Grenager, and C. Manning. 2005. In­corporating non-local information into information extraction systems by Gibbs sampling. In <i>Proc. of</i> <i>ACL, </i>pages 363-370.</p><p>D. Hacioglu and W. Ward. 2003. Question classifica­tion with support vector machines and error correct­ing codes. In <i>Proc. ofthe ACL/HLT, </i>vol. 2, pages 28-30.</p><p>Z. Huang, M. Thint, and Z. Qin. 2008. Question clas­sification using head words and their hypernyms. In <i>Proc. oftheEMNLP.</i></p><p>D. Klein and C. D. Manning. 2003. Accurate unlexi-calized parsing. In <i>Proc. of ACL 2003, </i>vol. 1, pages 423-430.</p><p>V. Krishnan, S. Das, and S. Chakrabarti. 2005. En­hanced answer type inference from questions using sequential models. In <i>Proc. ofthe HLT/EMNLP.</i></p><p>M. Lesk. 1986. Automatic sense disambiguation us­ing machine readable dictionaries: how to tell a pine cone from an ice cream cone. In <i>ACM Special Inter­est Group for Design ofCommunication Proceed­ings ofthe 5th annual international conference on Systems documentation, </i>pages 24-26.</p><p>X. Li and D. Roth. 2002. Learning question classi­fiers. In <i>the 19th international conference on Com­putational linguistics, </i>vol. 1, pages 1-7.</p><p>X. Li and D.Roth. 2006. Learning question classifiers: the role of semantic information. <i>Natural Language Engineering, </i>12(3):229-249.</p><p>C. D. Manning and D. Klein. 2003. Optimization, maxent models, and conditional estimation with­out magic. <i>Tutorial at HLT-NAACL 2003 and ACL</i> <i>2003.</i><i></i></p><p>M. D. Marneffe, B. MacCartney and C. D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In <i>Proc. ofLREC 2006.</i></p><p>A. Moschitti, S. Quarteroni, R. Basili and S. Manand-har 2007. Exploiting syntactic and shallow seman­tic kernels for question answer classification. In <i>Proc. of ACL 2007, </i>pages 776-783.</p><p>S. Petrov and D. Klein. 2007. Improved inference for unlexicalized parsing. In <i>Proc. ofthe HLT-NAACL.</i></p><p>J. Prager.  2006.  Open-domain question-answering.</p><p>In <i>Foundations and Trends in Information Retrieval, </i>vol. 1, pages 91-231, 2006.</p><p>N. Schlaefer, J. Ko, J. Betteridge, G. Sautter, M. Pathak and E. Nyberg. 2007. Semantic extensions of the Ephyra QA system for TREC 2007. In <i>Proc. ofthe TREC 2007.</i></p><p>D. Shen and D. Klakow. 2006. Exploring correlation of dependency relation paths for answer extraction.</p><p>In <i>Proc. ofthe ACL 2006.</i></p><p>H. Tanev, M. Kouylekov, and B. Magnini. 2004. Combining linguistic processing and web mining for question answering: Itc-irst at TREC-2004. In <i>Proc.</i></p><p><i>ofthe TREC 2004, NIST.</i></p><p>E. M. Voorhees and H. T. Dang. 2005. Overview of the TREC 2005 question answering track. In <i>Proc.</i></p><doubt alpha="61.4" length="70" tooSmall="False" monospace="0.0">ofthe TREC 2005, NIST.M. Wu, M. Duan, S. Shaikh, S. Small, andT.Strza-</doubt><p>lkowski. 2005. University at Albanys ILQUA in</p><p>TREC 2005. In <i>Proc. ofthe TREC 2005, NIST.</i></p><p>D. Zhang and W. S. Lee. 2003. Question classification using support vector machines. In <i>The ACM SIGIR conference in information retrieval, </i>pages 26-32.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">550</doubt></references></body></article>