<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="406"/><title>Systematic Knowledge Acquisition for Question Analysis</title><pubinfo>Proceedings of Recent Advancesin NaturalLanguage Processing,pages 406^12, Hissar, Bulgaria, 12-14 September 2011.</pubinfo><author surname="Nguyen" givenname="Dat Quoc"><org  name="Konan University" country="Japan" city="Kobe"/></author><author surname="Nguyen" givenname="Dai Quoc"><org  name="Konan University" country="Japan" city="Kobe"/></author><author surname="Pham" givenname="Son Bao"><org  name="Konan University" country="Japan" city="Kobe"/></author></firstpageheader><frontmatter><p><b>Systematic Knowledge Acquisition for Question Analysis</b></p><p><b>Dat Quoc Nguyent </b>ö and <b>Dai Quoc Nguyen^ </b>and <b>Son Bao Phamt </b>ï</p><p>t Faculty of Information Technology University of Engineering and Technology Vietnam National University, Hanoi {datnq, dainq, sonpb}@vnu.edu.vn * Information Technology Institute Vietnam National University, Hanoi</p></frontmatter><abstract>For the task of turning a natural language ques­tion into an explicit intermediate representa­tion of the complexity in question answering systems, all published works so far use rule-based approach to the best of our knowledge. We believe it is because of the complexity of the representation and the variety of question types and also there are no publicly available corpus of a decent size. In these rule-based ap­proaches, the process of creating rules is not discussed. It is clear that manually creating the rules in an ad-hoc manner is very expen­sive and error-prone. In this paper, we focus on the process of creating those rules manu­ally, in a way that consistency between rules is maintained and the effort to create a new rule is independent of the size of the current rule set. Experimental results are promising where our system achieves better performance and re­quires much less time and cognitive load com­pared to previous work. </abstract></header><body><section number="1" title="Introduction"><p>The goal of question answering systems is to give an­swers to the user's questions instead of ranked lists of related documents as used by most current search en­gines (Hirschman and Gaizauskas, 2001). Natural lan­guage question analysis component is the first compo­nent in any question answering systems. This compo­nent creates an intermediate representation of the input question, which is expressed in natural language, to be utilized in the rest of the system.</p><p>In this paper, we introduce a language independent approach to systematically build a knowledge base for analyzing natural language questions. Natural language questions will be transformed into intermediate repre­sentation elements which include construction type of question, class of question, keywords in question and semantic constraints between them.</p><p>Some question answering systems such as Aqualog (Lopez et al., 2007) and Vietnamese question answer­ing system (VnQAS) (Nguyen et al., 2009) manually</p><p>^Both authors contributed equally to this work.</p><p>defined a list of sequence pattern structures to analyze questions. As rules are created in an ad-hoc manner, these systems share a common difficulty in managing interaction between rules and keeping consistency. In our approach, we present an approach utilizing Ripple Down Rules (Compton and Jansen, 1990) (Richards, 2009) knowledge acquisition methodology to acquire rules in a systematic manner which avoids unintended interaction between rules.</p><p>In section 2, we provide some related works and de­scribe our overall system architecture in section 3. We present our knowledge acquisition approach for ques­tion analysis in section 4. We describe our experiments in section 5. Discussion and conclusion will be pre­sented in section 6.</p></section><section number="2" title="Related works"><subsection number="2.1" title="Question analysis in question answering systems"><p>Early NLIDB systems used pattern-matching technique to process user's question and generate corresponding answer (Androutsopoulos, 1995). A common technique for parsing input questions in NLIDB approaches is syntax analysis where a natural language question is directly mapped to a database query (such as SQL) through grammar rules. Nguyen and Le (Nguyen and Le, 2008) introduced a NLIDB question answering system in Vietnamese employing semantic grammars. Their system includes two main modules: QTRAN and TGEN. QTRAN (Query Translator) maps a natural lan­guage question to an SQL query while TGEN (Text Generator) generates answers based on the query result tables. QTRAN uses limited context-free grammars to analyze user's question into syntax tree via CYK algo­rithm.</p><p>Recently, some question answering systems that used semantic annotations generated high results in natural language question analysis. A well known annotation based framework is GATE (Cunningham et al., 2002) which have been used in many question answering sys­tems especially for the natural language question analy­sis module such as Aqualog (Lopez et al., 2007), Ques-tIO (Damljanovic et al., 2008), VnQAS (Nguyen et al., 2009).</p><p>Aqualog and VnQAS are ontology-based question answering systems for English and Vietnamese respectively.<page local="2" global="407"/> Both systems take a natural language question and an ontology as its input, and return answers for users based on the semantic analysis of the question and the corresponding elements in the ontology. Gen­eral architecture of these systems can be described as a waterfall model where a natural language question is mapped to an intermediate representation. The subse­quent modules of the system process the intermediate representation to provide queries with respect to the in­put ontology. These systems perform semantic and syn­tactic analysis of the input question through the use of processing resources wrapped as GATE plug-ins such as word segmentation, sentence segment and part-of-speech tagging.</p></subsection><subsection number="2.2" title="Single Classification Ripple Down Rules"><p>In this section we present the basic idea of Ripple-Down Rules (RDR) (Compton and Jansen, 1990) which inspired our approach. RDR allows one to add rules to a knowledge base incrementally without the need of a knowledge engineer. A new rule is only created when the KB performs unsatisfactorily on a given case. The rule represents an explanation for why the conclusion should be different from the KB's conclusion on the case at hand.</p><p><i>A Single Classification Ripple Down Rules </i>(SCRDR) tree is a binary tree with two distinct types of edges. These edges are typically called <i>except </i>and <i>if-not </i>edges. Associated with each node in a tree is a <i>rule. </i>A rule has the form: <i>if a then ß </i>where <i>a </i>is called the <i>condition </i>and ß the <i>conclusion.</i></p><p>Cases in SCRDR are evaluated by passing a case (a sentence to be classified in our case for example) to the root of the tree. At any node in the tree, if the condi­tion of a node <i>N</i>'s rule is satisfied by the case, the case is passed on to the exception child of N using the <i>ex­cept </i>link if it exists. Otherwise, the case is passed on to the N's <i>if-not </i>child. The conclusion given by this pro­cess is the conclusion from the last node in the RDR tree which <i>fired </i>(satisfied by the case). To ensure that a conclusion is always given, the root node typically con­tains a trivial condition which is always satisfied. This node is called the <i>default </i>node.</p><p>A new node is added to an SCRDR tree when the evaluation process returns the wrong conclusion. The new node is attached to the last node in the evaluation path of the given case with the <i>except </i>link if the last node is the <i>fired </i>rule. Otherwise, it is attached with the <i>if-not </i>link.</p><p>RDR based approaches have been used to tackle NLP tasks such as POS tagging (Nguyen et al., 2011), text classification and information extraction (Pham and Hoffmann, 2006).</p></subsection></section><section number="3" title="Our Question Answering System Architecture"><p>The architecture of our question answering system is shown in Figure 1. It includes two components: the Natural language question analysis engine and the Answer retrieval.</p><p>The question analysis component consists of three modules: preprocessing, syntactic analysis and seman­tic analysis. It takes the user question as an input and re­turns a query-tuple representing the question in a com­pact form. The role of this intermediate representation is to provide structured information of the input ques­tion for later processing such as retrieving answers. Our contribution focuses on the semantic analysis module by proposing a rule language and a systematic process­ing to create rules in a way that interaction between rules are controlled and consistency are maintained.</p><p>Similar to VnQAS (Nguyen et al., 2009), the an­swer retrieval component includes two main modules: Ontology Mapping and Answer Extraction. It takes an intermediate representation produced by the question analysis component and an Ontology as its input to gen­erate semantic answers.</p><p>To set the context for the discussion on the systematic knowledge acquisition process in the semantic analysis module, we will describe our question analysis compo­nent in details.</p><p>We wrapped existing linguistic processing modules for Vietnamese such as Word Segmentation, Part-of-speech tagger (Pham et al., 2009) as GATE plug-ins. Results of the modules are annotations capturing infor­mation such as sentences, words, nouns and verbs. Each annotation has a set of feature-value pairs. For example, a word has a feature <i>category </i>storing its part-of-speech tag. This information can then be reused for further processing in subsequent modules. New modules are specifically designed to handle Vietnamese questions using patterns over existing linguistic annotations. This is achieved using GATE JAPE (Java Annotation Pattern Engine) transducers, a set of JAPE grammars. A JAPE grammar allows one to specify regular expression pat­tern based on semantic annotations.</p><subsection number="3.1" title="Preprocessing module"><p>The preprocessing module generates <i>TokenVn </i>anno­tations representing a Vietnamese word with features such as part-of-speech. Vietnamese is a monosyllabic language; hence, a word may contain more than one to­ken.</p><p>However, the Vietnamese word segmentation module is not trained for question domain. There are question phrases, which are indicative of the question categories such as <i>"phâi không", </i>tagged as multiple <i>TokenVn </i>annotations. In this mod­ule we identify those phrases and mark them as single annotations with corresponding feature <i>"question-word" </i>and its semantic categories such as <i>HowWhy</i><i>cause </i>| <i>method, </i><i>YesNo</i><i>true or false,</i><i>What</i><i>something</i>,    <i>Wheï</i><i>lUme </i>| date,     <i>Where</i><i>iocation,</i> <i>Manynumber</i>, <i>Whoperson.</i><i> </i>In fact, this information will be used in creating rules in the semantic analysis module at a later stage.</p><page local="3" global="408"/><figure caption="Figure 1: Architecture of our question answering system."></figure><p>In addition, we marked phrases that refer to comparing-phrases (such as <i>"lôn höngreater </i><i>than </i><i>" "nhô hön hoäc bange</i><i>s than or equal to" </i>...) or special-words (for example: abbreviation of some words on special-domain) by single <i>TokenVn </i>annota­tions.</p></subsection><subsection number="3.2" title="Syntactic analysis"><p>This module is responsible for identifying noun phrases and the relations between noun phrases. The differ­ent modules communicate through the annotations, for example, this module uses the <i>TokenVn </i>annotations, which is the result of the preprocessing module.</p><p>Concepts and entities are normally expressed in noun phrases. Therefore, it is important that we can reliably detect noun phrases in order to generate the <i>query-tuple. </i>We use <i>JAPE </i>grammars to specify patterns over annotations. When a noun phrase is matched, an an­notation <i>NounPhrase </i>is created to mark up the noun phrase. In addition, its <i>type </i>feature is used to identify the concept and entity that is contained in the noun phrase using the following heuristic:</p><p>If the noun phrase contains a single noun (not in­cluding numeral nouns) and does not contain a proper noun, it contains a <i>concept. </i>If the noun phrase contains a proper noun or contains at least three single nouns, it contains an <i>entity. </i>Otherwise, concepts and entities are determined using a manual dictionary. In this step, a manual dictionary is built for describing concepts and their corresponding synonyms in the Ontology.</p><p>In addition, question-phrases are detected by using noun phrases and question-words identified by the pre­processing module. <i>QUTerm </i>or <i>QU-E-L-MC </i>annota­tions are generated to cover question-phrases with corresponding <i>category </i>feature which gives information about question categories.</p><p>The next step is to identify <i>relation</i>s between noun phrases or noun phrases and question-phrases. When a phrase is matched by one of the relation patterns, an annotation <i>Relation </i>is created to markup the relation.</p><p>For example, with the following question: <i>"liêt kê tât câ câc sinh viên cô quêquân ô Hà Nôi?" "list all students whose hometown is Hanoi?"</i></p><p>The phrase <i>"cô quê quân ô</i><i>have hometown </i><i>of</i><i> </i><i>" </i>is the relation phrase linking the question-phrase <i>"liêt kê tât câ câc sinh viên</i><i>iist ail students </i><i>" </i>and the noun-phrase <i>"Hà NÔi</i><i>Hanoi </i><i>"</i><i>.</i><i></i></p></subsection><subsection number="3.3" title="Semantic analysis module"><p>The semantic analysis module identifies the query-tuples to generate the intermediate representation of the input question using the annotations generated by the previous modules. We will present a systematic knowl­edge acquisition approach by building a SCRDR KB of rules in the next section.</p></subsection></section><section number="4" title="Ripple Down Rules for Question Analysis"><p>Unlike existing approaches for question analysis for En­glish (Lopez et al., 2007) and Vietnamese (Nguyen et al., 2009) where manual rules are created in an ad-hoc manner, we will describe a language independent ap­proach to analyze natural language questions by apply­ing Ripple Down Rules methodology to acquire rules incrementally. Rules are structured in an exception-structure and new rules are only added to correct errors of existing rules.</p><page local="4" global="409"/><p>A SCRDR knowledge base is built to identify the question structure and to produce the query-tuples as the intermediate representation. Figure 2 shows the GUI of our natural language question analyzer. We will first describe the intermediate representation used in our approach, and then propose a rule language for ex­tracting this intermediate representation for a given in­put question.</p><subsection number="4.1" title="Intermediate Representation of an input question"><p>Aqualog (Lopez et al., 2007) performs semantic and syntactic analysis of the input English question through the use of processing resources provided by GATE (Cunningham et al., 2002). When a question is asked, the task of the question analysis component is to trans­fer the natural language question to a Query-Triple with the following format (generic term, relation, second term). Through the use of JAPE grammars in GATE, AquaLog identifies terms and their relationship. Fol­lowing VnQAS (Nguyen et al., 2009), the intermediate representation used in our approach is more complex aiming to cover a wider variety of question types. It consists of a <i>question-structure </i>and one or more <i>query-tuple </i>in the following format:</p><p><i>(question-structure, question-class, Term\, Rela­tion, Term</i><i>2, </i><i>Term?,)</i></p><p>where <i>Terml </i>represents a concept (object class), <i>Term2 </i>and <i>Term?, </i>if exist, represent entities (ob­jects), <i>Relation </i>(property) is a semantic constraint be­tween terms in the question. This representation is meant to capture the semantic of the question.</p><p>Simple questions only have one <i>query-tuple </i>and its <i>question-structure </i>is the query-tuple's question-structure. More complex questions such as compos­ite questions have several sub-questions, each sub­question is represented by a separate <i>query-tuple, </i>and the <i>question-structure </i>captures this composition at­tribute. Composite questions such as:</p><p><i>"danh sâch tât câ câc sinh viên cüa khoa công nghê thông tin mà cô quê quân ô Hà Nôi?"</i></p><p><i>"list all students in the Faculty of Information Tech­nology whose hometown is Hanoi?"</i></p><p>has question structure of type <i>And </i>with two query-tuples where <b>? </b>represents a missing element: <i>( Unkn-Rel, List, sinh viênstudent, </i><i>? </i><i>, khoa công nghê thông</i> <i>tin</i><i>Faculty </i><i>of</i><i> Information Technology </i><i>, </i><i>?</i><i> </i><i>) </i>and <i>( Normal</i> <i>, List, sinh viên</i><i>student</i><i>, cô quê quân </i><i>has hometown</i><i>, Hà</i> <i>NÔi</i><i>Hanoi </i><i>, </i><i>?</i><i> </i><i>)</i><i>.</i></p><p>This representation is chosen so that it can repre­sent a richer set of question types. Therefore, some terms or relation in the tuple can be missing. Exist­ing noun phrase annotations and relation annotations are potential candidates for terms and relations re­spectively. Following VnQAS (Nguyen et al., 2009), we define the following question structures: <i>Normal, UnknTerm, UnknRel, Definition, Compare, ThreeTerm, Clause, Combine, And, Or, Affirm, Affirm_3Term, Af-</i> <i>firm_MoreTuples </i>and question categories: <i>HowWhy, YesNo, What, When, Where, Who, Many, ManyClass, List </i>and <i>Entity.</i></p></subsection><subsection number="4.2" title="Rule language"><p>A rule is composed of a condition part and a conclu­sion part. A condition is a regular expression pattern over annotations using JAPE grammar in GATE (Cun­ningham et al., 2002). It can also post new annotations over matched phrases of the pattern's sub-components. The following example of a pattern shows the posting an annotation over the matched phrase:</p><doubt alpha="62.9" length="116" tooSmall="False" monospace="0.0">( ( {TokenVn.string == "liêt kêlist"} | {TokenVn.string == "chî rashOTO"} ) {NounPhrase.type == Concept} ) : QU_LIST</doubt><p>This pattern would catch phrases starting with a To­kenVn annotation covering either the word "liêt kêlist" or the word "chî rashow", followed by a NounPhrase which must have feature type equal to Concept. When applying this pattern on a text fragment, QU_LIST an­notations would be posted over phrases matching this pattern. As annotations have feature value pairs, we can impose constraints on annotations in the pattern by re­quiring that a feature of an annotation must have a par­ticular value.</p><p>The rule's conclusion contains the question structure and the tuples corresponding to the intermediate repre­sentation where each element in the tuple is specified by a newly posted annotations from matching the rule's condition in the following order:</p><p>All newly posted annotations have the same prefix RDR and the rule index so that a rule can refer to anno­tations of its parent rules. Examples of rules and how rules are created and stored in exception structure will be explained in details in the next section.</p><p>Given a new input question, a rule's condition is con­sidered satisfied if the whole input question is matched by the condition pattern. The conclusion of the fired rule outputs the intermediate representation of the in­put question.</p><p>To create rules for capturing structures of ques­tions, we use patterns over annotations such as To-kenVn, NounPhrase, Relation, annotations capturing question-phrases like QUTerm, QU-E-L-MC (Entity, List, ManyClass). . . and their features.</p></subsection><subsection number="4.3" title="Knowledge Acquisition Process"><p>The following examples show how the knowledge base building process works. When we encountered the question:</p><p><i>"trudng dai hoc Công Nghê cô bao nhiêu sinh viên?" ("how many students are there in the College ofTechnology?")</i></p><p>[NounPhrase trüöng dai hoc Công Nghêthe <i>College </i><i>of</i><i> Technology </i>NounPhrase][Has cohas    Has]    [QU-E-L-MC    bao    nhiêu sinh</p><p>viênhow <i>many students </i>QU-E-L-MC]<page local="5" global="410"/></p><p>Figure 2: Question Analysis module to create the intermediate representation of question <i>"trudng dai hoc Công Nghê cô bao nhiêu sinh viên?"("how many students are there in the College of Technology?").</i></p><p>Supposed we start with an empty knowledge base, the fired rule is default rule that gives empty conclu­sion. This can be corrected by adding the following rule to the knowledge base:</p><doubt alpha="55.6" length="9" tooSmall="False" monospace="0.0">Rule: R10</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">(</doubt><doubt alpha="65.9" length="88" tooSmall="False" monospace="0.0">({NounPhrase}):NounPhrase ({Have}l{Has}l{Preposition}) ({QU-E-L -MC}):QUelmc ({QUTerm})?</doubt><doubt alpha="51.0" length="51" tooSmall="False" monospace="0.0">) : left ---- :left.RDR10_= {category1 = "UnknRel"}</doubt><doubt alpha="65.7" length="35" tooSmall="False" monospace="0.0">, :NounPhrase.RDR10_NounPhrase = {}</doubt><doubt alpha="55.6" length="27" tooSmall="False" monospace="0.0">, :QUelmc.RDR10_QUelmc = {}</doubt><doubt alpha="66.7" length="132" tooSmall="False" monospace="0.0">Conclusion:question-structure ofUnknReland tuple( RDR10_.categoryl , RDR10_QUelmc.QU-E-L-MC.category,     RDR10_QUelmc     ,     ? ,</doubt><p><i>RDR10_NounPhrase , ? )</i>.</p><p>If the condition of rule <b>R10 </b>matches the whole in­put question, a new annotation RDR10_ will be created covering the whole input question and new annotations RDR10_NounPhrase and RDR10_QUelmc will be cre­ated to cover sub-phrases of the input question.</p><p>If rule <b>R10 </b>is fired, the matched input question is deemed to have a query-tuple with question-structure taking the value of <i>category1 </i>feature of RDR10_ an­notation, question-class taking the value of <i>category </i>feature of QU-E-L-MC annotation co-covering the same span as RDR10_QUelmc annotation, <i>Terml </i>is the string covered by RDR10_QUelmc, <i>Term2 </i>is the string covered by RDR10_NounPhrase while <i>Term?</i><i> </i>and <i>Relation </i>are unknown.</p><p>When we encounter the question:</p><p><i>"trudng dai hoc Công Nghê cô bao nhiêu sinh viên là Nguyen Quôc Bat?" ("How many students named Nguyen Quoc Dat are there in the College ofTechnol-</i></p><doubt alpha="50.0" length="6" tooSmall="False" monospace="0.0">ogy?")</doubt><p>[RDR10_ <i>trudng dai hoc Công Nghê cô bao nhiêu sinh viên </i>RDR10_] [Are làAre Are] [NounPhrase Nguyên Quôc Dat^guyen <i>Quoc Dat </i>NounPhrase]</p><p>Rule <b>R10 </b>is the fired rule but gives the wrong con­clusion of question-structure of <i>UnknRel </i>and tuple <i>( UnknRel , ManyClass , sinh viên</i><i>student </i><i>, ? , trudng dai</i> <i>hoc Công </i><i>Nghêthe College </i><i>of</i><i> Technology </i><i>, ?</i><i> )</i>. The fol­lowing exception rule was added to knowledge base to correct that: <b>Rule: R38</b> , :NounPhrase.RDR38_NounPhrase = {} <b>Conclusion: </b>question-structure of <i>ThreeTerm </i>and tuple <i>( RDR38_.category1 , RDR10_QUelmc.QU-</i> <i>E-L-MC.</i><i>category , RDR10_QUelmc , ? , RDR10_NounPhrase , RDR38_NounPhrase )</i>.</p><doubt alpha="56.4" length="101" tooSmall="False" monospace="0.0">{RDR10_} ({Are}l{Is}) ({NounPhrase}):NounPhrase ):left ---- :left.RDR38_ = {category1 = "Three-Term"}</doubt><page local="6" global="411"/><p>Using rule <b>R38, </b>the output of the input question is question-structure of <i>ThreeTerm </i>and tuple <i>( Three-Term , ManyClass , sinh viên</i><i>student </i><i>, ? , trudng dai hoc Công </i><i>Nghêthe College of Technology </i><i>, Nguyen Quôc</i></p><p><i>BatNguyen Quoc Dat </i><i>)</i></p><p>With the question <i>"quê quân cüa nhüng sinh viên nào là Hà Nôi?" ("which students have hometown of</i></p><doubt alpha="62.5" length="8" tooSmall="False" monospace="0.0">Hanoi?")</doubt><p>[RDR10_ [RDR10_NounPhrase <i>quê quânhometown</i>RDR10_NounPhrase] [Preposition <i>cüa</i><i>of </i>Prepo­sition] [RDR10_QUelmc <i>nhüng sinh viên nào</i><i>which students </i>RDR10_QUelmc] RDR10_][Are <i>là</i><i>are   </i>Are]   [RDR38_NounPhrase   <i>Hà Nôi</i><i>Hanoi</i></p><p>RDR38_NounPhrase]</p><p>it will be satisfied by rule <b>R38. </b>But rule <b>R38 </b>gives the wrong conclusion of question-structure of <i>Three-Term </i>and tuple <i>( ThreeTerm , Entity , sinh viên</i><i>student </i><i>, ? , quê quân</i><i>hometown </i><i>, Hà Nôi</i><i>Hanoi </i><i>) </i>because <i>quê </i><i>quânhometown </i>is a relation for linking <i>sinh </i><i>viênstudent </i>and <i>Hà Nôi</i><i>Hanoi. </i>We can add a following exception rule <b>R76 </b>to correct the conclusion by using constrains via rule condition:</p><doubt alpha="55.6" length="9" tooSmall="False" monospace="0.0">Rule: R76</doubt><doubt alpha="46.7" length="15" tooSmall="False" monospace="0.0">({RDR38_}):left</doubt><p>---- :left.RDR76_ = {category1 = "Normal"} <b>Condition:    </b>RDR10_NounPhrase.hasAnno == NounPhrase.type == Concept</p><p><b>Conclusion: </b>question-structure of <i>Normal </i>and tu­ple <i>( RDR76_.category1 , RDR10_QUelmc.QU-E-L-MC.category, RDR10_QUelmc, RDR10_NounPhrase</i></p><doubt alpha="54.2" length="24" tooSmall="False" monospace="0.0">, RDR38_NounPhrase , ? )</doubt><p>The condition of rule <b>R76 </b>matches a RDR10_NounPhrase annotation that has a Noun-Phrase annotation covering their substring with <i>Concept </i>as its <i>type </i>feature. The extra annotation constrain <i>hasAnno </i>requires that the text covered by the annotation must contain the specified annotation. With the rule <b>R76, </b>we have the correct output containing the question-structure of <i>Normal </i>and tuple <i>( Normal , Entity , sinh </i><i>viênstudent </i><i>, quê </i><i>quânhometown </i><i>, Hà</i><i>NôiHanoi </i><i>, ? )</i>.</p></subsection></section><section number="5" title="Experiments"><p>We experiment our system for both Vietnamese and En­glish using the same intermediate representation.</p><subsection number="5.1" title="Question Analysis for Vietnamese"><p>For this experiment, we build a knowledge base of 92 rules from a corpus containing 400 questions and eval­uate its quality on an unseen corpus of 102 questions in the same domain of college (university). The corpus of 400 questions were generated based on a seed corpus of 115 questions. Table 1 shows the number of excep­tion rules in each layer where every rule in layer <i>n </i>is an exception rule of a rule in layer <i>n — </i>1. The only rule that is not an exception rule, is the default rule in layer 0. This indicates that the exception structure is indeed present and even extends to level 4.</p><p>SCRDR KB.</p><p>In our experiment, we implemented the question analysis component of VnQAS (Nguyen et al., 2009) on the same corpus as in building our knowledge base. Table 2 gives the number of correctly analyzed ques­tions of our system and system of (Nguyen et al., 2009) respectively where our system performs slightly better.</p><p>Our method took one expert about 13 hours to build a KB based on the training corpus. However, most of the time was spent in looking at questions to determine if they belong to the structure of interest and which phrases in the sentence need to be extracted for the in­termediate representation. The actual time required to create 92 rules by one expert is only about 5 hours in total. In contrast, implementing question analysis com­ponent of VnQAS (Nguyen et al., 2009) took about 75 hours for creating rules in an ad-hoc manner. Anecdo­tal account indicates that the cognitive load in creating rules in our approach is much less compared to that in VnQAS (Nguyen et al., 2009) as in our case, we do not have to consider other rules when crafting a new rule.</p><p>Table 3 presents the source of error for the 14 ques­tions that our system incorrectly extract. It clearly shows that most errors come from unexpected struc­tures. This could be easily rectified by adding more ex­ception rules to the current knowledge base, especially when we have a bigger training set that contain a larger variety of question structure types.</p></subsection><subsection number="5.2" title="Question Analysis for English"><p>For the experiment in English, we take 170 English question examples of AquaLog's corpus. Using our approach, we built a knowledge base of 59 rules including the default one.<page local="7" global="412"/> It took 7 hours to build the knowledge base, which includes 3 hours of actual time to create all rules. The table 4 shows the numbers of rules in English knowledge base layers.</p><footnote>http://technologies.kmi.open.ac.uk/aqualog/examples.html</footnote><table caption="Table 1: Number of exception rules in layers in our" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Layer</p></td><td class="cell"><p>Number of rules</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p>26</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2</p></td><td class="cell"><p>41</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3</p></td><td class="cell"><p>20</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>4</p></td><td class="cell"><p>4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 2: Number of correctly analyzed questions." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Type</p></td><td class="cell"><p>Number of</p></td><td class="cell"><p>Percent</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>questions</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Our system</p></td><td class="cell"><p>88</p></td><td class="cell"><p>86.3%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Question analysis compo-</p></td><td class="cell"><p>83</p></td><td class="cell"><p>81.4%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>nent of (Nguyen et al., 2009)</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Reason</p></td><td class="cell"><p>Number of</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>questions</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Unknown structures of questions</p></td><td class="cell"><p>12</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Word segmentation was not trained</p></td><td class="cell"><p>2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>for question-domain</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Table 3: Error results.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 4: Number of exception rules in layers in our En­glish SCRDR KB."></table><p>As the intermediate representation of our system is different to Aqualog and there is no common test set available, it is impossible to directly compare our ap­proach with Aqualog on the English domain. However, this experiment is indicative of the ability in using our system to quickly build a new knowledge base for a new domain and a new language.</p></subsection></section><section number="6" title="Conclusion"><p>We believe our approach is important especially for under-resourced languages where annotated data is not available. Our approach could be combined nicely with the process of annotating corpus where on top of as­signing a label or a representation to a question, the experts just have to add one more rule to justify their decision using our system. Incrementally, an annotated corpus and a rule-based system can be obtained simul­taneously.</p><p>The structured data used in the evaluation falls into the category of querying database or ontology but the problem of question analysis we tackle go beyond that, as it is a process that happens before the querying pro­cess. It can be applied to question answering in open domain against text corpora as long as the technique requires an analysis to turn the input question to an ex­plicit representation of some sort.</p><p>In this paper, we introduced a language independent approach for systematically acquiring rules for convert­ing a natural language question into an intermediate representation in a question answering system. Experi­mental results of our system on a wide range of ques­tions are promising with accuracy of 86.3% for the Vietnamese corpus. Notably, the time it takes to get the system up to this performance is much less compared to previous works.</p><p>In the future, we will extend our system to employ a <i>near match </i>mechanism to improve the generalization capability of existing rules in the knowledge base and to assist the rule creation process.</p></section><section title="Acknowledgements"><p>This work is partially supported by the Research Grant from Vietnam National University, Hanoi No.</p><p>QG.10.23.</p><p>The authors would like to acknowledge Vietnam Na­tional Foundation for Science and Technology Devel­opment (NAFOSTED) for their financial support to present the work at the conference.</p><table caption="Table 4: Number of exception rules in layers in our English SCRDR KB." class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Layer</p></td><td class="cell"><p>Number of rules</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p>9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2</p></td><td class="cell"><p>13</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3</p></td><td class="cell"><p>20</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>4</p></td><td class="cell"><p>11</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>5</p></td><td class="cell"><p>5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><references><p>L. Androutsopoulos. 1995. Natural language inter­faces to databases - an introduction. <i>Nat. Lang. Eng., </i>1:29-81.</p><p>P. Compton and R. Jansen. 1990. A philosophical basis for knowledge acquisition. <i>Knowl. Acquis., </i>2(3):241-257.</p><p>Hammish  Cunningham,  Diana  Maynard, Kalina</p><p>Bontcheva, and Valentin Tablan. 2002. GATE: A</p><p>Framework and Graphical Development Environ­ment for Robust NLP Tools and Applications. In <i>Proc.</i><i> ofACL, </i>pages 168-175.</p><p>Danica Damljanovic, Valentin Tablan, and Kalina Bontcheva. 2008. A text-based query interface to owl ontologies. In <i>Proc. ofLREC, </i>pages 205-212.</p><p>L. Hirschman and R. Gaizauskas. 2001. Natural lan­guage question answering: the view from here. <i>Nat.</i></p><p><i>Lang. Eng., </i>7(4):275-300.</p><p>Vanessa Lopez, Victoria Uren, Enrico Motta, and Michele Pasin. 2007. Aqualog: An ontology-driven question answering system for organizational seman­tic intranets. <i>Web Semant., </i>5(2):72-105.</p><p>Anh Kim Nguyen and Huong Thanh Le. 2008. Nat­ural language interface construction using semantic grammars. In <i>Proc. ofPRICAI, </i>pages 728-739.</p><p>Dai Quoc Nguyen, Dat Quoc Nguyen, and Son Bao Pham. 2009. A vietnamese question answering sys­tem. In <i>Proc. ofKSE, </i>pages 26-32.</p><p>Dat Quoc Nguyen, Dai Quoc Nguyen, Son Bao Pham, and Dang Duc Pham. 2011. Ripple down rules for part-of-speech tagging. In <i>Proc. ofCICLing, </i>pages 190-201.</p><p>Son Bao Pham and Achim Hoffmann. 2006. Efficient knowledge acquisition for extracting temporal rela­tions. In <i>Proc. ECAI, </i>pages 521-525.</p><p>Dang Duc Pham, Giang Binh Tran, and Son Bao Pham. 2009. A hybrid approach to vietnamese word seg­mentation using part of speech tags. In <i>Proc. ofKSE, </i>pages 154-161.</p><p>Debbie Richards. 2009. Two decades of ripple down rules research. <i>Knowl. Eng. Rev., </i>24(2):159-184.</p></references></body></article>