<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="413"/><title>A Semi-Automatic, Iterative Method for Creating a Domain-Specific Treebank</title><pubinfo>Proceedings of Recent Advances in Natural Language Processing,pages 413^-19, Hissar, Bulgaria, 12-14 September 2011.</pubinfo><author surname="Dima" givenname="Corina"><org  name="University of Tubingen" country="Germany" city="Tubingen"/></author><author surname="Hinrichs" givenname="Erhard"><org  name="University of Tubingen" country="Germany" city="Tubingen"/></author></firstpageheader><frontmatter><p><b>A Semi-Automatic, Iterative Method for Creating a Domain-Specific</b></p><p><b>Treebank</b></p><p><b>Corina Dima      Erhard Hinrichs</b></p><p>Department of Linguistics University of Tübingen</p><p>corina.dima, erhard.hinrichs@uni-tuebingen.de</p></frontmatter><abstract>In this paper we present the development process of NLP-QT, a question treebank that will be used for data-driven parsing in the context of a domain-specific QA sys­tem for querying NLP resource metadata. We motivate the need to build NLP-QT as a resource in its own right, by com­paring the Penn Treebank-style annotation scheme used for QuestionBank (Judge et al., 2006) with the modified NP annota­tion for the Penn Treebank introduced by Vadas and Curran (2007). We argue that this modified annotation scheme provides a better interface representation for seman­tic interpretation and show how it can be incorporated into the NLP-QT resource, without significant loss in parser perfor­mance. The parsing experiments reported in the paper confirm the feasibility of an iter­ative, semi-automatic construction of the NLP-QT resource similar to the approach taken for QuestionBank. At the same time, we propose to improve the iterative re­finement technique used for QuestionBank by adopting Hwa (2001)'s heuristics for selecting additional material to be hand-corrected and added to the data set at each iteration. </abstract></header><body><section number="1" title="Introduction"><p>Question-Answering (QA) systems have a long history in the field of natural language process­ing. In the 1970s and 1980s QA systems fo­cused on natural language interfaces to domain-specific data bases or expert systems. Such systems typically used hand-crafted, rule-based front ends for parsing and semantic interpretation. With the increased availability of large-scale textual re­sources, QA systems more recently have focused on domain-independent broad-coverage informa­tion retrieval applications that typically employ more shallow processing techniques for question analysis and answer matching.</p><p>The intended application for the research re­ported in the present paper is more in the tradi­tion of the earlier, domain-specific QA systems in that it aims to provide a natural language front-end to large repositories of metadata about language tools and resources that are made available by the CLARIN<footnote anchor="1"/> project. However, instead of relying on a parser with hand-crafted grammar rules, it em­ploys a robust data-driven parser that requires an­notated training data in the form of a treebank.</p><p>Since the natural language front end for the in­tended QA system is English, the simplest solu­tion would be to use a statistical parser such as the Berkeley (Petrov and Klein, 2007) or Stanford (Klein and Manning, 2003) parser with an existing language model obtained from the Penn Treebank (Marcus et al., 1993). However, it is well known that parser performance drops when analyzing text from domains other than that represented in the training data (Sekine, 1997; Gildea, 2001). In par­ticular, Judge et al. (2006) have shown that lan­guage models obtained from the Penn Treebank perform far worse on questions than on their orig­inal test data. The Bikel (2004) parser they employ has an F-Score of 82.97 when tested on Section 23 of the Penn-II Treebank and an F-Score of 78.77 when tested on the 4000 questions in Question-Bank. Judge et al. (2006) attribute this loss of performance to two factors:<page local="2" global="414"/> (i) in the genre of news­paper texts, which the Penn Treebank is based on, questions are not a high frequency syntactic con­struction, and (ii) if wh-type constructions occur at all in the Penn Treebank, they predominantly involve relative clause constructions or indirect questions, but not unembedded questions. There­fore, a parser trained on Penn Treebank data, rou­tinely misanalyses unembedded questions as these other two construction types. In fact, it was this poor parser performance that led Judge et al. to create QuestionBank, a special-purpose treebank based on SemEval data sets for Question Answer­ing (QA).The data include the SemEval QA data from 1999-2001, part of the 2003 set (2000 ques­tions), and another 2000 questions provided by the Cognitive Computation Group at the University of Illinois, which were also test data for develop­ing QA systems. Training a statistical parser on QuestionBank data, possibly in combination with Penn Treebank data, therefore seems to be an at­tractive alternative. In fact, this is precisely how Judge et al. train their parser. However, for rea­sons explained in more detail in sections 2 and 3, we will adopt annotation guidelines for questions that differ from the Penn Treebank-style annota­tion used in QuestionBank. Rather, we will follow a more hierarchical annotation style for NPs that has been proposed by Vadas and Curran (2007) and that provides an easier interface for semantic interpretation. Section 3 will introduce the Vadas and Curran (2007) annotation style and will moti­vate why it is appropriate for the QA system envis­aged here. Section 4 will present a set of parsing experiments for the Berkeley parser trained on dif­ferent combinations of treebank data discussed in sections 2 and 3. The final section summarizes the main results of this paper and discusses directions for future research.</p><footnote label="1">CLARIN project - http://www.clarin.eu</footnote></section><section number="2" title="Data Collection for Querying NLP Resource Metadata"><p>One of the main reasons to create a new data set of questions and not use some already existing set has to do with the specific subject domain of the QA system to be developed. All the questions should concern particular pieces of information associ­ated with language resources or with different ap­plication domains of natural language processing. In order to obtain a realistic data set of this sort, we harvested the questions from mailing lists like</p><p>LinguistList<footnote anchor="2"/> and Corpora List<footnote anchor="3"/>, as well as from the Stack Overflow<footnote anchor="4"/> questions tagged with "nlp".</p><p>The mailing lists have a history of 20 years and have a lot of extra content other than user queries. Therefore, all the posts had to be browsed through in order to manually extract only the relevant ques­tions from the whole post. For example, infor­mation about the person asking the question was deleted from the original posts, since such infor­mation is not relevant for a QA system. Spelling and grammar errors were then removed from the extracted questions. A number of 2500 questions were harvested until the moment of writing, but the goal is to gather a 10.000 questions corpus that should provide enough training and testing data when converted into a treebank.</p><p>The data below provide some typical examples that have been collected from the three sources:</p><p>(1) Where can I find a corpus of German newspapers from the 17th century until the 1950s?</p><p>(2) What good introductory books on the subject of natural language processing, parsing and tagging are</p><doubt alpha="83.3" length="6" tooSmall="False" monospace="0.0">there?</doubt><p>(3) Where can I find the Orleans corpus of spoken French (created by Michel Blanc and Patricia Biggs)?</p><p>(4) Where can I find a parallel corpus of translations in English, French, German and Italian, ideally containing news stories?</p><p>(5) Where can I find a free or available English tagger other than Brill's tagger?</p><p>Apart from the more restricted subject domain, the NLP Resource Metadata Questions signifi­cantly differ from the SemEval data used in Ques-tionBank in at least two other respects:</p><p>• The average length of the SemEval questions in QuestionBank is 47.58 characters and 9.45 words, whereas the NLP Questions average 81.17 characters and 12.88 words.</p><p>• Moreover, the distribution of questions types is quite different in the two cases. The Se-mEval data set used for QuestionBank is intended to query encyclopedic knowledge from sources such as Wikipedia. This means that the questions essentially include all pos­sible question words such as <i>who, what, which, where, when, why, how, </i>etc. When</p><footnote label="2">LinguistList - http://linguistlist.org/  3 Corpora List - http://www.hit.uib.no/corpora/  4 Stack Overflow - http://stackoverflow.com/</footnote><page local="3" global="415"/><p>Figure 1: Comparing annotations for the compound noun <i>second language acquisition materials: </i>Penn­style annotation on the left, Vadas and Curran (2007) style annotation on the right</p><p>Table 1: Distribution of question types in the two datasources; <i>How * </i>stands for questions like <i>how many, how much, how far, how long </i>etc.</p><p>querying NLP resource metadata, the empha­sis is to a large extent on <i>where </i>and <i>is there </i>questions; the percentage for each type of question in the two datasources is showed in</p><p>Table 1.</p></section><section number="3" title="Comparing the Annotation of Base NPs"><p>There is yet another property of both Question-Bank and the Penn Treebank that limits its use­fulness for the QA application considered here. This concerns the flat-structure annotation style for noun phrases adopted in both resources. For example, in the question <i>Where can I find a Ger­man corpus containing second language acquisi-</i> <i>tion materials?</i><i> </i>the compound noun <i>second lan­guage acquisition materials </i>would be annotated in these resources as a single flat NP, as shown in the left column of Figure 1. Such a flat annotation does not provide sufficient information about the scope of each member of the compound. It is pre­cisely this type of shortcoming that led Vadas and Curran (2007) to revise the Penn Treebank anno­tation style for NPs along the following lines:</p><p>• If the intended scope of a base NP leads to a strictly right-branching structure, then the Penn Treebank annotation remains un­changed.</p><p>• If the intended scope is partially or com­pletely left-branching, then an extra node is introduced into the tree for each left-branching structure. The label of this node is either NML or JJP, depending on the lexi­cal head of the local tree (noun or adjective, respectively).</p><p>The resulting annotation for the compound noun <i>second language acquisition materials </i>is shown in the right column of Figure 1.</p><p>From the point of view of semantic interpreta­tion, the more contoured Vadas and Curran (2007) annotation style is to be preferred since it reflects the type of answer that is required, namely <i>materi­als for second language acquisition, </i>but not for ex­ample <i>acquisition materials for second language, </i>or <i>the second (batch) oflanguage acquisition ma­terials.</i></p><p>It is precisely for this reason that we adopt the annotation style of Vadas and Curran (2007) for the NLP Resource Metadata Questions Treebank (henceforth abbreviated as NLP-QT).</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Question Word</p></td><td class="cell"><p>% in QB</p></td><td class="cell"><p>% in NLP-QT</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Are there</p></td><td class="cell"><p>0</p></td><td class="cell"><p>5.45</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>At what</p></td><td class="cell"><p>0.075</p></td><td class="cell"><p>0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>For what</p></td><td class="cell"><p>0.025</p></td><td class="cell"><p>0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>How</p></td><td class="cell"><p>3.75</p></td><td class="cell"><p>0.5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>How *</p></td><td class="cell"><p>8.2</p></td><td class="cell"><p>0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>In what</p></td><td class="cell"><p>0.825</p></td><td class="cell"><p>0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>In which</p></td><td class="cell"><p>0.15</p></td><td class="cell"><p>0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Is there</p></td><td class="cell"><p>0</p></td><td class="cell"><p>16.81</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>On what</p></td><td class="cell"><p>0.075</p></td><td class="cell"><p>0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>On which</p></td><td class="cell"><p>0.075</p></td><td class="cell"><p>0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>What</p></td><td class="cell"><p>57.35</p></td><td class="cell"><p>0.98</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>When</p></td><td class="cell"><p>5</p></td><td class="cell"><p>0.05</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Where</p></td><td class="cell"><p>6.075</p></td><td class="cell"><p>75.07</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Which</p></td><td class="cell"><p>1.925</p></td><td class="cell"><p>0.09</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Who</p></td><td class="cell"><p>11.375</p></td><td class="cell"><p>0.1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Why</p></td><td class="cell"><p>1.2</p></td><td class="cell"><p>0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Other</p></td><td class="cell"><p>3.822</p></td><td class="cell"><p>0.93</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="4" global="416"/></section><section number="4" title="Experimental Results"><p>This section summarizes the set of experiments that we have conducted with the Vadas and Curran (2007) annotation style for NPs and in particular with the NLP-QT data set. We discuss two types of experiments:</p><p>• comparing the performance of the parser us­ing different annotation styles for base NPs,</p><p>• experiments for optimizing the language model of a statistical parser in order to as­sist with the semi-automatic creation of the treebank.</p><p>All the experiments were performed with the Berkeley parser. The results are summarized in Table 2 and Table 3.</p><subsection number="4.1" title="Parsing Results for Different Annotation Styles"><p>Using Bikel (2004)'s parser, Vadas and Curran (2007) report that the parsing results slightly de­crease when the parser is trained on the Penn Tree-bank with the modified annotation style for NPs. As Table 2 shows, we obtain a similar result when testing on section 23 of the Penn Treebank, using the Berkeley parser trained on sections 02-21 of the same treebank: there is minor drop in F-score from 90.43 to 89.96. We also confirm Gildea's finding that testing a parser on test sets from a dif­ferent domain than the training sets results in a sig­nificant loss of performance: when using the same models that we used for the Penn Treebank exper­iments, the average F-score for test data from the Question Bank in a 10-fold cross-validation ex­periment is 79.944 for the model trained on the original Penn Treebank and 77.607 for the model trained on the modified Penn Treebank.</p><p>The above experiments were designed as a base­line for comparing the performance of the parser trained only on Penn Treebank data. But since our primary interest is in parsing questions as accu­rately as possible, we conducted a second set of experiments, summarized in the lower half of Ta­ble 2. Here additional training data from the Ques­tion Bank was added to both the original and the modified Penn Treebank training data. The de­crease in performance caused by adding the Ques-tionBank training data together with the modified NP annotation on section 23 is comparable to the one caused by adding the modified NP annotation alone (a decrease from 90.263 to 90.04, whereas for the original Penn Treebank data the F-score decreased from 90.43 to 89.96), but this slight de­crease is more than offset by the increase in se­mantic information obtained from the Vadas and Curran (2007) annotation for complex base NPs. Even more noteworthy is the big jump in F-score from 77.607 to 92.658 when adding the Question-Bank data to the training data.</p></subsection><subsection number="4.2" title="Semi-automatic Creation of NLP-QT"><p>The creation of a treebank is a time-consuming and expensive task if all the annotation has to be performed manually. It is therefore useful to in­vestigate whether at least parts of the annotation can be performed automatically or by a combina­tion of automatic analysis and manual post edit­ing. To this end, we performed a set of parsing ex­periments, again using the Berkeley parser, where the test data are taken both from the QuestionBank and a seed set of 500 manually annotated ques­tions from the NLP-QT. The results are shown in</p><p>Table 3.</p><p>As in the experiments shown in the previous subsection, the performance with a model trained purely on Penn Treebank data (with NPs annotated in the Vadas and Curran (2007) style) serves as a baseline (the model is called <i>np-wsj </i>in the table). This model is then enriched by first adding anno­tated data from Question Bank and then by adding the manually annotated questions from the NLP-QT. We refer to these models as <i>np-wsjqb </i>and <i>np-wsjqblq-500, </i>respectively. The results are very en­couraging on several dimensions:</p><p>1. overall parsing performance on the test data for both the <i>np-wsjqb </i>and the <i>np-wsjqblq-500 </i>models is very good</p><p>2. adding questions from the NLP-QT yields a desired increase in performance</p><p>3. almost two-thirds of all questions from the test data yield a completely correct parse.</p><p>These three findings together make a semi­automatic construction of the NLP-QT entirely feasible. In fact, we are currently constructing the NLP-QT treebank in this semi-automatic fashion, using the same iterative approach to treebank con­struction adopted for the QuestionBank data by Judge et al. This approach involves iterations of manual post correction of automatically generated<page local="5" global="417"/></p><p>Table 2: Comparison of parser performance when trained on different data sources with different anno­tation styles parses, adding this post-corrected data set to the previously used training material and then retrain­ing the parser with the enlarged data set.</p><p>One question that was not addressed in the ap­proach by Judge et al. concerns the selection of the additional trees that will be manually corrected and then added to the training and test material in the next iteration. As Hwa (2001) has pointed out, this selection process can be critical in minimizing the amount of data that needs to be hand-corrected during grammar induction. She suggests several simple heuristics for ranking the candidate trees, two of which will be considered here. One heuris­tic is based on the often observed fact that, on aver­age, longer sentences are harder to parse correctly than shorter ones. A second, related and some­what more fine-grained variant of the first heuris­tic is based on the number ofconstituents obtained by the automatic parse of a sentence. Since the automatic parse is often at least partially incor­rect, the constituent count of the parser will typ­ically be just an estimate of the actual constituent count and related complexity ofthe sentence. Hwa suggests that when trees are added, the selected trees should match the average constituent count and length profile of the trees that were incorrectly parsed in the previous iteration.</p><p>We adopt Hwa's approach in the construction of the NLP-QT treebank. In order to use it effec­tively, it is necessary to inspect the results of the parser and in particular create an automatic profile of the completely correct versus partially incorrect parses. This type of error analysis is the subject of the next section.</p></subsection><subsection number="4.3" title="Error Analysis"><p>Table 4 summarizes the profiling of the 500 ques­tions from the NLP-QT used in the 10-fold val­idation experiment. On average, 48.59 % of all sentences received an entirely correct parse. The average length in characters and in words as well as the average number of constituents of the cor­rectly parsed sentences differ significantly from the questions where the parse is only partially cor­rect.</p><p>These results provide a sound basis for apply­ing Hwa's selection method: in the next iteration of optimizing the statistical model for the parser, sampling should focus on questions that match as closely as possible the character, word, and con­stituent count of the partially incorrect parse trees.</p><p>In order to get an impression of the kinds of mistakes that are made by the Berkeley parser, we are presenting two partially incorrect parse trees for the sentences in 6 and 7.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Models</p></td><td class="cell"><p>Section 23 of Penn Treebank</p></td><td class="cell"><p>QuestionBank test section</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Prec.</p></td><td class="cell"><p>Recall</p></td><td class="cell"><p>F-score</p></td><td class="cell"><p>Prec.</p></td><td class="cell"><p>Recall</p></td><td class="cell"><p>F-score</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Orig. PTB</p></td><td class="cell"><p>90.480</p></td><td class="cell"><p>90.390</p></td><td class="cell"><p>90.430</p></td><td class="cell"><p>79.285</p></td><td class="cell"><p>80.617</p></td><td class="cell"><p>79.944</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PTB w/ NPs</p></td><td class="cell"><p>90.000</p></td><td class="cell"><p>89.920</p></td><td class="cell"><p>89.960</p></td><td class="cell"><p>77.546</p></td><td class="cell"><p>77.670</p></td><td class="cell"><p>77.607</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Orig. PTB + QB</p></td><td class="cell"><p>90.317</p></td><td class="cell"><p>90.211</p></td><td class="cell"><p>90.263</p></td><td class="cell"><p>93.618</p></td><td class="cell"><p>92.801</p></td><td class="cell"><p>93.207</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PTB w/ NPs + QB</p></td><td class="cell"><p>90.095</p></td><td class="cell"><p>89.985</p></td><td class="cell"><p>90.040</p></td><td class="cell"><p>92.592</p></td><td class="cell"><p>92.725</p></td><td class="cell"><p>92.658</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 3: Parser performance increases when adding hand-corrected question data to the training set" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Models</p></td><td class="cell"><p>Questions test set</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Prec.</p></td><td class="cell"><p>Recall</p></td><td class="cell"><p>F-score</p></td><td class="cell"><p>Exact match</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>np-wsj</p><p>np-wsjqb</p><p>np-wsjqblq_500</p></td><td class="cell"><p>78.525 91.256 92.128</p></td><td class="cell"><p>78.780 91.499 92.186</p></td><td class="cell"><p>78.651 91.375 92.157</p></td><td class="cell"><p>30.231 63.111 64.801</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 4: Average length and constituent count for the correctly/incorrectly parsed questions" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>% of total</p></td><td class="cell"><p>Avg. char. length</p></td><td class="cell"><p>Avg. word length</p></td><td class="cell"><p>Avg. const. no</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Correct</p></td><td class="cell"><p>48.59</p></td><td class="cell"><p>61.55</p></td><td class="cell"><p>11.41</p></td><td class="cell"><p>20.94</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Incorrect</p></td><td class="cell"><p>51.41</p></td><td class="cell"><p>100.96</p></td><td class="cell"><p>17.85</p></td><td class="cell"><p>31.96</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="6" global="418"/><p>The parsing experiments reported in the pa­per confirm the feasibility of an iterative, semi­automatic construction of the NLP-QT resource similar to the approach taken for QuestionBank. At the same time, we propose to improve the iter­ative refinement technique used for QuestionBank by adopting Hwa's heuristics for selecting addi­tional material to be hand-corrected and added to the data set at each iteration.</p><p>(6) Is there any freely available text corpus for Croatian, no smaller than 20k words?</p><p>(7) Where can I find information on chunking French and German texts?</p><p>The trees obtained by the Berkeley parser for these two sentences are shown in Figures 2 and 3, respectively. They exhibit the following typical at­tachment mistakes and misgroupings of conjuncts in a coordination structure:</p><p>The parse tree generated by the Berkeley parser for sentence 6 (Figure 2) contains several errors: two attachment errors (the PP <i>for Croatian </i>is not attached as a post-head modifier to the nominal head <i>text corpus, </i>but rather attached high as a sis­ter of the preceding NP. Likewise, the modifier starting with <i>no smaller ... </i>is treated as an ADJP rather than an NP and is attached as well as a sis­ter of the preceding NP and PP rather than to the complex NP <i>any ... for Croatian </i>in the gold parse. Moreover, the JJP <i>freely available </i>is incorrectly labelled as an ADJP.</p><p>The parse tree for sentence 7 (Figure 3) fails on the correct grouping and labelling of the co­ordinate structure <i>French and German texts. </i>The tagger treats the lexical token <i>chunking </i>as a noun (NN), rather than a gerund (VBG), and the lexical token <i>French </i>as a plural noun (NNS) rather than as an adjective (JJ). The parser then combines these two items into an NP, which is then coordinated with the NP <i>German texts.</i></p><p>By hand correcting parse trees similar to the ones just discussed and by including them in the data set for retraining the parsing model in the next iteration, the performance of the parser on the types of constructions in question will improve and thereby minimize the amount of manual post editing as much as possible.</p></subsection></section><section number="5" title="Conclusion and Future Work"><p>In this paper we have presented the development process of the NLP-QT resource that will be used for data-driven parsing in the context of a domain-specific QA system for querying NLP resource metadata. We have motivated the need to build NLP-QT as a resource in its own right by com­paring the Penn Treebank-style annotation scheme used for QuestionBank with the modified NP an­notation for the Penn Treebank introduced by Vadas and Curran (2007). We have argued that this modified annotation scheme provides a bet­ter interface representation for semantic interpretation and have shown how it can be incorporated into the NLP-QT resource, without significant loss in parser performance.</p><p>Another important aspect in the creation of a treebank how to ensure a consistent and correct annotation of the linguistic material. Automatic error detection techniques that can be used to test the accuracy of the annotation have already been described in works like Kvetoft and Oliva (2002), for the part of speech annotation level, and Dick­inson and Meurers (2005), for the syntactic anno­tation level. In future work on the NLP-QT, we plan to employ such methods in order to identify and to correct inconsistencies in the annotation.</p><figure caption="Figure 2: Incorrect (top) and correct (bottom) parse for example 6"></figure><figure caption="Figure 3: Incorrect (top) and correct (bottom) parse for example 7"></figure></section><references><p>Dan Bikel. 2004. <i>A distributional analysis of a lexical-ized statistical parsing model. </i>Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing (EMNLP 2004).</p><p>Markus Dickinson and W. Detmar Meurers. 2005. <i>Prune Diseased Branches to Get Healthy Trees! How to Find Erroneous Local Trees in a Treebank and Why It Matters. </i>Proceedings of the 4th Work­shop on Treebanks and Linguistic Theories (TLT 2005).</p><p>Daniel Gildea. 2001. <i>Corpus Variation and Parser Performance. </i>Proceedings of the 2001 Conference on Empirical Methods in Natural Language Process­ing (EMNLP 2001), 167-202.</p><p>Rebecca Hwa. 2001. <i>On Minimizing Training Cor­pus for Parser Acquisition. </i>Proceedings of the 2001 workshop on Computational Natural Language Learning - Volume 7.</p><p>John Judge, Aoife Cahill, and Josef van Genabith. 2006. <i>QuestionBank: Creating a Corpus ofParse-Annotated Questions. </i>Proceedings of the 21st Inter­national Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics (Coling-ACL 2006).</p><p>Pavel Kvetoft and Karel Oliva. 2002. <i>Achieving an Almost Correct PoS-Tagged Corpus. </i>Proceedings of the 5th International Conference Text, speech and dialogue (TSD 2002).</p><page local="7" global="419"/><doubt alpha="53.3" length="122" tooSmall="False" monospace="0.0">Is      there     any    freely available  text   corpus     for   Croatian     , no    smaller   than      20k    words ?</doubt><p>VBZ       EX DT        RB JJ NN       NN IN JJ , RB       JJR IN JJ NNS .</p><p>(a) Incorrect parse (b) Correct parse</p><p><u>CWHADVP)</u></p><p>Where WRB find information VB NN chunking French fWHADVP) nking French</p><doubt alpha="83.3" length="6" tooSmall="False" monospace="0.0">can MD</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">dp)</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">I</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">PRP</doubt><doubt alpha="55.6" length="9" tooSmall="False" monospace="0.0">-(SBARQ)-</doubt><doubt alpha="33.3" length="6" tooSmall="False" monospace="0.0">-(Np)-</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">on</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">IN</doubt><doubt alpha="37.5" length="8" tooSmall="False" monospace="0.0">I—(Np)-|</doubt><doubt alpha="83.3" length="6" tooSmall="False" monospace="0.0">NN NNS</doubt><doubt alpha="83.3" length="6" tooSmall="False" monospace="0.0">and CC</doubt><doubt alpha="100.0" length="6" tooSmall="False" monospace="0.0">German</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">JJ</doubt><doubt alpha="100.0" length="5" tooSmall="False" monospace="0.0">texts</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">NNS</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">(Hp</doubt><doubt alpha="33.3" length="6" tooSmall="False" monospace="0.0">:(Vp)-</doubt><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">(NP)</doubt><doubt alpha="55.6" length="9" tooSmall="False" monospace="0.0">-(SBARO)-</doubt><doubt alpha="40.0" length="5" tooSmall="False" monospace="0.0">-(VP&gt;</doubt><doubt alpha="50.0" length="6" tooSmall="False" monospace="0.0">&lt;JJP)-</doubt><doubt alpha="60.0" length="5" tooSmall="False" monospace="0.0">■Çnp&gt;</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">PP</doubt><p>Dan Klein and Christopher D. Manning. 2003. <i>Accu­rate Unlexicalized Parsing. </i>Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 1 (ACL 2003), 423-430.</p><p>Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. <i>Building a Large Annotated Corpus ofEnglish: The Penn Treebank. </i>Computa­tional Linguistics 19, 313-330.</p><p>Slav Petrov and Dan Klein. 2007. <i>Improved Inference for Unlexicalized Parsing. </i>Proceedings of NAACL HLT 2007.</p><p>Satoshi Sekine.  1997.  <i>The Domain Dependence of</i> <i>Parsing.</i><i> </i>Proceedings of the Fifth Conference on Applied Natural Language Processing, 96-102.</p><p>David Vadas and James R. Curran. 2007. <i>Adding Noun Phrase Structure to the Penn Treebank. </i>Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL-07), 240-247.</p></references></body></article>