<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="37"/><title>Automatic Acquisition of Training Data for Statistical Parsers</title><author surname="Howlett" givenname="Susan"><org  name="University of Sydney" country="Australia" city="Sydney"/></author><author surname="Curran" givenname="James R"><org  name="University of Sydney" country="Australia" city="Sydney"/></author></firstpageheader><frontmatter><p><b>Automatic Acquisition of Training Data for Statistical Parsers</b></p><p><b>Susan Howlett </b>and <b>James R. Curran</b></p><p>School of Information Technologies University of Sydney NSW 2006, Australia {showQ556,james}@it.usyd.edu.au</p></frontmatter><abstract>The limitations of existing data sets for train­ing parsers has led to a need for additional data. However, the cost of manually annotat­ing the amount and range of data required is prohibitive. For a number of simple facts like those sought in Question Answering, we com­pile a corpus of sentences extracted from the Web that contain the fact keywords. We use a state-of-the-art parser to parse these sentences, constraining the analysis of the more complex sentences using information from the simpler sentences. This allows us to automatically cre­ate additional annotated sentences which we then use to augment our existing training data. </abstract></header><body><section number="1" title="Introduction"><p>Determining the syntactic structure of sentences is a necessary step in analysing the content of text for a range of language processing tasks such as Ques­tion Answering (Harabagiu et al., 2000) and Ma­chine Translation (Melamed, 2004).</p><p>The structures that a parsing system assigns to sentences are governed by the grammar used. While some parsers make use of hand-crafted grammars, these typically cannot accommodate a wide variety of sentences and increasing coverage incurs signif­icant development costs (Cahill et al., 2008). This has led to interest in automatic acquisition of gram­mars from raw text or automatically annotated data such as the Penn Treebank (Marcus et al., 1993).</p><doubt alpha="50.9" length="53" tooSmall="False" monospace="0.0">e.g. Riezler et al. (2002) and Briscoe et al. (2006),</doubt><p>Automatically-acquired grammars may be clas­sified according to whether the learning algorithm used to estimate the language model is supervised or unsupervised. Supervised algorithms, e.g. Collins (1999) and Charniak (2000), require a large num­ber of sentences already parsed according to the desired formalism, while unsupervised approaches, e.g. Bod (2006) and Seginer (2007), operate directly on raw text. While supervised approaches have gen­erally proven more successful, the need for anno­tated training data is a major bottleneck.</p><p>Although the emergence of the Penn Treebank as a standard resource has been beneficial in parser development and evaluation, parsing performance drops when analysing text from domains other than that represented in the training data (Sekine, 1997; Gildea, 2001). In addition, there is evidence that lan­guage processing performance can still benefit from orders of magnitude more data (e.g. Banko and Brill (2001)). However, the cost of manually annotating the necessary amount of data is prohibitive.</p><p>We investigate a method of automatically creat­ing annotated data to supplement existing training corpora. We constructed a list of facts based on fac­toid questions from the TREC 2004 Question An­swering track (Voorhees, 2004) and the ISI Question Answer Typology (Hovy et al., 2002). For each of these facts, we extracted sentences from Web text that contained all the keywords of the fact. These sentences were then parsed using a state-of-the-art parser (Clark and Curran, 2007).</p><p>By assuming that the same grammatical relations always hold between the keywords, we use the anal­yses of simple sentences to constrain the analysis of more complex sentences expressing the same fact. The constrained parses then form additional training data for the parser. The results here show that parser performance has not been adversely affected; when the scale of the data collection is increased, we ex­pect to see a corresponding increase in performance.</p><page local="2" global="38"/></section><section number="2" title="Background"><p>The grammars used in parsing systems may be clas­sified as either hand-crafted or automatically ac­quired. Hand-crafted grammars, e.g. Riezler et al. (2002) and Briscoe et al. (2006), have the advan­tage of not being dependent on any particular cor­pus, however extending them to unrestricted input is difficult as new rules and their interactions with ex­isting rules are determined by hand, a process which can be prohibitively expensive (Cahill et al., 2008).</p><p>In contrast, the rules in acquired grammars are learned automatically from external resources, typ­ically annotated corpora such as the Penn Treebank (Marcus et al., 1993). Here, the system automat­ically determines all the rules necessary to gener­ate the structures exemplified in the corpus. Due to the automatic nature of this process, the rules deter­mined may not be linguistically motivated, leading to a potentially noisy grammar.</p><p>Systems that learn their model of syntax from an­notated corpora like the Penn Treebank are termed <i>supervised</i><i> </i>systems; in <i>unsupervised</i><i> </i>learning algo­rithms, possible sentence structures are determined directly from raw text. Bod (2006) presents an un-supervised parsing algorithm that out-performs a su­pervised, binarised PCFG and claims that this her­alds the end of purely supervised parsing. However these results and others, e.g. Seginer (2007), are still well below the performance of state-of-the-art su­pervised parsers; Bod explicitly says that the PCFG used is not state-of-the-art, and that binarising the PCFG causes a drop in its performance.</p><p>By extracting the grammar from an annotated cor­pus, the bottleneck is shifted from writing rules to the creation of the corpus. The 1.1 million words of newswire text in the Penn Treebank has certainly been invaluable in the development and evaluation of English parsers, however for supervised parsing to improve further, more data is required, from a larger variety of texts.</p><p>To investigate the importance of training data in language processing tasks, Banko and Brill (2001) consider the problem of confusion set disambigua­tion, where the task is to determine which of a set of words (e.g. <i>{to, too, two}) </i>is correct in a partic­ular context. Note that this is a problem for which large amounts of training data can be constructed extremely cheaply. They compare four different ma­chine learning algorithms and find that with a train­ing corpus of 1 billion words, there is little differ­ence between the algorithms, and performance ap­pears not to be asymptoting. This result suggests that for other language processing tasks, perhaps in­cluding parsing, performance can still be improved by large quantities of additional training data, and that the amount of data is more important than the particular learning algorithm used.</p><p>Quantity of data is not the only consideration, however. It has been shown that there is consider­able variation in the syntactic structures used in dif­ferent genres of text (Biber, 1993), suggesting that a parser trained solely on newswire text will show reduced performance when parsing other genres.</p><p>Evaluating parser performance on the Brown cor­pus, Sekine (1997) found that the best performance was achieved when the grammar was extracted from a corpus of the same domain as the test set, followed by one extracted from an equivalent size corpus con­taining texts from the same class (fiction or non-fiction), while parsers trained on a different class or different domain performed noticeably worse.</p><p>Comparing parsers trained on the Penn Treebank and Brown corpora, Gildea (2001) found that a small amount of training data from the same genre as the test set was more useful than a large amount of un­matched data, and that adding unmatched training data neither helped nor hindered performance.</p><p>To improve parsing performance on non­newswire genres, then, training data for these additional domains are necessary. The Penn Tree-bank project took 8 years and was an expensive exercise (Marcus et al., 1993), so larger and more varied annotated corpora are not likely to be forthcoming. Since unsupervised approaches still under-perform, it is necessary to explore how exist­ing annotated data can be automatically extended, turning supervised into <i>semi-supervised</i><i> </i>approaches.</p></section><section number="3" title="Parsing with CCG"><p>Combinatory Categorial Grammar (CCG; Steedman (2000)) is a mildly context sensitive, lexicalised grammar formalism, where each word in the sen­tence is assigned a <i>lexical</i><i> </i><i>category</i><i> </i>(or <i>supertag),</i><i> </i>either <i>atomic</i><i> </i>or <i>complex,</i><i> </i>which are then combined<page local="3" global="39"/></p><doubt alpha="100.0" length="6" tooSmall="False" monospace="0.0">Mozart</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">was</doubt><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">born</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">in</doubt><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1756</doubt><doubt alpha="50.8" length="59" tooSmall="False" monospace="0.0">NP(S[dcl]\NP)/(S[pss]\NP) S[pss]\NP ((S\NP)\(S\NP)) / NP NP</doubt><doubt alpha="37.5" length="16" tooSmall="False" monospace="0.0">(S\NP)\(S\NP )-&lt;</doubt><doubt alpha="54.5" length="11" tooSmall="False" monospace="0.0">S [pss }\NP</doubt><doubt alpha="66.7" length="9" tooSmall="False" monospace="0.0">S[dcl]\NP</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">&lt;</doubt><doubt alpha="66.7" length="6" tooSmall="False" monospace="0.0">S[dcl]</doubt><p>Figure 1: CCG derivation for the sentence <i>Mozart was born in 1756.</i></p><p>using a small number of generic <i>combinatory</i><i> </i><i>rules.</i></p><p>The possible atomic categories are <i>N</i>, <i>NP</i>, <i>PP</i><i> </i>and <i>S,</i><i> </i>representing nouns, noun phrases, preposi­tional phrases and clauses, respectively. They may carry additional features, e.g. <i>S[dcl] </i>represents a declarative clause. Complex categories are binary structures consisting of two categories and a slash, in the form <i>X</i><i> </i><i>/Y</i><i> </i>or <i>X \Y</i>. These represent con­stituents that when combined with a constituent with category <i>Y</i><i> </i><i>(argument</i><i> </i>category) form a constituent with category <i>X</i><i> </i><i>(result</i><i> </i>category). The forward and backward slashes indicate that the <i>Y </i>is to be found to the right and left of <i>X,</i><i> </i>respectively. Examples of complex categories are <i>NP/N </i>(determiner) and <i>(S\NP)/NP </i>(transitive verb).</p><p>The basic combinatory rules are <i>forward</i><i> </i>and <i>backward</i><i> </i><i>application,</i><i> </i>where complex categories are combined directly with their arguments to form the result. That is, <i>X/Y</i><i> Y = X </i>and <i>Y X\Y = X</i>. Instances of these rules are marked by the symbols &gt; and &lt; respectively. A derivation il­lustrating these rules is given in Figure 1. Here, the word <i>in </i>with category <i>((S</i><i>\NP)\(S\NP))/NP </i>combines with <i>1756, </i>an <i>NP, </i>using forward applica­tion to produce the constituent <i>in 1756 </i>with category <i>(S\NP)\(S\NP).</i><i></i></p><p>In addition, CCG has a number of rules such as <i>forward</i><i> </i><i>composition</i><i> </i>(&gt; <b>B) </b>(X <i>/Y</i><i> </i><i>Y</i><i> </i><i>/Z</i><i> </i><i>=</i><i> </i><i>X/Z</i><i> </i>) and type-raising capabilities (e.g. X <i>==</i><i> </i><i>T/(</i><i> T\X)</i>, represented by &gt; <b>T) </b>that increase the grammar's expressive power from context free to mildly con­text sensitive. These allow more complex analyses where, instead of the verb taking its subject <i>NP</i><i> </i>as an argument, the <i>NP</i><i> </i>takes the verb as an argument. These additional rules mean that the sentence in Fig­ure 1 may have alternative CCG analyses, for exam­ple the one shown in Figure 2. This <i>spurious</i><i> </i><i>ambi-</i> <i>guity</i><i> </i>has been shown not to pose a problem to effi­cient parsing with CCG (Clark and Curran, 2007).</p><p>As each of the combinatory rules is applied, predicate-argument dependencies are created, of the form <i>(hf,</i><i> </i><i>f,</i><i> s, ha, </i><i>l),</i><i> </i>where f is the category that governs the dependency, hf is the word carrying the category f, s specifies which dependency of f is be­ing filled, <i>ha</i><i> </i>is the head word of the argument, and <i>l </i>indicates whether the dependency is local or long­range. For example, in Figure 1, the word <i>in </i>results in the creation of two dependencies, both local:</p><doubt alpha="35.3" length="68" tooSmall="False" monospace="0.0">(in,((S\NP)\(S1\NP))/NP2,1,born,-)(in,((S\NP)\(S1\NP))/NP2,2,1756,-)</doubt><p>The values for <i>s </i>refer back to the subscripts given on the category f. Although the spurious derivations such as those in Figure 2 have a different shape, they result in the creation of the same set of dependen­cies, and are therefore equally valid analyses.</p><p>The predicate-argument dependencies to be formed are governed by variables associated with the categories nested within the category <i>f.</i><i> </i>For example, <i>in </i>is more fully represented by the cat­egory <i>(((Sy\NPz)y\(Syj\NPz)y)x/NPw,)x. </i>These variables represent the heads of the corresponding constituents; the variable X always refers to the word bearing the lexical category, in this case <i>in.</i></p><p>During the derivation, dependencies form be­tween the word filling variable X and the word fill­ing the variable directly associated with the depen­dency. In this example, the first dependency forms between <i>in </i>and the word filling variable Y, and the second forms between <i>in </i>and the word filling <i>W</i>. When <i>in </i>combines with <i>1756 </i>through forward appli­cation, <i>1756 </i>is identified with the argument <i>NPW2,</i><i> </i>causing W to be filled by <i>1756, </i>and so a dependency forms between <i>in </i>and <i>1756.</i></p><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">&gt;</doubt><page local="4" global="40"/><doubt alpha="65.2" length="23" tooSmall="False" monospace="0.0">Mozart was born in 1756</doubt><doubt alpha="47.0" length="66" tooSmall="False" monospace="0.0">NP       (S[del}\NP)/(S[pss}\NP) S[pss}\NP ((S\NP)\(S\NP))/NPNP-&gt;T</doubt><doubt alpha="45.5" length="22" tooSmall="False" monospace="0.0">S/(S\NP) (S\NP)\(S\NP)</doubt><doubt alpha="48.3" length="29" tooSmall="False" monospace="0.0">S/(S[pss\NP ) &gt;BS [pss }\NP &lt;</doubt><doubt alpha="44.4" length="9" tooSmall="False" monospace="0.0">S [del} &gt;</doubt><p>Figure 2: Alternative derivation for the sentence in Figure 1, illustrating CCG's spurious ambiguity.</p></section><section number="4" title="The C&amp;C Parser"><p>The parser we use (Clark and Curran, 2007) is a state-of-the-art parser based on CCG. The grammar is automatically extracted from CCGbank, a conver­sion of the Penn Treebank into CCG (Hockenmaier, 2003). As the training data is ultimately from the Penn Treebank, the grammar is subject to the same difficulties mentioned in Section 2.</p><p>The two main components of this parser are the supertagger, responsible for assigning each word its possible CCG lexical categories, and the parser, which combines the lexical categories to form the full derivation, using the CKY chart parsing algo­rithm as described in Steedman (2000). The su­pertagger and parser are tightly integrated so that, for example, if a spanning analysis is not found, more lexical categories can be requested.</p><p>The third component, the decoder, chooses the most probable of the spanning analyses found by the parser, according to the statistical model used. Clark and Curran (2007) discuss three separate mod­els, one of which involves finding the most probable derivation directly, while the other two involve opti­mising the dependency structure returned. Here we use the first model, called the normal-form model.</p><p>Clark and Curran (2007) evaluate their parser in two different ways. Their main method of evaluation is to compare the parser's predicate-argument de­pendency output against the dependencies in CCG-bank. They calculate labelled and unlabelled preci­sion, recall and F-score for the CCG dependencies plus category accuracy, the percentage of words as­signed the correct lexical category.</p><p>This method of evaluation, however, does not al­low easy comparison with non-CCG systems. Evalu­ating a CCG parser using the traditional metrics of precision, recall and F-score over Penn Treebank bracketings is problematic since CCG derivations, being binary-branching, can have a very different shape from the trees found in the Penn Treebank. This is particularly true of the spurious derivations, which will be heavily penalised even though they are correct analyses that lead to the correct predicate-argument dependencies.</p><p>To address this problem, Clark and Curran (2007) also evaluate their parser against the Briscoe and Carroll (2006) re-annotation of the PARC Depen­dency Bank (DepBank; King et al. (2003)), which contains 700 sentences from section 23 of the Penn Treebank, represented in the form of grammatical relations, e.g. ncsubj (non-clausal subject) and dobj (direct object). To do this, they convert the predicate-argument dependency output of the parser into grammatical relations, a non-trivial, many-to-many mapping. The conversion process puts the C&amp;C parser at a disadvantage, however the its per­formance still rivals that of the RASP parser (Briscoe et al., 2006) that returns DepBank grammatical rela­tions natively. Figure 3 illustrates the grammatical relations obtained from the analysis in Figure 1.</p></section><section number="5" title="Getting Past the Bottleneck"><p>Although some work has aimed at reducing the cost of training the C&amp;C parser (Clark and Curran, 2006), the question remains of whether annotated training data can be obtained for free.</p><p>In their first entry to the TREC Question-Answering task, Brill et al. (2001) reasoned that while finding the correct answer in the given cor­pus may sometimes require sophisticated linguistic or logical processing, a preliminary answer found on the Web can help to identify the final answer in the corpus. They argue that the sheer size of the Web means that an answer can often be found using sim­ple or shallow processing. We exploit the same idea of the redundancy of information on the Web here.</p><page local="5" global="41"/><figure caption="Figure 3: Grammatical relations forMozart was born in 1756."></figure><p>Figure 4: Desired analysis for the more complex sentence, with the analysis in Figure 3 indicated by dashed lines.</p><p>A parsing model trained on existing data can al­ready confidently parse a simple sentence such as <i>Mozart was born in 1756 </i>(Figure 3). Given the size of the Web, many such sentences should be eas­ily found. A longer sentence, such as <i>Wolfgang Amadeus Mozart (baptized Johannes Chrysosto-mus Wolfgangus Theophilus) was born in Salzburg in 1756, the second survivor out of six children, </i>is more complex and thus contains more opportunities for the parser to produce an incorrect analysis. How­ever, this more complex sentence contains the same grammatical relations between the same words as in the simple sentence (Figure 4).</p><p>Similar to the <i>one</i><i> </i><i>sense</i><i> </i><i>per</i><i> </i><i>collocation</i><i> </i>constraint (Yarowsky, 1993), we assume that any sentence con­taining the words <i>Mozart, born </i>and <i>1756 </i>will con­tain the same relationships between these words. We hypothesise that by constraining the parser to out­put an analysis consistent with these relationships, the correct analysis of the complex sentences can be found without manual intervention. These complex sentences can then be used as additional training data, allowing the parser to learn the general pattern of the sentence. For this process, we use grammat­ical relations rather than CCG dependencies as the former generalise across the latter and are therefore more transferable between sentences.</p></section><section number="6" title="Method"><p>Our procedure may be outlined as follows:</p><p>1. Manually select facts and identify keywords.</p><p>2. Collect sentences from HTML documents con­taining all keywords of a given fact.</p><p>3. Manually identify grammatical relations con­necting fact keywords.</p><p>4. Parse all sentences for the fact, using these grammatical relations as constraints.</p><p>5. Add successful parses to training corpus.</p><p>Further detail is given below. Section 6.1 covers items 1 and 2, 6.2 item 3 and 6.3 items 4 and 5.</p><subsection number="6.1" title="Sentence collection"><p>First, we compiled a list of 43 facts based on factoid questions from the TREC 2004 Question Answering track (Voorhees, 2004) and the ISI Question Answer Typology (Hovy et al., 2002). In order for our hy­pothesis to hold, it is necessary for the facts used to refer to relatively unambiguous entities. For each fact, we identified a set of keywords and submitted these as queries through the Google SOAP Search API. The query was restricted to files with .html or .htm extensions to simplify document process­ing. Each unique page returned was saved.</p><page local="6" global="42"/><p>Prior to splitting the documents into sentences, HTML tags were stripped from the text and HTML character entities replaced with their character equivalents. In order to incorporate some HTML markup information in the sentence boundary iden­tification process, each page was split into a number of chunks by dividing at certain manually-identified HTML tags, such as heading and paragraph markers, which are unlikely to occur mid-sentence. Each of these chunks was then passed through the sentence boundary identifier available in NLTK v.0.9.3<footnote anchor="1"/> (Kiss and Strunk, 2006). Ideally, the process would use a boundary identifier trained on HTML text including markup, to avoid these heuristic divisions.</p><p>To further simplify processing, sentences were discarded if they contained characters other than standard ASCII alphanumeric characters or a small number of additional punctuation characters. We also regarded as noisy and discarded sentences con­taining whitespace-delimited tokens that contained fewer alphanumeric characters than other characters.</p><p>Sentences were then tokenised using a tokeniser developed the C&amp;C parser for the TREC competi­tion (Bos et al., 2007). Sentences longer than 40 to­kens were discarded as this might indicate noise or an error in sentence identification. Finally, sentences were only kept if each fact keyword appeared ex­actly once in the tokenised sentence, to avoid having to disambiguate between repetitions of keywords.</p><p>At the conclusion of this process, each fact had between 0 and 299 associated sentences, for a to­tal of 3472 sentences. 10 facts produced less than 10 sentences; the average yield of the remaining 33 facts was just over 100 sentences each. This pro­cessing does result in the loss of a considerable num­ber of sentences, however we believe the quality and level of noise in the sentences to be a more impor­tant consideration than the quantity, since there are still many facts that could yet be used.</p></subsection><subsection number="6.2" title="Constraint identification"><p>For each of the 33 facts that yielded more than 10 sentences, we identified a set of grammatical re­lations that connects the keywords in simple sen­tences. These sets contained between two and six grammatical relations; for example the relations for</p><footnote label="1"> http://nltk.sourceforge.net</footnote><p>the Mozart fact were (ncsubj born Mozart), (ncmod born in) and (dobj in 1756). Since we assume that the keywords will be related by the same grammatical relations in all sentences (see Section 5), we use these grammatical relations as constraints on the analyses of all sentences for the corresponding fact. Future work will automate this constraint identification process.</p><p>To avoid the need to disambiguate between in­stances of particular words, when constraints are applied each word must be identified in the sen­tence uniquely. Although sentences that contained repetitions of fact keywords were discarded dur­ing the collection stage, the constraints identified in this stage of the process may incorporate additional words. In the Mozart fact, the constraints contain the word <i>in </i>in addition to the keywords <i>Mozart, born </i>and <i>1756. </i>The sentence in Figure 4, for example, contains two instances of <i>in, </i>so this sentence is dis­carded at this point, along with other sentences that contain multiple copies of a constraint word.</p></subsection><subsection number="6.3" title="Parsing with constraints"><p>Having identified the grammatical relations to use as constraints, the sentences for the corresponding fact are parsed with the constraints enforced using the process described below. Sentences from two of the 33 facts were discarded at this point because their constraints included grammatical relations of type conj, which cannot be enforced by this method.</p><p>As described in Section 3, dependencies are formed between variables on lexical categories. For example, a dependency is created between the words <i>in </i>and <i>1756 </i>in Figure 1 by filling a variable W in the category of <i>in. </i>To force a particular dependency to be formed, we determine which of the two words will bear the category that lists the dependency; this word we call the <i>head.</i><i> </i>We identify the variable as­sociated with the desired dependency and pre-fill it with the second word. The parser's own unification processes then ensure that the constraint is satisfied.</p><p>Since the constraints are expressed in terms of grammatical relations and not CCG dependencies, each constraint must first be mapped back to a num­ber of possible dependencies. First, the head word may be assigned several possible categories by the supertagger. In this case, if any category does not license the desired dependency, it is removed from<page local="7" global="43"/></p><p>Table 1: Number of sentences acquired for a selection of facts. Figures are given for the number of sentences parsed and failed by the normal (unconstrained) parser as an indication of parser coverage on Web sentences.</p><p>the list; all others have a variable filled. In this way, no matter which category is chosen, the con­straint is enforced. Secondly, one grammatical rela­tion may map to some dependencies governed by the first word and some governed by the second. That is, in some constraints, either word may operate as the head. To account for this, we parse the sentence several times, with different combinations of head choices, and keep the highest probability analysis.</p><p>When parsing with constraints, not all sentences can be successfully parsed. Those where a span­ning analysis consistent with the constraints cannot be found are discarded; the remainder are added to the training corpus. From the 31 facts still repre­sented at this stage, we obtained 662 sentences.</p></subsection></section><section number="7" title="Results"><p>Table 1 traces the quantity of data throughout the process described in Section 6, for a sample of the facts used. It also gives totals for all 43 facts and for the 31 facts represented at completion.</p><p>The first column lists the fact keywords. We used a range of fact types, including abbreviations (CNN stands for Cable News Network) and identi­ties (Canberra is the capital of Australia, hydrogen is the lightest element) as well as actions (Armstrong landed on the Moon in 1969) and passive structures (Martin Luther King, Jr. was assassinated in 1969). This last example is one where fewer than 10 sen­tences were collected, most likely due to the large number of keywords.</p><p>The # Pages column in Table 1 shows the num­ber of unique Web pages saved for each fact, while # Sents indicates the number of unique sentences containing all fact keywords, after tokenisation. The next two figures show the coverage of the baseline parser on the Web sentences. The # Discarded col­umn indicates the number of these sentences that were discarded during the constraining process due to constraint words being absent or appearing more than once. The final two columns show how many of the remaining sentences the parser found a spanning analysis for that was consistent with the constraints.</p><p>It is clear that there is considerable variation in the number of sentences discarded during the constrain­ing process. For some facts, such as the Columbus fact, the keywords form a chain of grammatical re­lations with no additional words necessary. In such cases, no sentences were discarded since any sen­tences that reached this point already met the unique keyword restriction.</p><p>The main reason for sentences being discarded during constraining is that some relationships are ex­pressed in a number of different ways. Prepositions and punctuation are frequently responsible for this. For example, the date of the Moon landing can be expressed as both <i>in 1969 </i>and <i>on July 20, 1969, </i>while the expansion of the abbreviation CNN may be expressed as <i>CNN: Cable News Network, CNN (Cable News Network) </i>and indeed <i>Cable News Net-</i><page local="8" global="44"/></p><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>Normal</p></td><td class="cell"><p></p></td><td class="cell"><p>Constrained</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Keywords</p></td><td class="cell"><p># Pages</p></td><td class="cell"><p># Sents</p></td><td class="cell"><p># Parse</p></td><td class="cell"><p># Fail</p></td><td class="cell"><p># Discarded</p></td><td class="cell"><p># Parse</p></td><td class="cell"><p># Fail</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Armstrong landed Moon 1969</i></p></td><td class="cell"><p>805</p></td><td class="cell"><p>20</p></td><td class="cell"><p>20</p></td><td class="cell"><p>0</p></td><td class="cell"><p>14</p></td><td class="cell"><p>0</p></td><td class="cell"><p>6</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>CNN Cable News Network</i></p></td><td class="cell"><p>286</p></td><td class="cell"><p>62</p></td><td class="cell"><p>62</p></td><td class="cell"><p>0</p></td><td class="cell"><p>41</p></td><td class="cell"><p>8</p></td><td class="cell"><p>13</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Canberra capital Australia</i></p></td><td class="cell"><p>601</p></td><td class="cell"><p>78</p></td><td class="cell"><p>77</p></td><td class="cell"><p>1</p></td><td class="cell"><p>64</p></td><td class="cell"><p>0</p></td><td class="cell"><p>14</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Columbus discovered America</i></p></td><td class="cell"><p>683</p></td><td class="cell"><p>299</p></td><td class="cell"><p>291</p></td><td class="cell"><p>8</p></td><td class="cell"><p>0</p></td><td class="cell"><p>174</p></td><td class="cell"><p>125</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Martin Luther King Jr</i></p></td><td class="cell"><p>675</p></td><td class="cell"><p>6</p></td><td class="cell"><p>6</p></td><td class="cell"><p>0</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>assassinated 1968</i></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Mozart born 1756</i></p></td><td class="cell"><p>734</p></td><td class="cell"><p>161</p></td><td class="cell"><p>160</p></td><td class="cell"><p>1</p></td><td class="cell"><p>93</p></td><td class="cell"><p>30</p></td><td class="cell"><p>38</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>hydrogen lightest element</i></p></td><td class="cell"><p>582</p></td><td class="cell"><p>85</p></td><td class="cell"><p>83</p></td><td class="cell"><p>2</p></td><td class="cell"><p>51</p></td><td class="cell"><p>9</p></td><td class="cell"><p>25</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Total (all 43 facts)</p></td><td class="cell"><p>24934</p></td><td class="cell"><p><b>3472</b></p></td><td class="cell"><p>3421</p></td><td class="cell"><p>51</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Total (31 facts for which</p></td><td class="cell"><p>19019</p></td><td class="cell"><p><b>3237</b></p></td><td class="cell"><p>3190</p></td><td class="cell"><p>47</p></td><td class="cell"><p>1661</p></td><td class="cell"><p><b>662</b></p></td><td class="cell"><p>914</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>constraints successfully applied)</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><doubt alpha="50.0" length="76" tooSmall="False" monospace="0.0">Model LP      LR      LF     LF(POS)   SENT ACCUP      UR      UFCAT ACC COV</doubt><doubt alpha="6.9" length="160" tooSmall="False" monospace="0.0">Baseline 85.53 84.71 85.12 83.38 32.14 92.37 91.49 91.93 93.05 99.06 New 85.64   84.77   85.21     83.54        32.03      92.41   91.47   91.94     93.08 99.06</doubt><p>Table 2: Performance of the baseline (Clark and Curran (2007) normal-form model) and new parser models on CCG-bank section 23. Most results use gold standard POS tags; LF (POS) uses automatically assigned tags.</p><p><i>work (CNN). </i>By selecting only one set ofconstraints, only one option can be used; sentences involving other formats are therefore discarded. To overcome these difficulties, it would be necessary to allow sev­eral different constraint chains to be applied to each sentence, taking the highest probability parse from all resulting analyses. This could also be used to overcome the word-uniqueness restriction.</p><p>To evaluate the effectiveness of the 662 addi­tional sentences, we trained two models for the c&amp;c parser, a baseline using the parser's standard train­ing data, sections 02-21 of CCGbank, and a sec­ond model using CCGbank plus the extra data. For details of the training process, see Clark and Cur-ran (2007). Following Clark and Curran (2007), we evaluate the performance of the two parsing models by calculating labelled and unlabelled precision, re­call and F-score over CCGbank dependencies, plus coverage, category accuracy (percentage of words with the correct lexical category) and sentence accu­racy (percentage of sentences where all dependen­cies are correct).</p><p>The difficulty we face here is that the additional data is Web text, while the evaluation data is still newswire text. Since genre effects are important, we cannot expect to see a large difference in results in this evaluation. Future work will compare the two models on a corpus of Web sentences collected ac­cording to the same procedure, using different facts.</p><p>Table 2 summarises the results of our evaluation. The figures for the performance of the baseline sys­tem are the latest for the parser, and slightly higher than those given in Clark and Curran (2007). It is clear that the results for the two systems are very similar and that adding 662 sentences, increasing the amount of data by approximately 1.7%, has had a very small effect. However, now that the automated procedure has been developed, we are in a position to substantially increase this amount of data without requiring manual annotation.</p></section><section number="8" title="Conclusion"><p>We have described a procedure for automatically an­notating sentences from Web text for use as train­ing data for a statistical parser. We assume that if several sentences contain the same set of relatively unambiguous keywords, for example <i>Mozart, born </i>and <i>1756, </i>then those words will be connected by the same chain of grammatical relations in all sentences. On this basis, we constrain a state-of-the-art parser to produce analyses for these sentences that contain this chain of relations. The constrained analyses are then used as additional training data for the parser.</p><p>Aside from the initial identification of facts, the only manual step of this process is the the choice of constraints. The automation of this step will involve the identification of reliable sentences, most likely short sentences consisting of only a single clause, from which the relation chain can be extracted. The chain will need to be supported by a number of such sentences in order to be accepted. This is the step that truly exploits the redundancy of information on the Web, as advocated by Brill et al. (2001).</p><p>Once the choice of constraints has been auto­mated, we will have a fully automatic procedure for creating additional annotated training data for a sta­tistical parser. Our initial results show that the train­ing data acquired in this manner can be used to aug­ment existing training corpora while still maintain­ing high parser performance. We are now in a po­sition to increase the scale of our data collection to determine how performance is affected by a training corpus that has been significantly increased in size.</p></section><section title="Acknowledgements"><p>We would like to thank the anonymous reviewers and the Language Technology Research Group at the University of Sydney for their comments. The first author is supported by a University of Sydney Honours scholarship.</p><page local="9" global="45"/></section><references><p>Michele Banko and Eric Brill. 2001. Scaling to very very large corpora for natural language disambiguation. In <i>Proceedings of the 39th Annual Meeting of the Asso-cationfor Computational Linguistics, </i>pages 26-33. Douglas Biber. 1993. Using register-diversified corpora for general language studies. <i>Computational Linguis­tics, </i>19(2):219-241, June. Rens Bod. 2006. An all-subtrees approach to unsuper-vised parsing. In <i>Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting ofthe Association for Computational Linguistics, </i>pages 865-872. Johan Bos, James R. Curran, and Edoardo Guzetti. 2007. The Pronto QA system at TREC-2007. In <i>Proceedings ofthe Sixteenth Text REtreival Conference. </i>Eric Brill, Jimmy Lin, Michele Banko, Susan Dumais, and Andrew Y. Ng.  2001. Data-intensive question answering. In <i>Proceedings ofthe Tenth Text REtreival Conference, </i>pages 393-400, November. Ted Briscoe and John Carroll.  2006.  Evaluating the accuracy of an unlexicalized statistical parser on the PARC DepBank. In <i>Proceedings ofthe Poster Session ofthe Joint Conference ofthe International Committee on Computational Linguistics and the Association for Computational Linguistics, </i>pages 41-48, July. Ted Briscoe, John Carroll, and Rebecca Watson. 2006. The second release of the RASP system. In <i>Proceed­ings of the COLING/ACL 2006 Interactive Presenta­tion Sessions, </i>pages 77-80, Sydney, Australia, July. Aoife Cahill, Michael Burke, Ruth O'Donovan, Stefan Riezler, Josef van Genabith, and Andy Way. 2008. Wide-coverage deep statistical parsing using auto­matic dependency structure annotation. <i>Computa­tional Linguistics, </i>34(1):81-124, March. Eugene Charniak. 2000. A maxmium entropy inspired parser. In <i>Proceedings ofthe First Annual Meeting ofthe North American Chapter ofthe Association for Computational Linguistics, </i>pages 132-139. Stephen Clark and James R. Curran. 2006. Partial train­ing for a lexicalized-grammar parser. In <i>Proceedings ofthe Human Language Technology Conference ofthe NAACL, Main Conference, </i>pages 144-151, June. Stephen Clark and James R. Curran.   2007. Wide-coverage efficient statistical parsing with CCG and log-linear models.       <i>Computational Linguistics,</i> 33(4):493-552, December.</p><p>Michael Collins. 1999. <i>Head-Driven Statistical Models for Natural Language Parsing. </i>Ph.D. thesis, Univer­sity of Pennsylvania, Philadelphia, PA. Daniel Gildea. 2001. Corpus variation and parser per­formance. In Lillian Lee and Donna Harman, edi­tors, <i>Proceedings ofthe 2001 Conference on Empir-</i> <i>ical Methods in Natural Language Processing, </i>pages 167-202.</p><p>Sanda Harabagiu, Dan Moldovan, Marius Pasca, Rada Milhalcea, Mihai Surdeanu, Razvan Bunescu, Roxana Girju, Vasile Rus, and Paul Morarescu. 2000. FAL­CON: Boosting knowledge for answer engines. In <i>Proceedings ofthe Ninth Text REtreival Conference.</i></p><p>Julia Hockenmaier. 2003. <i>Data and Models for Statis­tical Parsing with Combinatory Categorial Grammar. </i>Ph.D. thesis, University of Edinburgh, Edinburgh, UK.</p><p>Eduard Hovy, Ulf Hermjakob, and Deepak Ravichan-dran. 2002. A question/answer typology with surface text patterns. In <i>Proceedings ofthe DARPA Human Language Technology conference, </i>pages 247-251.</p><p>Tracy Holloway King, Richard Crouch, Stefan Riezler, Mary Dalrymple, and Ronald M. Kaplan. 2003. The PARC 700 dependency bank. In <i>Proceedings ofthe EACL03: 4th International Workshop on Linguisti­cally Interpreted Corpora, </i>pages 1-8.</p><p>Tibor Kiss and Jan Strunk. 2006. Unsupervised multi­lingual sentence boundary detection. <i>Computational Linguistics, </i>32(4):485-525.</p><p>Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated cor­pus of english: The Penn Treebank. <i>Computational Linguistics, </i>19(2):313-330.</p><p>I. Dan Melamed. 2004. Statistical machine translation by parsing. In <i>Proceedings ofthe 42nd Annual Meet­ing ofthe Association for Computational Linguistics, </i>pages 653-660.</p><p>Stefan Riezler, Tracy H. King, Ronald M. Kaplan, Richard Crouch, John T Maxwell III, and Mark John­son. 2002. Parsing the Wall Street Journal using a lexical-functional gramar and discriminative estima­tion techniques. In <i>Proceedings ofthe 40th Annual Meeting ofthe Association for Computational Linguis­tics, </i>pages 271-278.</p><p>YoavSeginer. 2007. Fast unsupervised incremental pars­ing. In <i>Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, </i>pages 384-391, Prague, Czech Republic, June.</p><p>Satoshi Sekine. 1997. The domain dependence of pars­ing. In <i>Proceedings of the fifth conference on Applied Natural Language Processing, </i>pages 96-102.</p><p>Mark Steedman. 2000. <i>The Syntactic Process. </i>Lan­guage, Speech, and Communication series. The MIT Press, Cambridge, MA.</p><doubt alpha="62.0" length="50" tooSmall="False" monospace="0.0">Ellen M. Voorhees. 2004. Overview of the TREC 2004</doubt><p>Question Answering track. In <i>Proceedings ofthe Thir­teenth Text REtreival Conference. </i>David Yarowsky. 1993. One sense per collocation. In <i>Proceedings of the workshop on Human Language Technology, </i>pages 266-271.</p></references></body></article>