<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="12"/><title>Prepositional Phrase Attachment in Shallow Parsing</title><pubinfo>International Conference RANLP 2009 - Borovets, Bulgaria,pages 12-17</pubinfo><author surname="Van Asch" givenname="Vincent"><org  name="University of Antwerp" country="Belgium" city="Antwerp"/></author><author surname="Daelemans" givenname="Walter"><org  name="University of Antwerp" country="Belgium" city="Antwerp"/></author></firstpageheader><frontmatter><p>Prepositional Phrase Attachment in Shallow Parsing</p><p>Vincent Van Asch CLiPS - Computational Linguistics University of Antwerp Prinsstraat 13 B-2000 Antwerpen, Belgium <i>Vincent. </i>VanAsch@ua.ac.be</p><p>Walter Daelemans CLiPS - Computational Linguistics University of Antwerp Prinsstraat 13 B-2000 Antwerpen, Belgium Walter.Daelemans@ua.ac.be</p></frontmatter><abstract>In this paper we extend a shallow parser [6] with prepositional phrase attachment. Although the PP attachment task is a well-studied task in a discriminative learning context, it is mostly ad­dressed in the context of artificial situations like the quadruple classification task [18] in which only two possible attachment sites, each time a noun or a verb, are possible. In this pa­per we provide a method to evaluate the task in a more natural situation, making it possible to compare the approach to full statistical pars­ing approaches. First, we show how to extract anchor-pp pairs from parse trees in the GENIA and WSJ treebanks. Next, we discuss the exten­sion of the shallow parser with a PP-attacher. We compare the PP attachment module with a statistical full parsing approach [4] and ana­lyze the results. More specifically, we investi­gate the domain adaptation properties of both approaches (in this case domain shifts between journalistic and medical language). </abstract></header><body><section title="Keywords"><p>prepositional phrase attachment, shallow parsing, machine learning of language</p></section><section number="1" title="Introduction"><p>Shallow parsing (also called partial parsing) is an ap­proach to language processing that computes a basic analysis of sentence structure rather than attempting full syntactic analysis.</p><p>Originally defined by Abney [1] task to be solved with handcrafted regular expressions (finite state methods) and limited to finding basic (non-recursive) phrases in text, the label shallow parsing has meanwhile broadened its scope to machine learn­ing methods and to a set of related tasks including part of speech tagging, finding phrases (chunking), clause identification, grammatical role labeling, etc. Especially the machine learning approach to shallow parsing, pioneered by Ramshaw and Marcus [17] has been investigated intensively, in part because of the availability of benchmark datasets and competitions (CoNLL shared tasks 1999 to 2001)<footnote anchor="1"/>.</p><footnote label="1">See http://ifarm.nl/signll/conll/</footnote><p>It has been argued in [10] and by others that full parsing often provides too much (or not enough) infor­mation for some frequent natural language processing tasks. For example, for information retrieval, find­ing basic NPs and VPs is arguably sufficient, and for information extraction and other text mining tasks, finding syntactic-semantic relations between verbs and base NPs (who did what when and where) is more im­portant than having an elaborate configurational syn­tactic analysis, provided this shallow analysis can be computed in a deterministic, efficient, robust, and ac­curate way. Another advantage is that the modules in a machine learning based shallow parser can be trained independently, and allow the inclusion of more information sources (input features) than is possible in statistical parsing (because of sparse data problems). This flexibility in feature engineering, inherent in dis­criminative, supervised learning approaches to shallow parsing should make the approach more flexible, e.g. when engineering features robust for domain shifts.</p><p>However, a shallow approach also has its short­comings, an important one being that prepositional phrases, which contain important semantic informa­tion for interpreting events, are left unattached. Fur­thermore, while statistical full parsing used to be more noise-sensitive and less efficient than shallow parsing, that is no longer necessarily the case with recent de­velopments in parse ranking.</p><p>In this paper, we extend an existing memory based shallow parser, MBSP [5, 6], with a machine learning based prepositional phrase attachment module, and compare it to PP attachment in a state of the art sta­tistical parser. The machine learning method chosen is memory-based learning. We also investigate the abil­ity of this Memory-based PP attachment (MBPA) to cope with the problem of domain adaptation, i.e. the often dramatic decrease in accuracy when testing a trained system on data from a domain different from the domain of the data on which it was trained.</p><p>The remainder of this paper starts with an explana­tion of how the corpus is prepared in order to use it for PP attachment, Section 2. In Section 3 we explain the architecture of the memory-based PP-attacher. Sec­tion 4 discusses the experiments and shows the results. In this section we also compare our system to a statis­tical parser, the Collins parser [3, 4]. An overview of related work can be found in Section 5. Finally, Sec­tion 6 concludes this paper and discusses options for further research.</p><page local="2" global="13"/></section><section number="2" title="Data preparation"><p>In this section, we explain the extraction of the train­ing and test data and the algorithm used to create instances from treebanks.</p><subsection number="2.1" title="Training and test data"><p>The memory-based PP-attacher is trained on sections 2 through 21 of the Penn Treebank 2 Wall Street Jour­nal corpus (WSJ) [14]. The development of the sys­tem is done using the first 2000 PPs of sections 0-1 of WSJ. Evaluation of the system is done on the next set of 2000 PPs and additional evaluation is done using the first 2000 PPs of the GENIA corpus [20].</p><p>The corpora used for training and testing consist of tree structures representing the syntactic structure of sentences, as shown in Figures la and lb. We trans­form the trees into a flat representation in order to be able to define one unique attachment site (anchor) for every prepositional phrase (PP). A flat representation of an anchor-PP pair consists of a pair of indices. The first element of the pair is the index in the sentence of the anchor; the second element is the index of the preposition. Word count starts at zero. For the sen­tence in Figure la the representation is (3, 4). For the sentence in Figure lb the representation is (1, 4).</p><p>eat    a   pizza iri^^ïÏP with Nils olives</p><p>(a) I eat a pizza with olives.</p><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">Nr</doubt><p>pip vip d't^^nn <b><i>\fi^iip</i></b></p><doubt alpha="54.8" length="31" tooSmall="False" monospace="0.0">eat    a   pizza  withd't^nI.1,</doubt><p>The extraction algorithm yields 8933 prepositions from sections 0-1 of the WSJ corpus. For 1.95% of the PP-nodes in those two sections no anchor is found. For sections 2-21 1.99% of the 95,955 PP-nodes remain without an anchor. Some anchor-PP pairs are removed in a post-processing step because we limit the task to preposition-NP PPs and disregard preposition-ADJP sequences.</p><p>Table 1 shows the chunk type distribution of the an­chors. A fairly equal amount of the anchors are nouns and verbs. A minor part has an adjective, comparative adjective or something else as the anchor point.</p><p>NP VP Other</p><doubt alpha="0.0" length="16" tooSmall="False" monospace="0.0">50.5% 45.8% 3.7%</doubt><table caption="Table 1: The distribution of the anchors among the chunk types"></table></subsection><subsection number="2.2" title="Extracting  chunks   and preposi­tional phrases"><p>The memory-based PP-attacher (MBPA) is defined as a module within a shallow parser [6]. The MBPA is trained on the WSJ, and it needs chunk and pos tag information from other modules in that shallow parser. In order to prevent indirect contamination of the train­ing data with test data, we retrained the modules of the shallow parser delivering input to the PP attach­ment module on Wall Street Journal sections 2-21, us­ing the script of the 2000 CoNLL shared task to extract IOB-style chunks from WSJ trees.</p><doubt alpha="63.2" length="19" tooSmall="False" monospace="0.0">DT   NNS   VBD ADVP</doubt><p><b>the  cells  were r</b><b>L</b> <b>with NNP</b></p><doubt alpha="94.7" length="19" tooSmall="True" monospace="0.0">previously infected</doubt><doubt alpha="75.0" length="4" tooSmall="False" monospace="0.0">li-i</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">brk</doubt><p>Fig. 2: A tree structure with undetermined cut-off</p><p>(b) I eat a pizza with a fork. Fig. 1: Example tree structures</p><p>The example tree structures in Figure 1 are rela­tively straightforward to rewrite into a flat represen­tation.</p><p>The basic setup had to be extended by rules for specific cases. A good example is conjunction. In the sentence:</p><p><i>I see cats on the roof and behind the windows.</i></p><p>the parent node of the PP-nodes is a node that also holds the conjunction. Therefore, in this case the al­gorithm does not take the sibling of the PP-node but it takes a sibling of the parent node of the PP-node.</p><p>Converting syntactic trees into a flat representation introduces approximation errors. Figure 2 is an illus­tration of the problems encountered when looking for syntactic phrases. It is unclear which node should be used as the break point. Since the evaluation is based on chunks, the decisions made in the flattening step may have an influence on the final results. To mini­mize the bias we use the algorithm that was used to prepare the training data for the memory-based chun-ker to extract chunks from the syntactic trees output by Collins.</p><p>Table 2 shows the results of the comparison between the shallow parser and the Collins parser. The table shows the scores at chunk level. A chunk is correctly identified if it has the same label and it spans the same words as the gold standard chunk. When either the label or the span is not correct, the chunk is a false positive, when a chunk in the gold standard is not present in the system's output it is considered a false negative. The shallow parser chunking module and Collins perform comparably well although the former has slightly better results.<page local="3" global="14"/> This implies that there is no reason not to use the memory-based chunker as the basis for the PP-attacher. The PreP chunk in Table 2 is not equal to a PP. In this paper, the label PP is used for the combination of a preposition and a noun phrase, the PreP chunk is a chunk that consists of one or more prepositions only.</p><p>The logical first step in finding anchors for prepo­sitional noun phrases is finding the PPs. When ex­tracting the anchor-PP pairs (see Section 2.1) PPs are recognized by the label of the nodes. Since there are no nodes in the input of the MBPA a different strategy is used. PPs are retrieved by a regular expression-like al­gorithm. All <i>preposition NP </i>sequences are considered to be PPs. There are two exceptions to this regular expression rule. Sequences like <i>preposition " NP </i>('in "very modest amounts'") and <i>preposition VBG NP </i>('in making paper') are also considered PPs.</p></subsection><subsection number="2.3" title="Creating the instances"><p>The core of the PP-attacher is a memory-based ma­chine learner (supervised, classification-based learn­ing). Every PP found by the algorithm discussed in the previous subsection is a trigger for creating sev­eral instances. One instance is created for every com­bination of the PP in focus and a candidate-anchor. Candidate-anchors are the NPs and VPs of the sen­tence that are not part of the PP itself. For example, <i>I eat a pizza with olives.</i><i></i></p><p>will induce the creation of 3 instances. One instance for the combination <i>I-with, </i>one for the combination <i>eat-with </i>and one for the combination <i>pizza-with. </i>In the classification task, the machine learner will have to decide whether an instance suggests a true anchor or not. The advantage of this approach is that the machine learner can investigate every possible anchor for its validity and not only the VP and NP in front of the PP. The drawback of this approach is that we have skewed data. There will be many more negative instances in the instance base as can be seen in Table 3.</p><p>The features of the instances were chosen on the basis of previous work in machine learning based PP</p><p>count percentage</p><doubt alpha="20.0" length="20" tooSmall="False" monospace="0.0">NP         43,0496Wo</doubt><doubt alpha="9.1" length="22" tooSmall="False" monospace="0.0">VP         42,285 5.9%</doubt><doubt alpha="19.0" length="21" tooSmall="False" monospace="0.0">NONE     630,72088.1%</doubt><doubt alpha="42.9" length="21" tooSmall="False" monospace="0.0">TOTAL     716,054TBWo</doubt><table caption="Table 3: Distribution of classes in sections 2-21 of WSJ"></table><p>attachment and related tasks: the number of commas between the PP and the candidate anchor, the num­ber of other punctuation marks between the PP and the candidate anchor, the token-distance between the PP and the candidate anchor, the preposition if the candidate anchor is an NP that is part of a PP, the lemma and POS-tag of the last token of the candidate anchor, the lemma and POS tag of the token just in front of the preposition of the PP, the lemma of the preposition, the lemma and POS-tag of the last to­ken of the NP of the PP, the number of NPs between the candidate anchor and the PP, the number of PPs between the candidate anchor and the PP, and NP anchor tendency. If a preposition is for 10% of the cases in the training corpus attached to an NP, the NP anchor tendency will be 10.</p></subsection></section><section number="3" title="The     memory-based PP-attacher"><p>The input of the MBPA module consists of sentences tagged with Part-of-Speech tags, IOB-chunk tags and the lemmata for every word by other modules of the shallow parser. The output of the system is a set of pairs, where each pair represents a PP with its corre­sponding attachment point.</p><subsection number="3.1" title="Machine Learning Approaches"><p>The machine learning approach we chose is memory-based learning, as implemented in the open source software package TlMBL<footnote anchor="2"/>. We used version 6.1 [7]. Memory-based learning (MBL) is a supervised induc­tive algorithm for learning classification tasks based on the k-nearest neighbor classification rule.</p><footnote label="2">Available from http://ilk.uvt.nl .</footnote><table caption="Table 2: Results of chunking with our system and Collins" class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Our system prec</p></td><td class="cell"><p>recall</p></td><td class="cell"><p>f-score</p></td><td class="cell"><p>Collins prec</p></td><td class="cell"><p>recall</p></td><td class="cell"><p>f-score</p></td><td class="cell"><p>share</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>-</p></td><td class="cell"><p>0.97</p></td><td class="cell"><p>0.97</p></td><td class="cell"><p>0.97</p></td><td class="cell"><p>0.96</p></td><td class="cell"><p>0.88</p></td><td class="cell"><p>0.92</p></td><td class="cell"><p>20.76%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ADJP</p></td><td class="cell"><p>0.79</p></td><td class="cell"><p>0.74</p></td><td class="cell"><p>0.76</p></td><td class="cell"><p>0.76</p></td><td class="cell"><p>0.77</p></td><td class="cell"><p>0.76</p></td><td class="cell"><p>1.41%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ADVP</p></td><td class="cell"><p>0.84</p></td><td class="cell"><p>0.83</p></td><td class="cell"><p>0.83</p></td><td class="cell"><p>0.86</p></td><td class="cell"><p>0.81</p></td><td class="cell"><p>0.83</p></td><td class="cell"><p>2.75%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CONJP</p></td><td class="cell"><p>0.56</p></td><td class="cell"><p>0.80</p></td><td class="cell"><p>0.66</p></td><td class="cell"><p>0.60</p></td><td class="cell"><p>0.60</p></td><td class="cell"><p>0.60</p></td><td class="cell"><p>0.04%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>INTJ</p></td><td class="cell"><p>0.67</p></td><td class="cell"><p>0.20</p></td><td class="cell"><p>0.31</p></td><td class="cell"><p>0.40</p></td><td class="cell"><p>0.20</p></td><td class="cell"><p>0.27</p></td><td class="cell"><p>0.02%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>LST</p></td><td class="cell"><p>1.00</p></td><td class="cell"><p>0.53</p></td><td class="cell"><p>0.70</p></td><td class="cell"><p>0.88</p></td><td class="cell"><p>0.47</p></td><td class="cell"><p>0.61</p></td><td class="cell"><p>0.03%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NP</p></td><td class="cell"><p>0.94</p></td><td class="cell"><p>0.95</p></td><td class="cell"><p>0.94</p></td><td class="cell"><p>0.93</p></td><td class="cell"><p>0.92</p></td><td class="cell"><p>0.93</p></td><td class="cell"><p>41.21%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PreP</p></td><td class="cell"><p>0.97</p></td><td class="cell"><p>0.98</p></td><td class="cell"><p>0.97</p></td><td class="cell"><p>0.97</p></td><td class="cell"><p>0.97</p></td><td class="cell"><p>0.97</p></td><td class="cell"><p>15.57%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PRT</p></td><td class="cell"><p>0.77</p></td><td class="cell"><p>0.79</p></td><td class="cell"><p>0.78</p></td><td class="cell"><p>0.79</p></td><td class="cell"><p>0.87</p></td><td class="cell"><p>0.83</p></td><td class="cell"><p>0.37%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SBAR</p></td><td class="cell"><p>0.88</p></td><td class="cell"><p>0.88</p></td><td class="cell"><p>0.88</p></td><td class="cell"><p>0.89</p></td><td class="cell"><p>0.91</p></td><td class="cell"><p>0.90</p></td><td class="cell"><p>1.83%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>UCP</p></td><td class="cell"><p>0.00</p></td><td class="cell"><p>0.00</p></td><td class="cell"><p>0.00</p></td><td class="cell"><p>0.00</p></td><td class="cell"><p>0.00</p></td><td class="cell"><p>0.00</p></td><td class="cell"><p>0.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>VP</p></td><td class="cell"><p>0.94</p></td><td class="cell"><p>0.93</p></td><td class="cell"><p>0.93</p></td><td class="cell"><p>0.83</p></td><td class="cell"><p>0.88</p></td><td class="cell"><p>0.85</p></td><td class="cell"><p>16.01%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>weighted mean</p></td><td class="cell"><p>0.94</p></td><td class="cell"><p>0.95</p></td><td class="cell"><p>0.94</p></td><td class="cell"><p>0.92</p></td><td class="cell"><p>0.91</p></td><td class="cell"><p>0.91</p></td><td class="cell"><p>100%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="4" global="15"/><p>However, the machine learner used to train the PP attachment module can be any algorithm that as­signs classes to instances. For comparison, we also implemented a system using maxent, an eager learn­ing method, as the machine learner. Maxent<footnote anchor="3"/> is an implementation of maximum entropy modeling. It is a general purpose machine learning framework that constructs a model that captures the distribution of outcomes for a given context in the training data [13].</p></subsection><subsection number="3.2" title="Heuristic decision making"><p>If the classifier would be able to predict the anchors with 100% accuracy, no post-processing would be nec­essary. Only one instance, the one with the correct anchor, would carry a positive class label and all other instances would have a negative classlabel. But due to misclassifications, multiple or no anchors may be identified by the machine learner. An extra step en­sures that the system presents one unique anchor for every PP. In case the PP in focus is classified posi­tively with exactly one anchor, that anchor-PP pair is returned as the solution. The other possible outcomes of the classification step are:</p><p>1. No instance for the PP in focus got a positive class =&gt; There is no anchor identified yet.</p><p>2. More than one instance for the PP in focus got a positive class =&gt; We have a decrease of possible anchors but still no unique anchor.</p><p>To resolve these cases, we need an extra step. A baseline algorithm is used if no anchor has been found (case 1). If there are still several candidate anchors to choose from, the entropy is used to reduce the set of candidates to just one unique anchor (case 2).</p><p>Baseline</p><p>The baseline is computed using a simple rule-based PP-attacher. If a rule fails, the next rule in the hi­erarchy is checked. The hierarchy of the rules of the baseline algorithm is:</p><p>1. Take the nearest NP or VP in front of the PP. We take an NP if in the training corpus the preposi­tion of the PP is associated more frequently with NP anchors. Otherwise we take the VP anchor.</p><p>2. Take the nearest anchor in front of the PP.</p><p>3. Take the nearest VP anchor behind the PP.</p><p>4. Take the nearest anchor behind the PP.</p><doubt alpha="100.0" length="7" tooSmall="False" monospace="0.0">Entropy</doubt><p>When the classification step results in a draw, the candidate with the lowest entropy will be the anchor. The entropy is calculated using the distribution of the classes of the nearest neighbors. When processing an instance with TiMBL we can obtain the (weighted) distribution of the classes of instances in memory that are in the neighborhood of the test instance. The en­tropy of an instance is computed using this distribu­tion. The formula is:</p><footnote label="3">Available from http : //homepages . inf . ed.ac.uk/ s0450736/maxent_toolkit .html.</footnote><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">n</doubt><doubt alpha="27.3" length="22" tooSmall="False" monospace="0.0">-^P(Ci)/052(P(Ci)) (1)</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">i=l</doubt><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">with</doubt><p>- n: the total number of different classes in the distri­bution</p><p><i>p(ç.\-  </i><u>the number of instances in the neighborhood with class i </u>V x/'        the total number of instances in the neighborhood</p><p>The memory-based learner has an optional weighing parameter. If weighing is applied, P(cj) is calculated using the weighted counts instead of the plain counts.</p><p>The candidate anchor with the lowest entropy is re­garded as the correct and unique anchor for a given PP. The rationale behind this decision is that choosing the candidate anchor with the lowest entropy means choosing the anchor for which the classifier was the most certain of its class.</p><p>Post-processing rules</p><p>For completeness, we also mention two post-processing rules that are applied because of some idiosyncrasies in the treebank data and common errors of the attacher-system. These rules are:</p><p>- If there are 2 consecutive prepositions the second preposition will always be attached to the first.</p><p>- If a PP is attached to a noun phrase anchor between parentheses, and the PP is not inside the parenthe­ses, then the noun phrase before the parentheses be­comes the anchor. This is done because the NP be­tween the parentheses is most of the time an elabo­ration/abbreviation of the noun phrase in front.</p></subsection></section><section number="4" title="Experiments and results"><p>We train 4 systems (baseline, MBL, maxent and a sta­tistical parser) on sections 2-21 of WSJ. In the first set of experiments, Section 4.1, we used the trained sys­tems to attach the second set of 2000 PPs of WSJ sections 0-1 to their anchors. In the second set of ex­periments, Section 4.2, we reuse the trained systems to attach 2000 PPs extracted from the GENIA corpus to their anchors.</p><p>For comparison, we parse every sentence fed to the MBPA with a state-of-the-art statistical parser, viz. Bikel's implementation of the Collins parser. Applying the PP extraction algorithm from Section 2.1 on the syntactic trees output by Collins will yield all anchor-PP pairs needed for evaluation.</p><subsection number="4.1" title="Training and testing on WSJ cor­pus"><p>Table 4 shows the accuracies of systems trained and tested on the WSJ corpus. We performed a \<footnote anchor="2"/> statis­tical test and found that maxent, MBL and Collins all significantly <i>(p</i><i> </i>&lt; 5%) differ from the baseline system. The variation between the accuracies of the machine learning systems is not found to be significant. The 'not retrieved' column is due to POS, chunking and/or syntactic tree errors in the pre-processing step. Look­ing at the first 200 errors MBPA and Collins made, shows that MBPA tends to misattach PPs at the start of the sentence.<page local="5" global="16"/> E.g. 'At the meeting, etc' Collins has a higher number of 'not retrieved' PPs because it often inserts adverbs and quotes into the PP. E.g. 'by commenting [PP publicly on the case]'.</p><p>Table 5 shows the accuracies of the steps involved in the PP-attacher system. The 91.21% accuracy for TiMBL means that for 79.05% of the PPs TiMBL identified a unique anchor and that 91.21% of these anchors were correct. For 11.7% of the PPs, TiMBL could not find an anchor so baseline had to take over. 44.4% of the PPs baseline handled, were correctly as­signed to their head. For 6.35% of the PPs, TiMBL found multiple anchors. Entropy handled these cases and found the correct anchor for 67% of them. Af­ter finding a unique anchor for every PP, the system made sure that for a sequence of PPs the last PP got attached to the previous. This happened for 1.2% of the PPs.</p><p>These results show that it is possible to develop a PP attachment module using supervised machine learning techniques, integrated module within a shallow parser, and reach state of the art accuracy when com­paring to one of the best statistical parsers available today. This way the semantically important informa­tion carried by relations between PPs and their an­chors becomes available to shallow analysis approaches with their advantages in terms of efficiency and flexi­bility. We were not able to find significant differences between lazy (TiMBL) and eager (Maxent) learning approaches for this problem.</p></subsection><subsection number="4.2" title="Domain Adaptation"><p>Adaptation of NLP systems to domains different from the one on which they were developed, is a crucial functionality to make the technology useful. Accuracy of systems deteriorates enormously when moving be­tween different domains. Accuracy drops of 20 to 40 percent are not uncommon for tasks such as parsing, named-entity recognition, word sense disambiguation, and machine translation when moving from the source domain to the new target domain. Usually, no or lim­ited labeled data exists for the target domain. We eval­uated the PP attachment systems trained on the WSJ using 2000 anchor-PP pairs from the GENIA corpus. The WSJ corpus consists of news articles on mainly financial issues in contrast to the medical abstracts of the GENIA corpus. Although one cannot always clearly say where the boundaries between domains are, we assume that medical and financial texts are suffi­ciently different. Table 6 shows the accuracies for the different systems. We performed these experiments to gain more insight into the relative robustness of dif­ferent approaches to PP attachment to domain shifts. The <i>x<footnote anchor="2"/> </i>test gave the same results as in the previous section: all systems perform significantly better than baseline but do not differ significantly from each other. As expected, the accuracy significantly decreases com­pared to the same-domain experiments.</p><p>Table 7 shows the robustness of the systems to a do­main shift from mainly financial to medical language. The higher the ratio, the lower the drop of accuracy. As can be seen, if no learning is involved (baseline) the system is most robust. A shallow approach is at an advantage here compared to full parsing because it allows more flexible feature engineering to allevi­ate the domain adaptation problem (e.g. by adding or removing specific lexical, syntactic, and semantic features to the classifiers. This is in general more dif­ficult in a statistical parsing approach because of data sparseness.</p></subsection></section><section number="5" title="Related work"><p>As Atterer and Schütze [2] state, the classic formula­tion of the task of PP attachment, as defined in [19] and [11], is a simplification. The classic formulation uses quadruples <i>(v,ni,p</i><i>,ri2) </i>that were manually se­lected from a corpus. This helps performance of PP attachment systems but for a natural language appli­cation these quadruples are not available. In their experiments, the PP attachment systems they evalu­ated did not significantly improve on a state-of-the-art parser, Collins parser [3, 4]. The PP-attacher system in this paper does not make use of this simplified rep­resentation and therefore can be regarded as more fit for the task of natural language PP attachment.</p><p>Foth and Menzel [9] implemented a PP attachment predictor for German and incorporated it in a rule-based dependency parser [8]. The PP attachment predictor was based on a collocation measure and sig­nificantly increased the accuracy on the PP attach­ment subtask.<page local="6" global="17"/> Basically, the collocation measure is a number indicating whether a word and a preposi­tion co-occur more often than chance. In this paper, we did not compute a collocation measure but for the NP anchor tendency feature we draw upon the same underlying idea.</p><table caption="Table 6: The accuracies on 2000 anchor-PP pairs from GENIA" class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Correct</p></td><td class="cell"><p>Incorrect</p></td><td class="cell"><p>Not retrieved</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Baseline</p></td><td class="cell"><p>69.20%</p></td><td class="cell"><p>27.00%</p></td><td class="cell"><p>3.80%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Collins</p></td><td class="cell"><p>78.80%</p></td><td class="cell"><p>19.35%</p></td><td class="cell"><p>1.85%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MB-attacher</p></td><td class="cell"><p>77.70%</p></td><td class="cell"><p>19.10%</p></td><td class="cell"><p>3.20%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Maxent-attacher</p></td><td class="cell"><p>77.15%</p></td><td class="cell"><p>19.65%</p></td><td class="cell"><p>3.20%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 4: The accuracies on 2000 anchor-PP pairs from WSJ" class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Correct</p></td><td class="cell"><p>Incorrect</p></td><td class="cell"><p>Not retrieved</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Baseline</p></td><td class="cell"><p>69.85%</p></td><td class="cell"><p>25.45%</p></td><td class="cell"><p>1.70%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Collins</p></td><td class="cell"><p>83.85%</p></td><td class="cell"><p>13.30%</p></td><td class="cell"><p>2.85%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MBL</p></td><td class="cell"><p>82.65%</p></td><td class="cell"><p>15.65%</p></td><td class="cell"><p>1.70%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Maxent</p></td><td class="cell"><p>81.40%</p></td><td class="cell"><p>16.90%</p></td><td class="cell"><p>1.70%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 5: The accuracies split into the different steps of MBPA" class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Accuracy (%)</p></td><td class="cell"><p>proportion (%)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>TiMBL</p></td><td class="cell"><p>91.21</p></td><td class="cell"><p>79.05</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Baseline</p></td><td class="cell"><p>44.44</p></td><td class="cell"><p>11.70</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Entropy</p></td><td class="cell"><p>66.93</p></td><td class="cell"><p>6.35</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>consecutive preps</p></td><td class="cell"><p>91.67</p></td><td class="cell"><p>1.20</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 7: The ratio of the accuracies GENIA/WSJ" class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Ratio</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Baseline</p></td><td class="cell"><p>0.991</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Collins</p></td><td class="cell"><p>0.940</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MB-attacher</p></td><td class="cell"><p>0.940</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Maxent-attacher</p></td><td class="cell"><p>0.948</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>As noticed in [21], the algorithm used to extract the pairs from the corpus has an influence on the accura­cies reported, and makes comparing of results among systems for different corpora and languages difficult. The noun attachment rate and the extraction proce­dure are two important features when comparing re­sults obtained using different corpora. As we tested our system and Collins' using the same training and test data, the comparison is reliable.</p><p>Other memory-based approaches to the problem of PP attachment can be found in [12] and [22]. [12] uses a memory-based PP-attacher combined with the MALTParser [16]. They showed that the dependency parser could not fully benefit from the separate PP-attacher although the PP-attacher module assigns PPs to their heads with a reasonable accuracy. The fea­tures they use for their PP-attacher system are lem­mata, POS-tags and distances between words.</p><p>In their paper, [15] mainly focus on how to disam­biguate between argument and adjunct PPs, but they provide an alternative way of extracting PPs from the WSJ treebank. Their final data contains quadruples and sets of multiple PP sequences.</p></section><section number="6" title="Conclusion"><p>In this paper we compared a shallow parsing approach to PP-attachment with a state of the art full parser. We used a flat representation of prepositional phrases and their associated attachment sites to train a ma­chine learner for the PP attachment task. We showed that a memory-based approach can obtain results for the PP attachment task comparable to a state-of-the-art full parser. The PP attachment system proposed in this article is not limited to the classical quadruple approximation of the PP attachment task and there­fore the system can be combined with any (shallow) parser that assigns part-of-speech tags, lemmata and chunk tags to natural language sentences. Such a PP attachment module can also be easily added to a full parser as a reattacher.</p><p>The shallow memory-based PP attachment module is fairly robust to a domain shift of the testing corpus but further research should focus on how to improve the robustness. Building a more robust PP attachment system would legitimate the use of the PP-attacher system       reattachment module in any full parser.</p></section><section title="Acknowledgements"><p>This research was made possible through financial sup­port from the University of Antwerp (BIOGRAPH GOA-project).</p></section><references><p>[1] S. Abney. Parsing by chunks. In <i>Principle-Based Parsing, </i>pages 257—278. Kluwer Academic Publishers, 1991.</p><p>[2] M. Atterer and H. Schütze. Prepositional phrase attachment without oracles. <i>Computational Linguistics, </i>33(4):469—476, 2007.</p><p>[3] D. Bikel. Intricacies of Collins' parsing model. <i>Computational Linguistics, </i>30(4):479-512, 2004.</p><p>[4] M. Collins. Head-driven statistical models for natural language parsing. <i>Computational Linguistics, </i>29(4):589—637, 2003.</p><p>[5] W. Daelemans, S. Buchholz, and J. Veenstra. Memory-based shallow parsing. In <i>Proceedings of CoNLL-99, </i>pages 53—60, Bergen, Norway, 1999.</p><p>[6] W. Daelemans and A. van den Bosch. <i>Memory-Based Lan­guage Processing. </i>Studies in Natural Language Processing. Cambridge University Press, Cambridge, 2005.</p><p>[7] W. Daelemans, J. Zavrel, K. van der Sloot, and A. van den Bosch. Timbl: Tilburg memory-based learner, version 6.1. Technical Report ILK 07-07, Tilburg University, 2007.</p><p>[8] K. Foth, M. Daum, and W. Menzel. Parsing unrestricted german text with defeasible constraints. In <i>Constraint Solving and Language Processing, </i>volume 3438 of <i>Lec­ture Notes in Computer Science, </i>pages 140—157. Springer, Berlin/Heidelberg, 2005.</p><p>[9] K. Foth and W. Menzel. The benefit of stochastic PP attach­ment to a rule-based parser. In <i>Proceedings of the COL­IN G f ACL 2006 Main Conference Poster Sessions, </i>pages 223-230, 2006.</p><p>[10] J. Hammerton, M. Osborne, S. Armstrong, and W. Daelemans. Introduction to special issue on machine learning approaches to shallow parsing. <i>Journal of Machine Learning Research, </i>2(Mar):551-558, 2002.</p><p>[11] D. Hindle and M. Rooth. Structural ambiguity and lexical relations. <i>Computational Linguistics, </i>19(1):103—120, 1993.</p><p>[12] S. Kübler, S. Ivanova, and E. Klett. Combining dependency parsing with PP attachment. In <i>Fourth Midwest Computa­tional Linguistics Colloquium, </i>2007.</p><p>[13] Z. Le. Maximum Entropy Modeling Toolkit for Python and C++, 2004. Version 20061005.</p><p>[14] M. Marcus, G. Kim, M. A. Marcinkiewicz, R. Maclntyre, A. Bies, M. Ferguson, K. Katz, and B. Schasberger. The Penn treebank: annotating predicate argument structure. In <i>HLT '94: Proceedings of the workshop on Human Language Tech­nology, </i>pages 114-119, Morristown, NJ, USA, 1994. Associa­tion for Computational Linguistics.</p><p>[15] P. Merlo and E. E. Ferrer. The notion of argument in prepositional phrase attachment. <i>Computational Linguistics, </i>32(3):341-377, 2006.</p><p>[16] J. Nivre. <i>Inductive Dependency Parsing, </i>volume 34 of <i>Text, Speech and Language Technology. </i>Springer, 2006.</p><p>[17] L. Ramshaw and M. Marcus. Text chunking using transformation-based learning. In D. Yarovsky and K. Church, editors, <i>Proceedings of the Third Workshop on Very Large Corpora, </i>pages 82—94. Association for Computational Linguis­tics, 1995.</p><p>[18] A. Ratnaparkhi. <i>Maximum Entropy Models for Natural Lan­guage Ambiguity Resolution. </i>PhD thesis, Computer and In­formation Science, University of Pennsylvania, 1998.</p><p>[19] A. Ratnaparkhi, J. Reynar, and S. Roukos. A maximum en­tropy for prepositional phrase attachment. In <i>Workshop on Human Language Technology, </i>pages 250—255, 1994.</p><p>[20] Y. Tateisi, A. Yakushiji, T. Ohta, and J. Tsujii. Syntax an­notation for the genia corpus. In <i>Proceedings of the IJCNLP 2005, Companion volume, </i>pages 222—227, October 2005.</p><p>[21] M. Volk. How bad is the problem of PP-attachment? A com­parison of English, German and Swedish. In <i>Proceedings of the Third ACL-SIGSEM Workshop on Prepositions, </i>pages 81-88, 2006.</p><p>[22] J. Zavrel, W. Daelemans, and J. Veenstra. Resolving PP at­tachment ambiguities with memory-based learning. In <i>Pro­ceedings CoNLL 1997, </i>pages 136-144, 1997.</p></references></body></article>