The distinction between complements and modifiers is im­portant for parsing and language interpretation, since it im­pacts upon both syntactic and semantic decisions. On the syntactic level, the distinction relates to questions such as whether a particular sentence is syntactically correct if a given dependent is not realised. On the semantic level, the complement-modifier distinction raises important is­sues such as whether a semantic representation is 'com­plete', with nothing needing to be inferred from context, and whether a particular preposition denotes an indepen­dent predicate, as opposed to being a contentless argument marker. The answers to these questions may significantly impact the quality of parsing and semantic interpretation. As noted by Meyers et al. (1996), incorrectly classifying a complement as a modifier may cause a syntactic parser to miss a parse, whilst incorrestly classifying a modifier as a complement may cause a parser to add a spurious parse. This would be a relevant distinction to make in a (deep) syn­tactic parser. In addition, in an accurate representation of predicate argument structure, syntactic phrase heads pred­icate of their complements, but modifiers predicate of the syntactic phrase heads. This distinction is therefore rele­vant for both deep parsers that combine syntax and seman­tics, and shallow semantic parsers that attempt to induce a predicate structure based on semantic role labelling. Unfortunately, the precise boundary between complements and modifiers is notoriously difficult to define. Existing lex­ical semantic resources, for example PropBank (Palmer et al., 2005), FrameNet (Johnson and Fillmore, 2000), Verb-Net (Kipper et al., 2000) and OntoNotes (Hovy et al., 2006) , do include information related to the complement-modifier distinction, but each applies slightly different cri­teria, depending on whether the emphasis is syntactic or semantic. A number of recent projects have attempted to merge information from different resources (Kwon and Hovy, 2006; Crabbe et al., 2006) and use them in pars­ing (Shi and Mihalcea, 2005; McConville and Dzikovska, 2007) . For such applications it is important to be able to understand and evaluate to what extent the different ap­proaches to making complement-modifier distinctions are compatible, and which approach is the most appropriate for a given application.

In this paper, we investigate whether the semantic criteria for distinguishing complements and modifiers used by the creators of the FrameNet corpus (i.e. 'core' versus 'non-core' semantic roles) correspond to syntactic intuitions, in particular to the (primarily) syntactic criteria used in the VerbNet lexicon, which only lists syntactic arguments of the verbs, based on whether they can participate in a num­ber of syntactic alternations.

We show that while there is a reasonably good correla­tion (0.85 agreement) between the semantic 'coreness' of FrameNet verb dependents and the complements listed in VerbNet, these notions do not align perfectly, and discuss the implications for using FrameNet as a source of syntactic information for parsing. We argue that for deep parsers con­cerned with both syntactic and semantic representations, it may be beneficial to separate the semantic and syntactic as­pects of complement/modifier distinction into separate fea­tures.

This paper proceeds as follows: section 2 presents some necessary background, including an introduction to both FrameNet and VerbNet, as well as our lexical harvesting project; section 3 discusses the methodology used in our investigation, in particular the way in which we combined use of VerbNet with our own linguistic intuitions in making a decision about complement-hood; section 4 presents the results; and section 5 discusses the implications for deep parsing.

FrameNet (Johnson and Fillmore, 2000) is a corpus of 140,000 English sentences (mainly drawn from the BNC), each annotated with both syntactic and semantic informa­tion. Underlying the corpus is an ontology of 800 'frames' (or semantic types), each of which is associated with a set of 'frame elements' (or semantic roles). Take for example the following sentence from the corpus:

(1) Overshadowed by Grigorovich, Kokonin nonetheless apparently eclipsed him in power in recent months.

In this example, the verb eclipse is associated with the Surpassing frame, which denotes situations where one entity is conceptualised as being superior to another in some way. This frame includes the following frame ele­ments, among others:

• Attribute - a property that invokes a scale (e.g. 'power', 'wealth')

• Item - the entity located closest to the end of the scale (i.e. the 'surpassor')

• Standard - the entity located farthest from the end of the scale (i.e. 'the surpassed')

• Time - the time when the Item is higher on the scale

Other verbs which are listed in the FrameNet ontology as 'evoking' the Surpassing frame include surpass, better, outdo and outshine.

The FrameNet annotation process then runs as follows, as­suming the example sentence in (1):

1. identify a target word for the annotation, for example the main verb eclipsed

2. identify the semantic frame which is evoked by the target word in the sentence - in this case the relevant frame is Surpassing

3. identify the sentential constituents which realise each frame element associated with the frame, i.e. Overshadowed by Grigorovich, [Kokonin] Itemnonetheless apparently eclipsed [him]Standard [in power]Attribute [in recent months]Time.

Finally, some basic syntactic information about the target word and the constituents realising the various frame ele­ments is also added:

• the part-of-speech of the target word (i.e. V, N, A or

• the syntactic category of each constituent realising a frame element (e.g. NP, PP, VPto, Sfin)

• the syntactic role, with respect to the target word, of each constituent realising a frame element (for exam­ple Ext (subject), Obj (object) or Dep (other depen­dent))

Thus, each sentence in the corpus can be seen to be anno­tated on at least three independent 'layers', as exemplified in Figure 1. The FrameNet corpus has proved to be a use­ful linguistic resource for a number of computational lin­guistics applications, for example semantic role labelling (Gildea and Jurafsky, 2002), information extraction (Sur-deanu et al., 2003), and question answering (Kaisser and Webber, 2007).

We took the lexicon we had harvested from the FrameNet corpus, and extracted all and only those entries (incorpo­rating an orthographic base form, a semantic type, and a subcategorisation frame) which specify at least one PP de­pendent. Along with the information about annotated de­pendents, each entry was also associated with the corpus sentence which it had been harvested from. A total of 17,000 verb entries were extracted in this way. The next step was to select a random sample of these for manual checking. Unfortunately, this proved to be some­what more problematic than just picking a random sub­set. The FrameNet project's approach to annotation has proceeded on a 'frame-by-frame' basis rather than focus­ing on fully annotating running text, meaning that each frame in turn is fully annotated with respect to its lexical units, before moving on to the next identified frame. As a result, some frames contain many more lexical units than others, and thus are associated with many more annotated instances. Thus, of the 261 FrameNet frames implicated in our set of extracted verb entries, the most common 10% ac­count for 65% of the total number of entries. In particular, just the one frame (Self .motion) accounts for a quarter of all the extracted entries.

We attempted to counteract this bias by limiting each frame to a maximum of two entries, and moreover restricting each verb in a frame to a maximum of one entry. We were thus left with 593 verb entries in our sample, involving a total of 432 subcategorised Core PP dependents and 204 non-Core ones.

The annotation task involved going through each PP depen­dent in turn and deciding whether or not VerbNet classifies it as a complement of the target verb. The relevant decision tree runs as follows:

1. If the verb is listed in VerbNet with the appropriate sense:

• if the VerbNet class lists the relevant subcategorisation frame, including the PP dependent in question, then mark the dependent as a 'complement'

• if the VerbNet entry does not list the relevant subcate-gorisation frame, including the PP dependent in ques­tion:

- if you think that a more complete VerbNet entry for the relevant verb would list the relevant sub-categorisation frame, including the PP dependent in question, then mark the dependent as a 'com­plement'

- if not, then mark the dependent as a 'non-complement' 2. If the verb is not listed in VerbNet with the appropriate sense:

• if you think that a more complete VerbNet which did contain an entry for the relevant verb would list the appropriate subcategorisation frame, including the PP dependent in question, then mark the dependent as a 'complement'

• if not, then mark the dependent as a 'non-complement'

Thus, we first of all determined whether the relevant sense of the target verb was included in VerbNet. If so, and if a matching subcategorisation frame including the dependent as a complement was listed, then it was deemed to be a complement PP.

In cases where either the relevant sense does not exist in VerbNet, or where the relevant verb sense does exist but a matching subcategorisation frame is not listed, things be­come a little more complicated, due to the incomplete na­ture of VerbNet as a resource.

Take for example, the example sentence in Figure 2. Al­though the verb eclipse does not appear in VerbNet, the syn­onymous verb surpass does (cf. John surpassed/eclipsed Mary in raw talent). By close inspection of the subcat-egorisation frames listed in the VerbNet class in which surpass appears (i.e. exceed-90), we judge that the Attribute dependent of the verb eclipse in Figure 2 is a VerbNet complement — the class contains the subcategorisation frame NP:Theme1 V NP:Theme2 P NP:Attrib, where Theme1, Theme2 and Attrib match Item, Standard andAttribute respectively. Incases wheretherelevantverbsensedoesappearinVerb-Net, but there is no matching subcategorisation frame in­cluding thePPinquestionweappliedourownlinguisticin-tuitions (using standardly assumed criteria for distinguish­ing complements from modifiers) to judge whether a more complete version of VerbNet should list the PP dependent as a complement.

One quarter of the verb entries in our sample were anno­tated by both authors, in order to test for inter-annotator agreement. The results were as follows:

• Is the appropriate sense of the target verb listed in VerbNet? Agreement: 0.95, kappa: 0.90

• Assuming both annotators agree that the target verb appears in VerbNet, is the PP dependent listed as a complement? Agreement: 0.97, kappa: 0.93

• Assuming that both annotators agree that the target verb appears in VerbNet but the PP dependent is not listed, is it a complement of the target verb? Agree­ment: 0.80, kappa: 0.60

• Assuming that both annotators agree that the target verb is not listed in VerbNet, is the PP dependent a complement of the verb? Agreement: 0.94, kappa:

The results of our investigation into the relation between coreness and compement-hood are presented in Table 1. The three 'experiments' listed are as follows:

• Experiment 1 only takes into account the 433 depen­dents whose verb senses were adjudged to be listed in VerbNet, ignoring annotator judgements about whether an unlisted PP dependent is a complement or not (i.e. it assumes that VerbNet is a complete resource for the verbs it lists)

• Experiment 2 also assumes the subset of 433 depen­dents whose verb senses were adjudged to be listed in VerbNet, but includes annotator judgements for un­listed PP dependents

• Experiment 3 includes results for all 634 PP de­pendents in the sample, including those whose target verbs do not appear in VerbNet

For each experiment, we present both the total assignments of dependents to classes (the columns represent FrameNet's Core versus non-Core distinction and the rows represent the judgements about syntactic complement-hood made by the annotators, in conjunction with VerbNet) as well as the interannotator agreement and Cohen's kappa scores. Note that agreement increases significantly when we take into account annotator judgement in cases where Verb­Net fails to list the relevant PP dependent. Note also that the amount of chance agreement depends on the relative proportion of Core and non-Core dependents annotated in the FrameNet corpus. It appears that the former have been annotated more completely, since over the corpus as a whole there are twice as many Core PPs listed than non-Core ones — we were able to find many PP modifiers which had been ignored by annotators. In conclusion, it appears that around 13% of syntactic PP complements will be lost if we simply delete all non-Core dependents from our harvested lexicon, and 9% of the de­pendents retained will not be syntactic complements. Finally, we manually examined all instances where a FrameNet Core PP dependent was judged not to be a com­plement of the relevant target verb. A significant propor­tion of these (around one third) appeared to involve some kind of bracketing mismatch between syntax and seman­tics, where the Core dependent annotated in FrameNet is not a syntactic dependent of the target verb. Take the fol­lowing example:

(3) She looked away quickly, and unfastened [the waistband]FASTENER [of her uniform skirt]CONTAINING_OBJECT.

Here the target verb is unfasten. The FrameNet annotation recognises both the waistband and ofher uniform skirt as distinct dependents of the verb. However, on the syntactic level it is the entire noun phrase the waistband ofher uni­form skirt which is a syntactic dependent of the verb (the direct object).

As can be seen from our analysis, while the distinction be­tween Core and non-Core semantic roles in the FrameNet ontology is highly correlated with the kind of syntactic cri­teria for complement-hood used in verb lexicons like Verb-Net, the two do not align perfectly. In other words, there are a significant number of cases where either a non-Core semantic dependent is realised by a verbal complement or a Core semantic dependent is realised by a verbal modi­fier. Although many of these mismatches can be put down to either annotation errors (generally failing to assign a par­ticular use of some verb to the optimal frame) or question­able annotation policy (e.g. annotating as semantic depen­dents phrases which are definitely not syntactic dependents of the target verb), there remain a large number of such mis­matches which are simply a result of the incompatibility of semantic and syntactic notions of 'coreness'. The fact that the complement-modifier distinction is so dif­ficult to pin down has important implications for parsing and semantic interpretation. As pointed out in the introduc­tion, a parser needs to have access to a complete, accurate list of what complements go with which verbs, if correct parses are not to be missed or spurious parses to be added. Consider the following contrasting examples:

(b) John fell on the table (c) John slept on the table

The semantic representations corresponding to possible in­terpretations of those utterances are shown in Table 2. In (4a), 'rely' requires an on-PP complement (denoting the thing relied upon). In (4b), the on-PP is optional syntacti­cally, and can be either a complement or an adjunct seman-tically, depending on whether it denotes the trajectory of the fall, or the location where John fell. In (4c), the on-PP is syntactically and semantically optional, and it is definitely not a complement, since the location of the event is in no way unique to the sleep predicate. Unless the parser has access to this kind of information, then it will not be able to judge, for example, that the second sentence has two dis­tinct interpretations, whereas the other two only have one interpretation each.

In addition, note that there is an important contrast between the on-PP complements of rely and fall. In the former, the preposition on does not contribute any meaning to the sentence, other than to make clear what role its NP com­plement plays in the situation (i.e. it is basically a case-marker, cf. John trusted the map). The intrinsic meaning

Table 2: Possible semantic representations using FrameNet roles for utterances in example (4), and their relation to the complement/modifier distinction of the preposition on, involving some object being in con­tact with the top horizontal surface of some other object, plays no role in the predicate argument structure of this sentence, which can best be represented as something like relyjjn(John,map). Furthermore, on is the only preposition which can be used here to encode the relevant semantic role (apart from its stylistic variant upon). In the second sentence, on the other hand, the preposition does contribute its intrinsic meaning to the predicate argu­ment structure — as the place where the trajectory of the fall ends. Thus, the predicate argument structure of this sentence is more like fall(John,to(on(table))), with the in­ference that, at the culmination of the event of falling, John is in fact 'on the table'. Also, a number of other preposi­tions can be used here to realise the relevant semantic role, e.g. off, under, over, through, etc.

The contrast between argument-marker and predicative uses of prepositions is important both for applications con­necting language to reasoning, and for applications using shallow semantic representations. For example, Kaisser and Webber (2007) describe the use of FrameNet in ques­tion answering where questions are paraphrased using verbs in the same frame. If we consider the paraphrases that apply in our cases, an appropriate paraphrase for (4a) would be John trusted the map, leaving out the preposi­tion on, while an appropriate paraphrase for (4b) would be John dropped on the table, where the preposition is re­tained, since it is crucial for the meaning. With this in mind, we decided to investigate how dif­ferent parsers handle the distinction between predicative and non-predicative uses of prepositions. We considered: (a) shallow semantic parsers that output FrameNet frames (Gildea and Jurafsky, 2002); (b) the LinGO English Re­source Grammar (Copestake and Flickinger, 2000), a deep HPSG grammar that produces semantic representations us­ing first-order logic,; and (c) the TRIPS parser Allen et al. (2007), a unification based parser for dialogue sys­tems. These systems represent several different ways of using logical forms: semantic parsers are used for question answering (Kaisser and Webber, 2007) and information re­trieval (Surdeanu et al., 2003); the LinGO ERG has been used for translating spoken dialogue (Kay et al., 1994); and the TRIPS parser produces semantic representations that are easy to map into representations used by domain spe­cific reasoners common in dialogue systems. It turns out that these three parsers have all taken different approaches to the distinction between predicative and non-predicative prepositions. Shallow semantic parsers output frame representations that identify frame element names, but not their specific meanings. So for (4a), they will iden­tify on the map as a Intermediary, and in 4(b) treat on the table as Goal, leaving open the question whether it is the table alone, or the whole PP, that fills the appropriate slot. The LinGO ERG parser represents some prepositions as argument markers, but not always consistently, e.g. on in (4a) will be represented as a case marker, but for in John left for Boston will be represented as a regular preposition, as will on in (4b). In contrast, the TRIPS dialogue parser makes strictly semantic decisions, and marks all PPs where the preposition is not clearly an argument marker as com­plements (thus ensuring that the the preposition predicates are included in the logical form), but at the expense of be­ing unable to rule out certain syntactically anomalous utter­ances, such as *John put it.

These differences in approach are not surprising, given that the various syntactic and semantic criteria for identifying complements may not align, as we have shown in this pa­per, and therefore creating a lexicon that accurately iden­tifies such distinctions is a difficult task. We are currently working on methods to better represent this distinction in our lexicon.

We propose that for parsing lexicons, especially for deep parsers, a possible solution is to replace a single distinction with several finer-grained features, addressing the key is­sues raised in the introduction: Is some dependent syntacti­cally required to complete the utterance? Is some (optional) prepositional or adjectival phrase a possible dependent of a given verb? And does a particular preposition correspond to an independent predicate (regardless of whether the de­pendent can be classified as a complement)? The method adopted by VerbNet of defining syntactic frames based on alternations is a good approach to defining syntactic complements. It also takes the first step towards identifying those prepositions that may contribute meaning towards the predicate vs. those which do not, by defining classes of equivalent prepositions (such as location and direction) which may appear in the same PP. However, since VerbNet is incomplete, we are currently de­veloping a corpus-based approach to answer these ques­tions. We are particularly interested in the answer to the third question, whether a preposition in a given PP depen­dent is an argument marker, or contributes meaning to the logical form. This information is, to our knowledge, not coded in any existing resources. Adding it would provide essential information for building semantic representations, and therefore make such representations more usable in in­terpretation tasks.

In this paper, we have attempted to evaluate the extent to which the distinction between semantically Core and non-Core dependents, as used in the FrameNet corpus, corre­sponds to the traditional distinction between syntactic com­plements and modifiers of a verb. We used the VerbNet verb database as our gold standard for making judgements about complement-hood, in conjunction with our own intuitions in cases where we considered VerbNet to be incomplete. We concluded that there is enough agreement between the two notions (0.85) to make practical the simple expedient of equating core PP dependents in FrameNet with PP com­plements in the wide-coverage verb lexicon we harvested from FrameNet. Doing so means that we lose around 13% of PP complements, whilst around 9% of the PP dependents left in the lexicon are not complements. We then discussed the implications of this result for deep parsing, suggesting that for parsers concerned with both syntactic and semantic representations, it may be beneficial to separate the seman­tic and syntactic aspects of complement/modifier distinc­tion into separate features

The work reported here was supported by grants N000140510043 and N000140510048 from the Office of Naval Research.