The theory of frame semantics (Fillmore, 1982) holds that meaning is largely structured by holis­tic units of knowledge, called frames. Each frame encodes a conventionalized gestalt event or sce­nario, often with conceptual dependents (partic­ipants, props, or attributes) filling roles to elab­orate the specific instance of the frame. In the FrameNet lexicon (Fillmore et al., 2003), each frame defines core roles tightly coupled with the particular meaning of the frame, as well as more generic non-core roles (Ruppenhofer et al., 2006). Frames can be evoked with linguistic pred­icates, known as lexical units (LUs); role fillers can be expressed overtly and linked to the frame via (morpho)syntactic constructions. However, a great deal of conceptually-relevant content is left unexpressed or is not explicitly linked to the frame via linguistic conventions; rather, it is expected that the listener will be able to infer the appro­priate relationships pragmatically. Certain types of implicit content and implicit reference are for­malized in the theory of null instantiations (NIs) (Fillmore, 1986; Ruppenhofer, 2005). A complete frame-semantic analysis of text thus incorporates covert and overt predicate-argument information.

In this paper, we describe a system for frame-semantic analysis, evaluated on a semantic role labeling task for explicit and implicit arguments (§2). Extending the SEMAFOR 1.0 frame-semantic parser (Das et al., 2010a; outlined in §3), we detect null instantiations via a simple two-stage pipeline: the first stage predicts whether a given role is null-instantiated, and the second stage (§4) predicts how it is null-instantiated, if it is not overt. We report performance on the SemEval 2010 test set under the full-SRL and Nl-only conditions.

The SemEval 2007 task on frame-semantic pars­ing (Baker et al., 2007) provided a small (about 50,000 words and 2,000 sentences) dataset of news text, travel guides, and bureaucratic accounts of weapons stockpiles. Sentences in this dataset were fully annotated with frames and their argu­ments. The SemEval 2010 task (Ruppenhofer et al., 2010) adds annotated data in the fiction do­main: parts of two Sherlock Holmes stories by Arthur Conan Doyle. The SemEval 2010 train­ing set consists of the SemEval 2007 data plus one document from the new domain. This doc­ument has about 7800 words in 438 sentences; it has 1492 annotated frame instances, including 3169 (overt and null-instantiated) argument anno­tations. The test set consists of two chapters from another story: Chapter 13 contains about 4000 words, 249 sentences, and 791 frames; Chapter 14 contains about 5000 words, 276 sentences, and 941 frames (see also Table 3). Figure 1 shows two annotated test sentences. All data released for the 2010 task include part-of-speech tags, lemmas, and phrase-structure trees from a parser, with head annotations for constituents.

Our starting point is SEMAFOR 1.0 (Das et al., 2010a), a discriminative probabilistic frame-semantic parsing model that operates in three stages: (a) rule-based target selection, (b) proba­bilistic disambiguation that resolves each target to a FrameNet frame, and (c) joint selection of text spans to fill the roles of each target through a sec­ond probabilistic model.

Cognizer CP

THINK that I shall be in a position to MAKE the situation rather more

I to you before long .

It has been an exceedingly DIFFICULT and most complicated business .

Figure 1. Two consecutive sentences in the test set, with frame-semantic an­notations. Shaded regions represent frames: they include the target word in the sentence, the corresponding frame name and lexical unit, and arguments. Horizontal bars mark gold argument spans—white bars are gold annotations and black bars show mistakes of our NI-only system.

Chapter 13 Chapter 14 Table 1. Overt argument labeling performance.

Stage (c), known as argument identification or

SRL, is most relevant here. In this step, the system takes the target (frame-evoking) phrase t and cor­responding frame type / predicted by the previous stages, and independently fills each role of / with a word or phrase from the sentence, or the sym­bol other to indicate that the role has no (local) overt argument. Features used to inform this de­cision include aspects of the syntactic dependency parse (e.g. the path in the parse from the target to the argument); voice; word overlap of the argu­ment with respect to the target; and part-of-speech tags within and around the argument. SEMAFOR as described in (Das et al., 2010a) does not dis­tinguish between different types of null instantia­tions or find non-local referents. Given perfect input to stage (c), the system achieved 68.5% F\ on the SemEval 2007 data (exact match, evaluat­ing overt arguments only). The only difference in our use of SEMAFOR's argument identification module is in preprocessing the training data: we use dependency parses transformed from the head-augmented phrase-structure parses in the task data.

Table 1 shows the performance of our argument identification model on this task's test data. The SRL systems compared in (Ruppenhofer et al., 2010) all achieved precision in the mid 60% range, but SEMAFOR achieved substantially higher re­call, Fi, and label accuracy on this subtask. (The table also shows how performance of our model degrades when half or all of the new data are not used for training; the 9% difference in recall sug­gests the importance of in-domain training data.)

In this subtask, which follows the argument iden­tification subtask (§3), our system seeks to char­acterize non-overt core roles given gold standard local frame-argument annotations. Consider the following passage from the test data:

"That's lucky for him—in fact, it's lucky for all of you, since you are all on the wrong side of the law in this matter. I am not sure that as a consci­entious detective [Authorities my] flrst duty is not to arrest [suspect the whole household], [Jorges0]

The frame we are interested in, AR REST, has four core roles, two of which (Authorities and Sus­pect) have overt (local) arguments. The third core role, Charges, is annotated as having anaphoric or definite null instantiation (DNI). "Definite" means that the discourse implies a specific referent that should be recoverable from context, without marking that referent linguistically. Some DNIs in the data are linked to phrases in syntactically non­local positions, such as in another sentence (see Figure 1). This one is not (though our model in­correctly labels this matter from the previous sen­tence as a DNI referent for this role). The fourth core role, Offense, is not annotated as a null in­stantiation because it belongs to the same CoreSet as Charges—which is to say they are relevant in a similar way to the frame as a whole (both pertain to the rationale for the arrest) and only one is typ­ically expressed. We will use the term masked to refer to any non-overt core role which does not need to be specified as null-instantiated due to a structural connection to another role in its frame.

The typology of NIs given in Ruppenhofer (2005) and employed in the annotation distin­guishes anaphoric/definite NIs from existential or indefinite null instantiations (INIs). Rather than having a specific referent accessible in the dis­course, INIs are left vague or deemphasized, as in

Chapter 13 Chapter 14 Predicted overt DNI

cS INI masked inc. total

Table 2. Performance on the full task and the NI-only task. The NI model was trained on the new SemEval 2010 document, "The Tiger of San Pedro" (data from the 2007 task was excluded because none of the null instantiations in that data had annotated referents).

Table 3. Instantiation type confusion ma­trix for the full model (argument identifi­cation plus NI detection). Parenthesized numbers count the predictions of the cor­rect type which also predicted the same (argument or referent) span. On the NI-only task, our system has a similar distri­bution of NI detection errors.

the thing(s) eaten in the sentence We ate.

The problem can be decomposed into two steps: (a) classifying each null instantiation as definite, indefinite, or masked; and (b) resolving the DNIs, which entails finding referents in the non-local context. Instead, our model makes a single NI pre­diction for any role that received no local argument (other) in the argument identification phase (§3), thereby combining classification and resolution.

We have described a system that implements a clean probabilistic model of frame-semantic struc­ture, considering overt arguments as well as var­ious forms of null instantion of roles. The sys­tem was evaluated on SemEval 2010 data, with mixed success at detecting null instantiations. We believe in-domain data sparseness is the predom­inant factor limiting the robustness of our super­vised model.

This work was supported by DARPA grant NBCH-1080004 and computational resources provided by Yahoo. We thank the task organizers for providing data and conducting the evaluation, and two reviewers for their comments.