Semantic role labeling (SRL) has been defined as a sentence-level natural-language processing task in which semantic roles are assigned to the syntac­tic arguments of a predicate (Gildea and Jurafsky, 2002). Semantic roles describe the function of the participants in an event. Identifying the seman­tic roles of the predicates in a text allows knowing who did what to whom when where how, etc.

However, semantic role labeling as it is cur­rently defined misses a lot of information due to the fact that it is viewed as a sentence-internal task. Hence, relations between different local se­mantic argument structures are disregarded. This view of SRL as a sentence-internal task is partly due to the fact that large-scale manual annotation projects such as FrameNet and PropBank typ­ically present their annotations lexicographically by lemma rather than by source text.

It is clear that there is an interplay between lo­cal argument structure and the surrounding dis­course (Fillmore, 1977). In early work, Palmer et al. (1986) discussed filling null complements from context by using knowledge about individual pred­icates and tendencies of referential chaining across sentences. But so far there have been few attempts to find links between argument structures across clause and sentence boundaries explicitly on the basis of semantic relations between the predicates involved. Lwo notable exceptions are Fillmore and Baker (2001) and Burchardt et al. (2005). Fillmore and Baker (2001) analyse a short newspaper arti­cle and discuss how frame semantics could benefit discourse processing but without making concrete suggestions of how to model this. Burchardt et al. (2005) provide a detailed analysis of the links be­tween the local semantic argument structures in a short text; however their system is not fully imple­mented either.

With the shared task, we aimed to make a first step towards taking SRL beyond the domain of individual sentences by linking local semantic ar­gument structures to the wider discourse context. The task addresses the problem of finding fillers for roles which are neither instantiated as direct dependents of our target predicates nor displaced through long-distance dependency or coinstantia-tion constructions. Often a referent for an unin­stantiated role can be found in the wider context, i.e. in preceding or following sentences. An ex­ample is given in (1), where the CHARGES role

!http://framenet.icsi.berkeley.edu/ http://verbs.Colorado.edu/~mpalmer/ projects/ace.html

(ARG2 in PropBank) of cleared is left empty but can be linked to murder in the previous sentence.

(1) In a lengthy court case the defendant was tried for murder. In the end, he was cleared.

Another very rich example is provided by (2), where, for instance, the experiencer and the ob­ject of jealousy are not overtly expressed as depen­dents of the noun jealousy but can be inferred to be Watson and the speaker, Holmes, respectively.

(2) Watson won't allow that I know anything of art but that is mere jealousy because our views upon the subject differ.

This paper is organized as follows. In Section 2 we define how the concept of Null Instantiation is understood in the task. Section 3 describes the tasks to be performed, and Section 4, how they are evaluated. Section 5 presents the participant systems, and Section 6, their results. Finally, in Section 7, we put forward some conclusions.

The theory of null complementation used here is the one adopted by FrameNet, which derives from the work of Fillmore (1986). Briefly, omissions of core arguments of predicates are categorized along two dimensions, the licensor and the in­terpretation they receive. The idea of a licensor refers to the fact that either a particular lexical item or a particular grammatical construction must be present for the omission of a frame element (FE) to occur. For instance, the omission of the agent in (3) is licensed by the passive construction.

ßProtagonist

The omission is a constructional omission be­cause it can apply to any predicate with an appro­priate semantics that allows it to combine with the passive construction. On the other hand, the omis­sion in (4) is lexically specific: the verb arrive al­lows the Goal to be unspecified but the verb reach, also a member of the Arriving frame, does not.

(4) We arrived 0Goal at 8pm.

The above two examples also illustrate the sec­ond major dimension of variation. Whereas, in (3) the protagonist making the mistake is only existen-tially bound within the discourse (instance of in­definite null instantiation, INI), the Goal location in (4) is an entity that must be accessible to speaker and hearer from the discourse or its context (def­inite null instantiation, DNI). Finally, note that the licensing construction or lexical item fully and reliably determines the interpretation. Whereas missing by-phrases have always an indefinite in­terpretation, whenever arrive omits the Goal lexi­cally, the Goal has to be interpreted as definite, as it is in (4).

The import of this classification to the task here is that we will concentrate on cases of DNI, be they licensed lexically or constructionally.

As noted above, we allowed participants to ad­dress three different tasks: SRL only, NI only, full task. For role recognition and labeling we used a standard evaluation set-up, i.e., accuracy for role labeling and precision, recall, F-Score for role recognition.

The NI Unkings were evaluated slightly differ­ently. In the gold standard, we identified refer­ents for null instantiations in the discourse con­text. In some cases, more than one referent might be appropriate, e.g., because the omitted argument refers to an entity that is mentioned multiple times in the context. In this case, a system is given credit if the NI is linked to any of these expressions. To achieve this we create equivalence sets for the ref­erents of NIs (by annotating coreference chains). If the NI is linked to any item in the equivalence set, the link is counted as a true positive. We can then define NI linking precision as the number of all true positive links divided by the number of links made by a system, and NI linking recall as the number of true positive links divided by the number of links between an NI and its equivalence set in the gold standard. NI linking F-Score is then the harmonic mean between NI linking preci­sion and recall.

Since it may sometimes be difficult to determine the correct extent of the filler of an NI, we score an automatic annotation as correct if it in­cludes the head of the gold standard filler in the predicted filler. However, in order to not favor sys­tems which link NIs to very large spans of text to maximize the likelihood of linking to a correct ref­erent, we introduce a second evaluation measure, which computes the overlap (Dice coefficient) be­tween the words in the predicted filler (P) of an NI and the words in the gold standard one (G):

Example (6) illustrates this point. The verb won in the second sentence evokes the Fin-ish_competition frame whose competition role is omitted. From the context it is clear that the competition role is semantically filled by their first TV debate (head: debate) and last night's debate (head: debate) in the previous sentences. These two expressions form the equivalence set for the competition role in the last sentence. Any sys­tem that would predict a linkage to a filler that covers the head of either of these two expressions would score a true positive for this NI. However, a system that linked to last night's debate would have an NI linking overlap of 1 (i.e., 2*3/(3+3)) while a system linking the whole second sentence Last night's debate was eagerly anticipated to the NI would have an overlap of 0.67 (i.e., 2*3/(6+3))

(6) US presidential rivals Republican lohn McCain and Democrat Barack Obama have yesterday evening attacked each other over foreign policy and the econ­omy, in [their first TV debate]competition-[Last night's debate]competition was ea­gerly anticipated. Two national flash polls suggest that [Obama] competitor

„n riC'competition llMikFinish-Competition V

While a fair number of people expressed an inter­est in the task and 26 groups or individuals down­loaded the data sets, only three groups submitted results for evaluation. Feedback from the teams that downloaded the data suggests that this was due to coinciding deadlines and to the difficulty and novelty of the task. Only the SEMAFOR group addressed the full task, using a pipeline of argument recognition followed by NI identifica­tion and resolution. Two groups (GETARUNS++ and SEMAFOR) tackled the NI only task, and also two groups, the SRL only task (CLR and SE­MAFOR).

All participating systems were built upon ex­isting systems for semantic processing which were modified for the task. Two of the groups, GETARUNS++ and CLR, employed relatively deep semantic processing, while the third, SE­MAFOR, employed a shallower probabilistic sys­tem. Different approaches were taken for NI link­ing. The SEMAFOR group modeled NI linking as a variant of role recognition and labeling by ex­tending the set of potential arguments beyond the locally available arguments to also include noun phrases from the previous sentence. The system then uses, among other information, distributional semantic similarity between the heads of potential arguments and role fillers in the training data. The GETARUNS++ group applied an existing system for deep semantic processing, anaphora resolution and recognition of textual entailment, to the task. The system analyzes the sentences and assigns its own set of labels, which are subsequently mapped to frame semantic categories. For more details of the participating systems please consult the sepa­rate system papers.

Table 2: Shalmaneser (SHA), SEMAFOR (SEM) and CLR performance on the SRL task (across both chapters)

The results on the SRL task are shown in Table 2. To get a better sense of how good the perfor­mance of the submitted systems was on this task, we applied the Shalmaneser statistical semantic parser (Erk and Padö, 2006) to our test data and report the results. Note, however, that we used a Shalmaneser trained only on FrameNet version 1.3 which is different from the version 1.4 alpha that was used in the task, so its results are lower than what can be expected with release 1.4 alpha.

We observe that although the SEMAFOR and the CLR systems score a higher precision than Shalmaneser for argument recognition, the SE­MAFOR system scores considerably higher recall than Shalmaneser, whereas the CLR system scores a much lower recall.

In this paper we described the SemEval-2010 shared task on "Linking Events and Their Partic­ipants in Discourse". The task is novel, in that it tackles a semantic cross-clausal phenomenon that has not been treated before in a task, namely, link­ing locally uninstantiated roles to their coreferents at the text level. In that sense the task represents a first step towards taking SRL beyond the sen­tence level. A new corpus of fiction texts has been annotated for the task with several types of seman­tic information: semantic argument structure, co-reference chains and NIs. The results scored by the systems in the NI task and the feedback from participant teams shows that the task was more dif­ficult than initially estimated and that the evaluation should have focused on more specific aspects of the NI phenomenon, rather than on the com­pleteness of the task. Future work will focus on modeling the task taking this into account.

Josef Ruppenhofer and Caroline Sporleder are supported by the German Research Foundation DFG (under grant PI 154/9-3 and the Cluster of Excellence Multimodal Comput­ing and Interaction (MMCI), respectively). Roser Morante's research is funded by the GOA project BIOGRAPH of the University of Antwerp. We would like to thank Jinho Choi, Markus Dräger, Lisa Fuchs, Philip John Gorinski, Russell Lee-Goldman, Ines Rehbein, and Corinna Schorr for their help with preparing the data and/or implementing software for the task. Thanks also to the SemEval-2010 Chairs Katrin Erk and Carlo Strapparava for their support during the task organization period.