The task of labeling frame-evoking words with ap­propriate frames is similar to WSD, while the task of assigning frame elements is called Semantic Role Labeling (SRL), and has been the subject of several shared tasks at ACL and CoNLL. For example, in the sentence "Matilde said, 'I rarely eat rutabaga,"' said evokes the Statement frame, and eat evokes the Ingestion frame. The role of SPEAKER in the Statement frame is filled by Matilda, and the role of MESSAGE, by the whole quotation. In the Inges­tion frame, I is the Inges tor and rutabaga fills the INGES TIB LES role. Since the ingestion event is con­tained within the MESSAGE of the Statement event, we can represent the fact that the message conveyed was about ingestion, just by annotating the sentence with respect to these two frames.

After training on FN annotations, the participants' systems labeled three new texts automatically. The evaluation measured precision and recall for frames and frame elements, with partial credit for incorrect but closely related frames. Two types of evaluation were carried out: Label matching evaluation, in which the participant's labeled data was compared directly with the gold standard labeled data, and Se­mantic dependency evaluation, in which both the gold standard and the submitted data were first con­verted to semantic dependency graphs in XML for­mat, and then these graphs were compared.

There are three points that make this task harder and more interesting than earlier SRL tasks: (1) while previous tasks focused on role assignment, the current task also comprises the identification of the appropriate FrameNet frame, similar to WSD, (2) the task comprises not only the labeling of individ­ual predicates and their arguments, but also the inte­gration of all labels into an overall semantic depen­dency graph, a partial semantic representation of the overall sentence meaning based on frames and roles, and (3) the test data includes occurrences of frames that are not seen in the training data. For these cases, participant systems have to identify the closest known frame. This is a very realistic sce­nario, encouraging the development of robust sys­tems showing graceful degradation in the face of un­known events.

The basic concept of Frame Semantics is that many words are best understood as part of a group of terms that are related to a particular type of event and the participants and "props" involved in it (Fill-more, 1976; Fillmore, 1982). The classes of events are the semantic frames of the lexical units (LUs) that evoke them, and the roles associated with the event are referred to as frame elements (FEs). The same type of analysis applies not only to events but also to relations and states; the frame-evoking ex­pressions may be single words or multi-word ex­pressions, which may be of any syntactic category. Note that these FE names are quite frame-specific; generalizations over them are expressed via explicit FE-FE relations.

The Berkeley FrameNet project (hereafter FN) (Fillmore et al., 2003) is creating a computer- and human-readable lexical resource for English, based on the theory of frame semantics and supported by corpus evidence. The current release (1.3) of the FrameNet data, which has been freely available for instructional and research purposes since the fall of 2006, includes roughly 780 frames with roughly 10,000 word senses (lexical units). It also contains roughly 150,000 annotation sets, of which 139,000 are lexicographic examples, with each sentence an­notated for a single predicator. The remainder are from full-text annotation in which each sentence is annotated for all predicators; 1,700 sentences are an­notated in the full-text portion of the database, ac­counting for roughly 11,700 annotation sets, or 6.8 predicators (=annotation sets) per sentence. Nearly all of the frames are connected into a single graph by frame-to-frame relations, almost all of which have associated FE-to-FE relations (Fillmore et al., 2004a)

The major part of the training data for the task con­sisted of the current data release from FrameNet (Release 1.3), described in Sec.2 This was supple­mented by additional training data made available through SemEval to participants in this task. In ad­dition to updated versions of some of the full-text an­notation from Release 1.3, three files from the ANC were included: from Slate.com, "Stephanopoulos

Crimes" and "Entrepreneur as Madonna", and from the Berlitz travel guides, "History of Jerusalem".

A number of groups downloaded the training or test­ing data, but in the end, only three groups submitted results: the UTD-SRL group and the LTH group, who submitted full results, and the CLR group who submitted results for frames only. It should also be noted that the LTH group had the testing data for longer than the 10 days allowed by the rules of the exercise, which means that the results of the two teams are not exactly comparable. Also, the results from the CLR group were initially formatted slightly differently from the gold standard with regard to character spacing; a later reformatting allowed their results to be scored with the other groups'.

The LTH system used only SVM classifiers, while the UTD-SRL system used a combination of SVM and ME classifiers, determined experimentally. The CLR system did not use classifiers, but hand-written symbolic rules. Please consult the separate system papers for details about the features used.

The labels-only matching was similar to previous shared tasks, but the dependency structure evalua­tion deserves further explanation: The XML seman­tic dependency structure was produced by a program called fttosem, implemented in Perl, which goes sentence by sentence through a FrameNet full-text XML file, taking LU, FE, and other labels and using them to structure a syntactically unparsed piece of a sentence into a syntactic-semantic tree. Two basic principles allow us to produce this tree: (1) LUs are the sole syntactic head of a phrase whose semantics is expressed by their frame and (2) each label span is interpreted as the boundaries of a syntactic phrase, so that when a larger label span subsumes a smaller one, the larger span can be interpreted as a the higher node in a hierarchical tree. There are a fair num­ber of complications, largely involving identifying mismatches between syntactic and semantic headed-ness. Some of these (support verbs, copulas, mod­ifiers, transparent nouns, relative clauses) are anno­tated in the data with their own labels, while oth­ers (syntactic markers, e.g. prepositions, and auxil­iary verbs) must be identified using simple syntactic heuristics and part-of-speech tags.

For this evaluation, a non-frame node counts as matching provided that it includes the head of the gold standard, whether or not non-head children of that node are included. For frame nodes, the partici­pants got full credit if the frame of the node matched the gold standard.

The strictness of the requirement of exact bound­ary matching (which depends on an accurate syntac­tic parse) is compounded by the cascading effect of semantic classification errors, as seen by comparing the F-scores in Table 3 with those in Table 2. The difficulty of the task is reflected in the F-scores of around 35% for the most difficult text in the most difficult condition, but participants still managed to reach F-scores as high as 75% for the more limited task of Frame Identification (Table 2), which more closely matches traditional Senseval tasks, despite the lack of a full sense inventory. The difficulty posed by having such an unconstrained task led to understandably low recall scores in all participants (between 25 and 50%). The systems submitted by the teams differed in their sensitivity to differences in the texts: UTD-SRL's system varied by around 10% across texts, while LTH's varied by 15%.

There are some rather encouraging results also. The participants rather consistently performed bet­ter with our more complex, but also more useful and realistic scoring, including partial credit and grad­ing on semantic dependency rather than exact span match (compare the top and bottom halves of Table 3). The participants all performed relatively well on the frame-recognition task, with precision scores av­eraging 63% and topping 85%.

The testing data for this task turned out to be espe­cially challenging with regard to new frames, since, in an effort to annotate especially thoroughly, almost 40 new frames were created in the process of an­notating these three specific passages. One result of this was that the test passages had more unseen frames than a random unseen passage, which prob­ably lowered the recall on frames. It appears that this was not entirely compensated by giving partial credit for related frames.

This task is a more advanced and realistic version of the Automatic Semantic Role Labeling task of Senseval-3 (Litkowski, 2004). Unlike that task, the testing data was previously unseen, participants had to determine the correct frames as a first step, and participants also had to determine FE boundaries, which were given in the Senseval-3.

A crucial difference from similar approaches, such as SRL with PropBank roles (Pradhan et al., 2004) is that by identifying relations as part of a frame, you have identified a gestalt of relations that enables far more inference, and sentences from the same passage that use other words from the same frame will be easier to link together. Thus, the FN SRL results are translatable fairly directly into formal representations which can be used for rea­soning, question answering, etc. (Scheffczyk et al., 2006; Frank and Semecky, 2004; Sinha and Narayanan, 2005).

Despite the problems with recall, the participants have expressed a determination to work to improve these results, and the FN staff are eager to collabo­rate in this effort. A project is now underway at ICSI to speed up frame and LU definition, and another to speed up the training of SRL systems is just begin­ning, so the prospects for improvement seem good.

This material is based in part upon work sup­ported by the National Science Foundation under Grant No. IIS-0535297.