The semantic role labeling (SRL) task has re­ceived considerable attention in recent years, with previous tasks in Senseval-2 (Litkowski, 2004), Semeval-1 (Baker et al., 2007), and CoNLL (Carreras & Marquez, 2004; Carreras & Marquez, 2005). The current task, Linking Events and their Participants in Discourse, con­tinues the evolution of SRL tasks with the intent of identifying Null Instantiations, i.e., frame elements that are absent from the local context, but potentially recoverable from the wider dis­course context.

CL Research participated in one subtask, role recognition and labeling, unable to implement techniques for the null instantiation subtask. This paper describes our efforts thus far (clearly a work in progress), specifically the implementa­tion of a development interface (section 2), the use of a specially constructed FrameNet dictionary (section 3), techniques for performing the role recognition and labeling task (section 4), our results (section 5), and future developments (sec­tion 6).

CL Research participated in the linking task by extending its Linguistic Task Analyzer (LTA), an interface also used for such tasks as word­sense disambiguation and recognizing textual entailment. LTA includes a wide array of mod­ules, including a full-scale parser, post-parsing semantic analysis routines, the use of XML func­tionality for creating and analyzing input and output, and access to several integrated dictiona­ries (used for semantic analysis). Modification of LTA for the linking task involves using existing functionality and implementing new functionali­ty specific to the task. We describe LTA in some detail to illustrate steps that might be relevant to a symbolic approach to the linking task.

Each task in LTA consists of a set of items to be analyzed, in this case, an identifier for each sentence in the document being analyzed. LTA loads the appropriate XML files (usually the an­notation file and the gold file) and provides vari­ous data for each sentence, including the number of terminals, non-terminals, frames, frame ele­ments that have been recognized, true positives, false positives, false negatives, and a characteri­zation of problems that have been encountered. Summary statistics are given, showing such things as the total number of frames and the scor­ing for the current annotation (when a gold file is available).

Whenever a sentence is selected in the LTA, the text is shown (accomplished by querying the XML for the selected sentence and retrieving all its terminals). LTA provides a capability for selecting all sentences matching particular criteria, e.g., all sentences containing a Color frame or all sentences having targets that have problematic entries in the FrameNet dictionary.

LTA contains a basic command to run and evaluate the system against the selected sen­tences. This can be used during development to test the effect of changes to the underlying code for performing any of the tasks. During the test phase, all sentences are selected, the Run and Evaluate command is executed, the XML test file is modified with the insertion of frame ele­ments constituting the system's answers, and the XML file is saved for the official submission. For the official submission, this took less than a minute for each of the two chapters.

A single sentence can be selected in the LTA for detailed examination. This Sentence Detail shows (1) the sentence itself (as in the main form), (2) a tree of the frames in the sentence, along with each of the frame elements that have been identified, minimally showing the target, and the text that has been identified for the frame element, and (3) from the training data, the frame element differences from the gold file, along with their terminal or non-terminal id references.

The Sentence Detail also has buttons to (1) score the annotation against the gold file for the sentence, (2) identify the missing core frame elements, (3) examine the FrameNet entries for the targets, and (4) perform the task. The func­tionality underlying the scoring and the task per­formance are called from the main form when all or selected sentences are to be processed (e.g., in the Run and Evaluate command).

Implementation of the scoring functionality for the Sentence Detail form attempts to follow the implementation in the official scorer. We have not yet captured every nuance of the scorer; however, we seem to have 99.9 percent agree­ment.

The Sentence Detail functionality is at the heart of the investigation and implementation of techniques for performing the tasks. At this time, we must view the implementation as only in its initial stages, minimally capable of performing the role recognition and labeling task. Further details about the implementation, including its shortcomings, will be described below.

Central to the performance of the linking task is the use of a dictionary constructed from the Fra­meNet data. This dictionary is in a format used by the CL Research DIMAP dictionary mainten­ance program. The FrameNet dictionary at­tempts to capture all the information in Frame­Net, in a form that can be easily accessed and used for tasks such as the linking task. This dic­tionary is also used in general word-sense dis­ambiguation tasks, when all words in a text are simultaneously disambiguated with several dic­tionaries. The FrameNet dictionary has almost 11,000 entries of four main types: frames, frame-to-frame relations, normal entries, and frame elements . This dictionary was initially described in Litkowski (2007), but is described in more detail in the following subsections in order to show how the information in these en­tries is used in the linking task.

To perform the role recognition and labeling task, the system first retrieves all the frames for the sentence and then iterates over each. The frame name and the target are retrieved. From the target XML, the id reference is used to re­trieve the part of speech and lemma from the tar­gets terminal node. With this information, an attempt is made to add child nodes to the frame node in the XML, thus supplying the system's performance of the task. After any nodes have thus been added, it is only necessary to save the modified XML as the output file.

The first step in adding child nodes is to obtain the lexical entries from the FrameNet dictionary for the frame and the lemma. Since the lemma may have multiple senses, we obtain the specific sense that corresponds to the frame. We iterate through the features for the sense, focusing on those providing syntactic patterns, such as those in Table 1. We deconstruct the feature value into its frame element name and its frequency. We then call a function with the feature name and the target's id reference to see if we can find a matching constituent; if successful, we create a child node of the frame with the frame element name and the id reference (for the child <fe-node> of frame element <fe> node).

The matching constituent function operates on the syntactic pattern, calling specific functions to search the XML terminals and non-terminals for constituent that fit the syntactic criterion. At present, this only operates on four patterns: DEN(), Poss(Gen), NP(Ext), and N(Head). As an example, for Poss(Gen), we select the non­terminals with the target as the "head" and search these for a terminal node marked as PRP$. A special constituent matching function was also written to look for the Supported frame element in the Support frame.

CL Research's results for the role recognition and labeling task are shown in Table 2. These results are generally consistent across the two chapters in the test and with results obtained with the training data during development. Combining

As can be seen, for entries with patterns (albeit a low recall), a substantial number of frame ele­ments could be recognized with high precision from a very small number of constituent match­ing functions. A detailed analysis of the results, identifying the contribution of each pattern rec­ognition and the problem of false positives, has not yet been completed. One such observation is that when the same syntactic pattern is present for more than one frame element, such as NP(Ext) for both Loser and Winner in the case of beat as shown in Table 1, the same constituent will be identified for both.

A significant shortcoming in the system oc­curs when there are no syntactic patterns availa­ble for a particular sense (27 percent of the tar­gets). For example, the lemma hour frequently appears in the training set as the target of either the Measure_duration or Calendric_unit frames, but it has no syntactic patterns (i.e., the FrameNet data contain no annotations for this lexical unit), while decade, also used in the same frames, does have syntactic patterns. This is a frequent occurrence with the FrameNet dictio­nary.

As should be clear from the preceding descrip­tion, there are many opportunities for improve­ment. First, several improvements can be made in the LTA to improve the ability to facilitate development. The LTA has only barely begun exploitation of the many integrated modules that are available. Additional functionality needs to be developed so that it will be possible to deter­mine the effect of any changes in constituent matching, i.e., what is the effect on recall and precision. The sentence detail form can be im­proved to provide better insights into the relation between syntactic patterns and their matching constituents.

Secondly, major improvements appear likely from greater exploitation of the FrameNet dictio­nary. At present, no use is made of the frequency information or the weighting of choices for matching constituents. When a given lemma has no syntactic patterns, it is likely that some use of the patterns for other lexical units in the frame can be made. It is also possible that some general patterns can be discerned using the frame ele­ment taxonomy.

It is important to see how far the FrameNet da­ta can be further exploited and where other lexi­cal data, such as available in WordNet or in more traditional lexical databases, can be used. The data developed for this linking task provide many opportunities for further exploration.