The SemEval-2007 task for extracting frame se­mantic structures relies on the human annotated data available in the FrameNet (FN) database. The Berkeley FrameNet project (Baker et al., 1998) is an ongoing effort of building a semantic lexicon for English based on the theory of frame semantics. In frame semantics, the meaning of words or word ex­pressions, also called target words (TW), comprises aspects of conceptual structures, or frames, that de­scribe specific situations. The semantic roles, or frame elements (FE), associated with a target word are locally defined in the frame evoked by the tar­get word. Currently, the FN lexicon includes more than 135,000 sentences extracted from the British National Corpus containing more than 6,100 target words that evoke more than 825 semantic frames.

For this task, we extended our previous work at Senseval-3 (Bejan et al., 2004) by (1) experiment­ing with additional features, (2) adding new classifi­cation sub-tasks to accomplish all the requirements, and (3) integrating these sub-tasks into a pipeline ar­chitecture.

Given a sentence, the frame semantic structure ex­traction task consists of recognizing the word ex­pressions that evoke semantic frames, assigning the correct frame to them and, for each target word, detecting and labeling the corresponding frame el­ements properly. The task also requires the de­termination of syntactic realizations associated to a frame element, such as grammatical function (GF) and phrase type (PT). The following illustrates a sentence example annotated with frame elements to­gether with their corresponding grammatical func­tions and phrase types for the target word "tie":

AEOI's activities and facilities have been tied to several universities

To extract semantic structures similar to those il­lustrated in the example we divide the SemEval-2007 task into four sub-tasks: (1) target word frame disambiguation (twfd); (2) FE boundary detection (febd); (3) GF label classification (gflc) and (4) FE label classification (felc). The sub-tasks twfd and gflc are natural extensions of the approach de­scribed in (Bejan et al., 2004) for the task of se­mantic role labeling at Senseval-03. We design ma­chine learning classifiers specific for each of the four sub-tasks and arrange them in a pipeline architecture such that a classifier can use information predicted by its previous classifiers. The system architecture is illustrated in Figure 1. In the data processing step, we parse each sentence into a syntactic tree using the Collins parser and extract named entities using an in

Target word J

Figure 1: System architecture.

house implementation of a named entity recognizer. We also extract from the FN lexicon mappings of target words and the semantic frames they evoke.

Various features corresponding to constituents were extracted and passed to SVM and ME clas­sifiers. For example, in Figure 2, the frame dis-

f FE Classif. sConcept_1 Concept_2 Evidence Cognizer Concepts Circumstances Time Place NNS I Frequency

Figure 2: Classification examples for each sub-task.

ambiguation sub-task extracts features correspond­ing to the constituent tied in order to predict the right frame between the semantic frames that can be evoked by this target word. In this figure, the correct categories for each sub-task are shown in boldface.

The complete set of features extracted for all the classification sub-tasks is illustrated in Figure 3. These represent a subset of features used in previ­ous works (Gildea and Jurafsky, 2002; Florian et al., 2002; Surdeanu et al., 2003; Xue and Palmer, 2004;

Bejan et al., 2004; Pradhan et al., 2005) for auto­matic semantic role labeling and word sense disam­biguation. Figure 3 also indicates whether or not a feature is selected for a specific classification task.

In the remaining part of this section we describe in detail each classification sub-task and the features that have the most salient effect on improving the corresponding classifiers.

We report experimental results on all four classi­fication sub-tasks. In our experiments we trained two types of classification models for each sub-task: SVM and ME. In order to optimize the performance measure of each sub-task and to find the best config­uration of classification models we used 20% of the sub-tasks training data as validation data. Table 1 lists the best configuration of classification models as well as the best sub-task results when running the experiments on the validation data. For frame disambiguation, we obtained 76.71% accuracy com­pared to a baseline of 60.72% accuracy that always predicts the most annotated frame for each of the 556 target words. The results for gflc and felc sub-tasks listed in Table 1 were achieved by using gold FE boundaries.

Table 1: Task results on the validation set.

The SemEval-2007 organizers provided fully an­notated training files, a scorer to evaluate these training files, and testing files containing flat sen­tences. In the evaluation process, a semantic depen­dency graph corresponding to a fully system anno­tated sentence is created and then matched with its gold dependency graph. The matching process not only evaluates every semantic structure of a target word, but also considers frame-to-frame and FE-to-FE graph relations between the semantic structures. In addition, various scoring options were consid­ered: exact or partial frame matching, partial credit for evaluating the named entities, evaluation of the flat frame elements labels, and an option for match­ing only the frames in evaluation. The evaluation for flat frame elements labels is similar with the evalu­ation performed at Senseval-3. The only difference is that for this scorer the FE boundaries must match exactly.

In Table 2, we present the averaged precision, recall and F1 measures for evaluating the seman­tic dependency graphs and detecting the semantic frames on the testing files. The "Options" col­umn represents the configuration parameters of the scorer: (E)xact/(P)artial frame matching, seman­tic (D)ependency or (L)abels only evaluation, and (Y)es/(N)o named entity evaluation.

Table 2: System results on the test set.

Although the system achieved good precision scores on the test data, the recall values caused the system to obtain unsatisfactory Fl-measure values. We expect that the recall will increase by consid­ering various heuristics for a better mapping of the frame elements to constituents in parse trees.

We described a system that participated in SemEval-2007 for the task of extracting frame semantic struc­tures. We showed that a pipeline architecture of the SVM and ME classifiers as well as an adequate se­lection of the classification models can improve the performance measures of each sub-task.