The role-semantic paradigm (Gildea and Jurafsky (2002), inter alia) is a prominent model in automatic semantic anal­ysis with a wide range of proposed applications. With a few exceptions, role-based semantic analysis relies cru­cially (Gildea and Palmer, 2002; Punyakanok et al., 2005) on some sort of syntactic representation as input. This en­ables the analyzer to extract syntactic features that are used by statistical classifiers. The connection between syntax and semantics has also been noted in annotation projects of semantic treebanks; for instance, the SALSA project (Bur-chardt et al., 2006) and the Prague Dependency Treebank (Hajic, 1998) have annotated semantic structures on top of syntactic treebanks.

The CoNLL 2004 and 2005 shared tasks (Carreras and Marquez, 2005) were the first events that conducted an impartial evaluation of role-semantic labelers on the Prop-Bank corpus. More recently, SemEval organized a simi­lar evaluation (Baker, 2007) using the FrameNet corpus. While nearly all participants in the CoNLL shared tasks used a constituent representation that underlies the Prop-Bank annotation, the best-performing system (Johansson and Nugues, 2007b) of SemEval 2007 used dependency graphs.

These evaluations are not directly comparable however. They use different corpora, annotation, and training meth­ods and from these results, it is difficult to conclude what is the optimal representation to use in the parsing step. Some grammatical features used in constituents and dependencies may be directly equivalent. Conversely, some other features are tied to a specific representation. In addition, the contri­bution of the features and their behavior as a function of the training set and its size would still need more exploration. This paper addresses the question of how the syntactic representation influences the performance of automatic se­mantic analysis in the paradigm of Frame Semantics (Fill-more, 1976; Baker et al., 1998). To study the impact of syntactic representations, we performed a set of experi­ments in which we compare the performance of constituent-based and dependency-based semantic analyzers on the three main subtasks of FrameNet-based role-semantic anal­ysis.

The task of frame-semantic analysis is usually divided into three main subtasks: detection and disambiguation of target words, detection of semantic arguments, and finally classi­fication of arguments (see Figure 1).

PERCEIVER_PASSIVE,

PERCEPTION_EXPERIENCE

J " ^PHENOMENON

EXPERIENCERi

FOCAL_P. y EVENT

DESIRING

Although it is generally agreed that the best performance is obtained when the tasks are solved jointly, it is easiest from an engineering point of view, and computationally less ex­pensive, to treat them as independent tasks that are solved sequentially.

For both the constituent-based and the dependency-based semantic analyzer, we implemented the three subtasks as statistical classifiers using support vector machines; the first classifier assigns a frame for a given target word, the sec­ond decides whether a given node in a constituent or depen­dency parse tree represents an argument for a given predi­cate, and the final one assigns a semantic role to a node that has been identified as an argument by the previous step. The second step also makes use of a set of filtering rules to reduce the number of potential arguments (Xue and Palmer, 2004). Table 1 shows the features used by the clas­sifiers. The features used by the constituent-based and the dependency-based classifiers are indicated by C and D, re­spectively.

The following three subsection describes the features used by the classifiers. All examples are given with respect to Figure 2.

The purpose of the experiments is twofold: first, to inves­tigate whether dependency-based semantic analyzers are competitive with constituent-based analyzers; secondly, to measure the influence of the richness of the set of gram­matical functions. In addition, we would like to measure the influence of formalisms rather than of parsing perfor­mance, so we used two parsers of each type.

To train and evaluate the dependency-based semantic ana­lyzers, we parsed the training and test corpora using two freely available parsers: MaltParser (Nivre et al., 2007) and MSTParser (McDonald and Pereira, 2006). They were both trained on a dependency treebank that had been automatically converted from the Penn Treebank using the LTH constituent-to-dependency conversion tool (Johansson and Nugues, 2007a), achieving a labeled accuracy of 87.4% and 86.9% on section 23 of the WSJ part of the Treebank, respectively. In addition, we trained a model for Malt-Parser using Penn2Malt which gives a dependency representation with a smaller set of grammatical functions. This model achieves a labeled accuracy of 90.3% on WsJ section 23.

For the constituent-based analyzers, we used the popular Collins' (1997) and Charniak's (2000) parsers. Although grammatical functions (subject, locative, temporal,...) are available in the Treebank, they are not available in the out­put of these parsers. The published labeled precision and recall figures for these parsers are 88.1/87.5 and 89.5/89.6, respectively. Collins' parser comes with three different parsing models; in the experiments, we report the result for the best-performing of these.

This gives us in total five semantic analyzers that we studied for each experiment: MaltParser and MST­Parser with LTH-style dependencies, MaltParser with Penn2Malt dependencies, and Collins' and Charniak's constituent parsers. In the experiments, we studied only target words that were adjectives, adverbs, noun, and verbs, since annotated data are more reliable for these word classes. All tests were run on the test data from the semEval-2007 task on Frame-semantic structure Extrac­tion (Baker, 2007). The test corpus consists of 120 sen­tences and contains 970 target words and 1,663 semantic arguments, not counting null-instantiated arguments.

For English, there exist both constituent and dependency parsers, and this study has shown that frame-semantic an­alyzers can use either representation. However, many lan­guages have only one type of parser as for Danish or Czech where annotated corpora only use dependencies. We hope these experiments will help clarify the design of seman­tic parsers by itemizing the available features and outlining their contribution.

The most significant difference seems to be whether the output syntactic structure contains information about gram­matical functions or not. In this study, we compared a de­pendency grammar that had a rich set of functions (39 la­bels) for verb dependents with one that had a small set (five labels), and showed that the richer set leads to a significant improvement in argument classification accuracy for verbs. For noun and adjective modifiers on the other hand, there was no difference in the set of grammatical functions, and consequently no difference in classification performance.