Automatic lexical acquisition is a fundamental problem in natural language processing (Boguraev and Pustejovsky, 1996). Detailed knowledge about verbs in particular is crit­ical in many NLP and IR tasks, yet manual determination of such knowledge for large numbers of verbs is a dif­ficult, time-consuming and resource intensive task (Dang et al., 1998; Fellbaum, 1998; Levin, 1993; Miller et al., 1990). Furthermore, while syntactic properties of verbs such as subcategorization frames may be gleaned from a machine readable dictionary (Dorr, 1997), or from exam­ples of usage in a corpus (Brent, 1993; Briscoe and Car­roll, 1997; Lapata, 1999; Manning, 1993; McCarthy and Korhonen, 1998), the extraction of semantic properties of verbs poses a more challenging problem. On the assump­tion that syntactic properties of verb usage, and their fre­quency distributions, reflect underlying semantic proper­ties of the verb, recent research has developed statistical corpus-based methods for learning selectional restrictions (Resnik, 1996; Riloff and Schmelzenbach, 1998), verbal aspect (Klavans and Chodorow, 1992; Siegel, 1999), and lexical semantic classification (Aone and McKee, 1996; La­pata and Brew, 1999; Schulte im Walde, 1998; Stevenson et al., 1999; Stevenson and Merlo, 2000).

Classification in particular has recently attracted atten­tion, as it imposes a hierarchical organization that enables an NLP system to efficiently maintain and exploit gener­alizations over lexical items (Palmer, to appear). One type of classification of verbs that has inspired recent work is the one in (Levin, 1993), which explores the hypothesis that the semantics of a verb determines the possible syntactic ex­pressions of its arguments. Based on this assumption, Levin has developed a very influential framework, where verbs are classified into semantic classes that are revealed empiri­cally by diathesis alternations—alternations in the syntactic expressions of arguments.

This kind of approach is very fruitful for NLP for two reasons. First, it organises the verbal lexicon around gener­alizations about argument structure—i.e., the thematic roles assigned to the arguments of a verb. These generalisa­tions play a major role in many language engineering tasks. For example, knowledge about argument structure is cru­cial for dealing with dependency relations in parsing and generation (e.g., (Srinivas and Joshi, 1999; Stede, 1998)): for handling thematic divergences in machine translation (Dorr, 1997); for document profiling in information re­trieval (Klavans and Kan, 1998); and for template filling in information extraction (Riloff and Schmelzenbach, 1998). Secondly, it supports corpus-based approaches. Corpus-based approaches to lexical semantic classification have drawn on Levin's hypothesis (that the semantics of a verb determines the possible syntactic expressions of its argu-ments)by applying the converse reasoning—that is, assum­ing that similar subcategorization alternations correspond to similar underlying semantics. This method is especially promising for automatic lexical acquisition, since it is the syntactic properties of arguments that are more easily ex-tractable from a corpus (Dorr and Jones, 1996; Lapata and 2000).

These approaches have thus far led to some promis­ing preliminary successes. For example, in using fre­quencies of syntactic features to automatically classifying verbs which share subcategorization frames but differ in argument structure, (Stevenson and Merlo, 2000) achieve 69.5% accuracy in a task whose baseline performance is 34%, and (Lapata and Brew, 1999) achieve 83.9% accuracy compared to a 61.3% baseline, both attaining reductions in error rate of over 50%. However, while accuracies of 70­84% seem like respectable initial results, the problem arises that we simply do not know what is a reasonable expecta­tion for performance in this kind of task.

Our observation is that (human) classification based on alternations and argument structures of verbs engenders a lively theoretical debate on class membership of verbs, which requires complex linguistic information and exper­tise. This leads us to believe that the task is intrinsically difficult, and also likely to show differences in classifica­tion between experts. There is reason then to hypothesize that the expert-defined upper bound of classification algo­rithms will be lower than 100% accuracy, and show vari­ability. Our goal then is to determine a realistic approxima­tion to this upper bound, in order to enable more informed evaluation of verb classification algorithms.

We report here the results of two experiments—using a forced-choice task and a non-forced choice task—which measure expert accuracy (compared to a gold standard) in classifying verbs into three pre-defined classes, as well as inter-expert agreement. The forced-choice version corre­sponds most closely to the typical computational classifi­cation task, while the non-forced choice version provides a measure on what is assumed to be a more natural task for a human expert. To preview our results, we find that the highest expert accuracy is 86.5% agreement with the gold standard, and that expert agreement is not very high (a kappa value between .53 and .66). Moreover, the two experiments show comparable results.

This study provides several results of general interest:

1. It provides the experts' accuracy on a controlled clas­sification task—providing the upper bound of perfor­mance for an automatic verb classification algorithm on a representative classification task.

2. It provides the degree of agreement between experts in both experiments—indicating how stable the expert performance is.

3. It provides results that are comparable across the two experiments—indicating that the controlled experi­ment is representative of a natural classification task.

In choosing a representative type of classification task for our experiments on expert performance, we focused on the verb classification task that we have been addressing through automatic corpus-based learning methods (Steven­son and Merlo, 1999; Stevenson and Merlo, 2000). In our computational experiments, we have been using frequen­cies of syntactic distributions from a very large corpus to trainan automatic classifier to discriminate three lexical se­mantic classes from (Levin, 1993). This choice of task for our experiments with human experts clearly helps us in the practical problem of evaluating our own automatic classi­fier. However, the task is also representative of much of the recent work on verb classification, which is largely inspired by Levin's work, as discussed above.

Thus, we adopted a definition of verb classes based on their argument structure, and we used Levin's classification as the benchmark. Our experimental focus is also notewor­thy in that the differences in the classes lies in their ar­gument structure, not in their possible subcategorizations. That is, the three verb classes represented in the stimuli each support both the transitive and intransitive subcatego-rization frame. This allows us to manipulate only the ar­gument structure variable, while holding subcategorization constant across the classes.

The stimuli were thus chosen from the following three classes, as exemplified in these sentences:

Unergative: The horse raced past the barn.

The jockey raced the horse past the barn.

Unaccusative: The butter melted in the pan.

The cook melted the butter in the pan.

Object-drop: The boy washed the hall. The boy washed.

Each class is distinguished by the content of the thematic roles assigned by the verb. For object-drop verbs, the subject is an Agent and the optional object is a Theme, yielding the thematic assignments (Agent, Theme) and (Agent) for the transitive and intransitive alternants respec­tively. Unergatives and unaccusatives differ from object-drop verbs in participating in the causative alternation, and also differ from each other in their core thematic argument. In an intransitive unergative, the subject is an Agent, and in an intransitive unaccusative, the subject is a Theme. In the causative transitive form of each, this core semantic argu­ment is expressed as the direct object, with the addition of a Causal Agent (the causer of the action) as subject in both cases.

The fact that the classes from which we draw the experi­mental stimuli are distinguished by argument structure and not subcategorization is important for two reasons. First, the task represents a key problem in verb classification, as alluded to above, since the ability to automatically learn thematic distinctions among verbs is a necessary compo­nent of automatic lexicon acquisition. Thus, it is important to determine a realistic upper bound on performance for this kind of task. Second, this property entails a fine-grained discrimination task that provides a challenging testbed for automatic classification, and (we hypothesize) for human expert classification as well. The verbs cannot be distin­guished simply by the subcategorization frames they allow,

Table 1: Percent Agreement (%Agr) and Pair-wise Agree­ment ( , Calculated by the Kappa Statistic) of Three Ex­perts (E1, E2, E3), Compared to Each Other and to a Gold Standard (Levin).

enabling us to explore the level of performance that can be expected when discriminating verbs based on argument structure alone.

The format, the task and the stimuli of both experi­ments where chosen to produce a controlled experimen­tal situation. We decided to measure the upper bound for a very narrowly characterised classification task first, and to measure a slightly more relaxed version of it in com­parison. Moreover, we needed to be able to compare re­sponses. Therefore, we performed a closed-form ques­tionnaire study, where the number and types of the target classes are defined in advance, for which we prepared a forced-choice and a non-forced-choice variant. The forced-choice study provides data for a maximally restricted ex­perimental situation, which corresponds most closely to the automatic verb classification task. However, we are also in­terested in slightly more natural results—provided by the non-forced-choice task—where the experts can assign the verbs to an "Others" category.

Going back to the original issues raised in the introduc­tion, we can then draw the following conclusions. First, accuracy by experts in the 3-way forced choice task is be­tween 71% and 86.5%, confirming the initial intuition that the task is difficult. Since it is not likely for the classifi­cation task to be solved at more than expert accuracy by automatic methods, we can conclude that 86.5% is a more realistic upper limit on the performance that we can expect in this task from an automatic classifier.

Second, the degree of agreement among experts in both tasks is relatively low. Carletta (1996) suggests that a kappa value of 0.8 or greater is an indication of good agreement. In neither of our tasks do highly trained experts reach that degree of agreement. The À' values we obtain (between .53 and .66) are in a way rather striking because in fact they are quite low.

One might think that if classification is so difficult, then it might not be useful. We remark however that an indi­vidual expert consistently over- or under-estimates specific classes (compared to the gold standard). The fact that the experts make consistent mistakes suggests that they inter­preted the defining properties given in the instructions, such as agent or theme, differently from each other. If this is the case, the disagreement can probably be partly solved by dis­cussion and negotiation, as is done for instance in tagging and bracketting a corpus. If establishing a new classifica­tion, we can therefore reccommend to follow corpus annotation practice, and adopt a two stage classification process, where experts can discuss results after the first stage, and reclassify verbs after discussion.

Furthermore, it might be objected that the classes we have chosen make the task too hard. We should recall that the way we have chosen the verbs kept their subcate-gorization alternations constant between the classes, while changing only their argument structure. This set-up likely constitutes a more difficult discrimination task than when both argument structures and subcategorization frames can vary. From existing classification work, we know that many verbs can be classified by only looking at their subcate-gorization frames (Lapata and Brew, 1999). However, in some cases, subcategorization information might not be sufficiently informative for the purposes of classification. Our verbs are a case in point, since the three classes share subcategorization frames but differ in their pattern of the­matic assignments. Thus, it seemed preferable to us to keep the subcategorization frame constant rather than varying both argument structures and subcategorization frames at the same time, as it provided a controlled experimental sit­uation.

Finally, we observe that the measure of agreement across the two tasks is similar for the target items: in the first task, #=.53, .59, and .66 among experts, and À'=.56, .74, and .80 with the gold standard; in the second task, A'=.64 among experts and À'=.57 and .80 with the gold standard. The average accuracy in the two experiments is also very close: 80% in the first, and 77% in the second. We conclude that, although not ideal, a forced-choice task can be representative of a natural task, and is informative about the difficulty of more open-ended classifications.

In conclusion, we have demonstrated an expert-based upper bound of 86.5%, far below the default maximum ac­curacy of 100%, in a lexical semantic classification task ap­plied to verbs that share subcategorization frames but dif­fer in argument structure (i.e., in the thematic roles they assign). Our results indicate that in a verb classification task of this kind, which depends on complex linguistic in­formation and relationships, it is important to experimen­tally determine a realistic upper bound of accuracy for the task from human expert performance, in order to enable in­formed evaluation of automatic methods.