This paper addresses the issue of word-sense ambigu­ity in extraction from machine-readable resources for the construction of large-scale knowledge sources. We describe two experiments: one which ignored word-sense distinctions, resulting in 6.3% accuracy for seman­tic classification of verbs based on (Levin, 1993); and one which exploited word-sense distinctions, resulting in 97.9% accuracy. These experiments were dual pur­pose: (1) to validate the central thesis of the work of (Levin, 1993), i.e., that verb semantics and syntactic be­havior are predictably related; (2) to demonstrate that a 15-fold improvement can be achieved in deriving se­mantic information from syntactic cues if we first divide the syntactic cues into distinct groupings that correlate with different word senses. Finally, we show that we can provide effective acquisition techniques for novel word senses using a combination of online sources, in particular, Longman's Dictionary of Contemporary En­glish (LDOCE) (Procter, 1978), Levin's verb classifica­tion scheme (Levin, 1993), and WordNet (Miller, 1985). We have used these techniques to build a database of 10,000 English verb entries containing semantic infor­mation that we are currently porting into languages such as Arabic, Spanish, and Korean for multilingual NLP tasks such as foreign language tutoring and ma­chine translation.

As machine-readable resources (i.e., online dictionaries, thesauri, and other knowledge sources) become read­ily available to NLP researchers, automated acquisition has become increasingly more attractive. Several re­searchers have noted that the average time needed to construct a lexical entry can be as much as 30 min­utes (see, e.g., (Neff and McCord, 1990; Copestake et al., 1995; Walker and Amsler, 1986)). Given that we are aiming for large-scale lexicons of 20-60,000 words, automation of the acquisition process has become a ne­cessity.

Previous research in automatic acquisition focuses primarily on the use of statistical techniques, such as bilingual alignment (Church and Hanks, 1990; Kla-vans and Tzoukermann, 1996; Wu and Xia, 1995), or extraction of syntactic constructions from online dic­tionaries and corpora (Brent, 1993; Dorr, Garman, and Weinberg, 1995). Others who have taken a more knowledge-based (interlingual) approach (Lonsdale, Mi-tamura, and Nyberg, 1996) do not provide a means for systematically deriving the relation between sur­face syntactic structures and their underlying semantic representations. Those who have taken more argument structures into account, e.g., (Copestake et al., 1995), do not take full advantage of the systematic relation be­tween syntax and semantics during lexical acquisition.

We adopt the central thesis of Levin (1993), i.e., that the semantic class of a verb and its syntactic behav­ior are predictably related. We base our work on a correlation between semantic classes and patterns of grammar codes in the Longman's Dictionary of Con­temporary English (LDOCE) (Procter, 1978). While the LDOCE has been used previously in automatic ex­traction tasks (Alshawi, 1989; Farwell, Guthrie, and Wilks, 1993; Boguraev and Briscoe, 1989; Wilks et al., 1989; Wilks et al., 1990) these tasks are primarily con­cerned with the extraction of other types of informa­tion including syntactic phrase structure and broad ar­gument restrictions or with the derivation of semantic structures from definition analyses. The work of San-fdippo and Poznanski (1992) is more closely related to our approach in that it attempts to recover a syntactic-semantic relation from machine-readable dictionaries. However, they claim that the semantic classification of verbs based on standard machine-readable dictionaries (e.g., the LDOCE) is "a hopeless pursuit [since] stan­dard dictionaries arc simply not equipped to offer this kind of information with consistency and exhaustive-ness."

Others have also argued that the task of simplify­ing lexical entries on the basis of broad semantic class membership is complex and, perhaps, infeasiblc (see, e.g., Boguraev and Briscoe (1989)). However, a number of researchers (Fillmore, 1968; Grimshaw, 1990; Gru­ber, 1965; Guthrie et al., .1991; Hearst, 1991; Jackend-olF, 1983; Jackendoff, 1990; Levin, 1993; Pinker, 1989; Yarowsky, 1992) have demonstrated conclusively that there is a clear relationship between syntactic context and word senses; it is our aim to exploit this relationship for the acquisition of semantic lexicons.

3 Syntax-Semantics Relation: Verb Classification Based on Syntactic Behavior

The central thesis of (Levin, 1993) is that the seman­tics of a verb and its syntactic behavior are predictably related. As a demonstration that such predictable rela­tionships are not confined to an insignificant portion of the vocabulary, Levin surveys 4183 verbs, grouped into 191 semantic classes in Part Two of her book. The syn­tactic behavior of these classes is illustrated with 1668 example sentences, an average of 8 sentences per class.

Given the scope of Levin's work, it is not easy to verify the central thesis. To this end, we created a database of Levin's verb classes and example sentences from each class, and wrote a parser to extract basic syn­tactic patterns from the sentences. We then character­ized each semantic class by a set of syntactic patterns, which we call a syntactic signature, and used the re­sulting database as the basis of two experiments, both designed to to discover whether the syntactic signatures tell us anything about the meaning of the verbs. The first experiment, which we label Verb-Based, ignores word-sense distinctions by assigning one syntactic sig­nature to each verb, regardless of whether it occurred in multiple classes. The second experiment, which we label Class-Based, implicitly takes word-sense distinc­tions into account by considering each occurrence of a verb individually and assigning it a single syntactic sig­nature according to class membership.

The remainder of this section describes the assign­ment of signatures to semantic classes and the two ex­periments for determining the relation of syntactic in­formation to semantic classes. We will see that our clas­sification technique shows a 15-fold improvement in the experiment where we implicitly account for word-sense distinctions.

Verbs: break, chip, crack, crash, crush, fracture, rip, shatter, smash, snap, splinter, split, tear

Example Sentences:

Crystal vases break easily.

The hammer broke the window.

The window broke.

Tony broke her arm.

Tony broke his finger.

Tony broke the crystal vase.

Tony broke the cup against the wall.

Tony broke the glass to pieces.

Tony broke the piggy bank open.

Tony broke the window with a hammer.

Tony broke the window.

*Tony broke at the window.

*Tony broke herself on the arm.

*Tony broke himself.

*Tony broke the wall with the cup.

A break.

(a) the semantic classification given by Levin.

(b) the syntactic classification based on the derived syntactic signatures.

To return to the Change of State verbs, we now con­sider the syntactic signature of the verb break, rather than the signature of the semantic class as a unit. The verb break belongs not only to the Change of State class, but also four other classes: 10.6 Cheat, 23.2 Split, 40.8.3 Burt, and 48.1.1 Appear. Each of these classes is characterized syntactically with a set of sentences. The union of the syntactic patterns corresponding to these sentences forms the syntactic signature for the verb. So although the signature for the Change of State class has 13 frames, the verb break has 39 frames from the other classes it appears in.

Conceptually, it is helpful to consider the difference between the intension of a function versus its exten­sion. In this case, we are interested in the functions that group the verbs syntactically and semantically. In-tensionally speaking, the definition of the function that groups verbs semantically would have something to do with the actual meaning of the verbs. Likewise, the in­tension of the function that groups verbs syntactically would be defined in terms of something strictly syntac­tic, such as subcategorization frames. But the inten­sions of these functions are matters of significant the­oretical investigation, and although much has been ac­complished in this area, the question of mapping syntax to semantics and vice versa is an open research topic. Therefore, we can turn to the extensions of the func­tions: the actual groupings of verbs, based on these two separate criteria. The semantic extensions are sets of verb tokens, and likewise, the syntactic extensions are sets of verb tokens. To the extent that these functions map between syntax and semantics intensionally, they will pick out the same verbs extensionally.

So for the verb-based experiment, our technique for establishing the relatedness between the syntactic signa­tures and the semantic classes, is mediated by the verbs themselves. We compare the two orthogonal groupings of the inventory of verbs: the semantic classes defined by Levin and the sets of verbs that correspond to each of the derived syntactic signatures. When these two groupings overlap, we have discovered a mapping from the syntax of the verbs to their semantics, via the verb tokens. More specifically, we define the overlap index as the number of overlapping verbs divided by the av­erage of the number of verbs in the semantic class and the number of verbs in the syntactic signature. Thus an overlap index of 1.00 is a complete overlap and an over­lap of 0 is completely disjoint. In this experiment, the sets of verbs with a high overlap index are of interest.

When we parsed the 1668 example sentences in Part Two of Levin's book (including the negative examples), these sentences reduce to 282 unique patterns. The 191 sets of sentences listed with each of the 191 semantic classes in turn reduces to 748 distinct syntactic signa­tures. Since there are far more syntactic signatures than the 191 semantic classes, it is clear that the mapping between signatures and semantic classes is not direct. Only 12 mappings have complete overlaps. That means 6.3% of the 191 semantic classes have a complete over­lap with a syntactic signature.

The results of this experiment are shown in Table 2. Three values are shown for each of the six variations in the experiment: the mean overlap, the median overlap, and the percentage of perfect overlaps (overlaps of value 1.00). In every case, the median is higher than the mean. Put another way, there is always a cluster of good overlaps, but the general tendency is to have fairly poor overlaps.

The six variations of the experiment are as follows. The first distinction is whether or not to count the neg­ative evidence. We note that the use of negative exam­ples, i.e., plausible uses of the verb in contexts which are disallowed, was a key component of this experi­ment. There are 1082 positive examples and 586 nega­tive examples. Although this evidence is useful, it is not available in dictionaries, corpora, or other convenient resources that could be used to extend Levin's classi­fication. Thus, to extend our approach to novel word senses (i.e., words not occurring in Levin), we would not be able to use negative evidence. For this reason, we felt it necessary to determine the importance of nega­tive evidence for building uniquely identifying syntactic signatures. As one might expect, throwing out the neg­ative evidence degrades the usefulness of the signatures across the board. The results which had the negative evidence are shown in the left-hand column of numbers in Table 2, and the results which had only positive evi­dence are shown in the right-hand side.

The second, three-way, distinction involves preposi­tions, and breaks the two previous distinctions involv­ing negative evidence into three sub-cases. Because we were interested in the role of prepositions in the sig­natures, we also ran the experiment with two dilferent parse types: ones that ignored the actual prepositions in the pp's, and ones that ignored all information except for the values of the prepositions. Interestingly, we still got useful results with these impoverished parses, al­though fewer semantic classes had uniquely-identifying syntactic signatures under these conditions. These re­sults are shown in the three major rows of Table 2.

The best result, using both positive and negative ev­idence to identify semantic classes, gives 6.3% of the verbs having perfect overlaps relating semantic classes to syntactic signatures. See Table 2 for the full results.

As we saw above, word sense disambiguation is critical to the success of any lexical acquisition algorithm. The Levin-based verbs arc already disambiguated by virtue of their membership in différent classes. The difficulty, then, is to disambiguate and classify verbs that do not occur in Levin. Our current direction is to make use of the results of the first two experiments, i.e., the re­lation between syntactic patterns and semantic classes, but to use two additional techniques for disambiguation and classification of non-Levin verbs: (1) extraction of synonym sets provided in WordNet (Miller, 1985), an online lexical database containing thesaurus-like rela­tions such as synonymy; and (2) selection of appropri­ate synonyms based on correlations between syntactic information in Longman's Dictionary of Contemporary English (LDOCE) (Procter, 1978) and semantic classes in Levin. The basic idea is to first determine the most likely candidates for semantic classification of a verb by examining the verb's synonym sets, many of which in­tersect directly with the verbs classified by Levin. The "closest" synonyms are then selected from these sets by comparing the LDOCE grammar codes of the unknown word with those associated with each synonym candi­date. The use of LDOCE as a syntactic filter on the semantics derived from WordNet is the key to resolv­ing word-sense ambiguity during the acquisition pro­cess. The full acquisition algorithm is as follows:

The research reported herein was supported, in part, by Army Research Office contract DAAL03-91-C-0034 through Battelle Corporation, NSF NYI IRI-9357731, Alfred P. Sloan Research Fellow Award BR3336, and a General Research Board Semester Award.