This paper presents a set of tools and methods for acquiring, manipulating, and analyzing machine-readable dictionaries. We give several detailed examples of the use of these tools and methods for particular analyses. A novel aspect of our work is that it allows the combined processing of multiple machine-readable dictionaries. Our examples describe analyses of data from Webster's Seventh Collegiate Dictionary, the Longman Dictionary of Contemporary English, the Collins bilingual dictionaries, the Collins Thesaurus, and the Zingarelli Italian dictionary. We describe existing facilities and results they have produced as well as planned enhancements to those facilities, particularly in the area of managing associations involving the senses of polysemous words. We show how these enhancements expand the ways in which we can exploit machine-readable dictionaries in the construc­tion of large lexicons for natural language processing systems.

1. Introduction.

It has become clear that the construction of computer systems that process natural language requires the creation of large computerized lexicons with extensive and accurate syntactic and semantic information about words (see Byrd(1986a), Ingria(1986)). It is also clear that it will be impossible to build these lexicons in the number and sizes required with only the manual labor of individual computer scientists, linguists, or lexicogra­phers. There are too many systems requiring too much information about too many words for the manual approach to succeed.

Fortunately, researchers are beginning to realize that the wealth of information in published dictionaries can be tapped by semi-automatic and fully-automatic meth-

* Department of Linguistics, University of Pisa and Institute of Computational Linguistics of CNR, Pisa. ** Department of Psychology, Hunter College of CUNY. ***Courant Institute of Mathematical Sciences, NYU.

ods, in order to help build the computerized lexicons we require. Work reported in Alshawi(1985), Calzolari (1983,1984a,b), Lesk(1986), Michiels(1982), etc., de­scribe various attempts to decipher and extract infor­mation in machine-readable versions of published dic­tionaries (henceforth: MRDs).

The present paper is intended to contribute to that literature by describing the tools and methods used by the Lexical Systems project at IBM Research. Individ­ual tools and their use in that project have been de­scribed elsewhere (Chodorow, et al.(1985), Byrd and Chodorow(1985), Byrd, et al.(1986b), Chodorow and Ravin(1987), Neff and Byrd(1987)). This paper differs from previous descriptions of our own and others' work, however, by presenting a framework and a set of general tools and methods that can be re-applied for a variety of lexicological and lexicographical applica­tions. Furthermore, it addresses problems involved in coherently combining information from different MRDs. In particular, we show how we can transfer information from one dictionary to another — even across languages — and correctly associate that infor­mation with the word senses to which it applies.

The plan of this paper roughly parallels the logical sequence of problems that face any project in compu­tational lexicology. Section 2 on acquisition and storage describes the initial stages of capturing and organizing dictionary data. Section 3 outlines the tools and meth­ods we use to exploit the contents of machine readable dictionaries. Applications of the tools and methods are described in the final two sections which discuss applied computational lexicography for monolingual and bilin­gual dictionaries.

2. Acquisition and Storage of Machine-Readable Dictionaries. 2.1. normalizing machine-readable dictionaries. Once machine readable dictionary tapes are acquired from the publishers, they usually have to be cleaned up and/or normalized. We have encountered three basic types of publisher tapes: one type, represented by Longman and Webster's Seventh, has some information segmented into separate records labelled with an iden­tifying code and other information implicit in font change codes. A second type, represented by Collins, consists of unsegmented text interspersed with font-change codes. A third type will be the new computer­ized OED (Tompa (1985)), where each type of informa­tion is specifically delimited at the beginning and end, and where there are any number of nesting levels. Much of the work described here was done with data of the second type; however, it is applicable also to the first type, and of course to the third.

In the simplest sense, cleaning up usually means the removal of typesetting codes and other formatting in­formation, or possibly removal of errors (usually done by hand). However, since the typesetting codes (e.g. begin-italic, begin-roman) are clues to the kind of infor­mation that follows the codes (for example, part-of-speech information appears in italics in the Collins dictionaries), they cannot simply be washed out. In­stead, the raw data undergoes a process of normaliza­tion, the goal of which is to identify and label all of the information in the dictionary tapes in such a way that it can be easily accessed without detailed knowledge of the formatting conventions. This is our rationale for the creation of lexical data bases.

We have carried out this normalization process in two stages. The goal of the first stage is to segment the information on the tapes into separate entries, storing the results in a direct access file, where the key is the dictionary headword and the data is simply the body of the entry. Dictionary data is stored in a Dictionary Access Method (DAM) file, from which it can be conveniently retrieved, by headword, for subsequent processing. DAM is completely described in Byrd, et al.(1986b). For this discussion, we note that DAM allows the efficient storage and retrieval of entries for words in alphabetical order and in random order by headword. Having the dictionary in a DAM file allows its entries to be accessed through the WordSmith on­line dictionary system, to be described later.

The goal of the second stage of the normalization process is to segment and identify the data in the entry, creating a second DAM file in lexical data base format. Both stages involve some rather idiosyncratic proce­dures that are tied to the various ways different publish­ers code the information on the tapes, particularly since the tapes were not prepared with any consideration of later computational use. However, as will be apparent later, our second-stage normalization is a more general approach that we envision using with a variety of machine-readable dictionary resources. In what fol­lows, we discuss the normalization of the Collins bilin­gual dictionaries.

The first stage of normalization, picking out the headwords and the associated information, was mostly straightforward: the headwords are preceded by a unique font code. Exceptions to this straightforward process were driven by the principle that the end result of the process should be a direct access dictionary usable by both people and other systems. This involves definition of a file keyword which differs from that of the dictionary headword in two important ways: (1) each keyword must be unique, and (2) there should be a keyword for each spelling of a word, with the important exception of inflected forms predictable by morpholog­ical rules, as explained below. The first constraint assures that dictionary lookup for a word will get all of the information associated with that word. The second assures that a lookup operation will find a word where it first expects to find it, and not inside of some other entry.

Three examples will suffice. In adherence to the first constraint, we combined the entries for the noun quackand the verb quack in the English-Italian Collins dic­tionary. In adherence to the second constraint, we created new entries for alternate spellings of words: the German-English headord Abtak(e)lung was made into two entries, Abtaklung and Abtakelung, the associated information was stored with just one of them, and cross-references to the other spelling were stored with both. Since the notation used for alternate spellings of this sort does not indicate which one is the more common usage, we stored the entry arbitrarily with the shorter form. A third example: where the German-English dictionary stores a number of compound words with the first element of the compound (e.g. Abend-: has -essen, -friede(n) -gesellschaft, etc.), we created full entries for all of them (e.g. Abendessen, Abendfriede, Abendgesellschaft, etc). In the process of reconstituting the compounds, however, we retained information on the location of the break to support possible later work on German noun compounds and to conform to another principle: "do not throw away anything."

An exception to the constraint that there should be separate entries for each spelling of a word, noted

, Nicoletta Calzolari, Martin S. Chodorow, Judith L. Klavans, Mary S. Neff, Omneya A. Rizk

before, is that we do not require creation of separate entries for those words which can be derived by morphological rules. For example, from German Lehrer(inf), which is shorthand for Lehrer and Lehrerin Mentale], we created only an entry for Lehrer, leaving the (in f) part for later segmentation. Identification of these exceptions is possible using heuristics and does not depend on having a system of morphological rules for the language of the dictionary. These heuristics are different for each dictionary.

The program which accomplished stage one segmen­tation was written in a variant of PL/I using recursion to handle the interaction of morphological variants, alter­nate spellings, and reconstruction of compounds. The results have been used directly for several WordSmith applications.

Once we had the dictionary entries segmented into DAM files, they were ready for the second stage of normalization: identifying and labelling the different types of data in the entries in a standard and consistent way. As we set out to create data bases from the first set of DAM files, it quickly became apparent that the usual data base architecture, characterized by a fixed number of fields for each entry and a fixed amount of space for each, would never work for dictionary entries, because there might be one value for some fields (like spelling), but multiple values for others (like part-of-speech, translation, etc.), or no value at all for still others (e.g. alternate spelling). Moreover, some information, like translation, is associated with values for other informa­tion, like part-of-speech. These facts lead to the obser­vation that dictionary entries are best represented as shallow hierarchies of attribute-value pairs. We will return to this observation in the section on "lexical data bases".

We have developed a parser, the goal of which is to convert each dictionary entry into a tree structure where at each node there is an attribute name and a value, which can be a simple value, a list of attribute-value pairs, or an entire subtree. Figure 1 contains the result of applying the parser to the entry for alert in the Collins English-Italian dictionary. The attributes in the figure should be familiar to users of bilingual dictionar­ies; they include hyphenation code ("hyph"), undeco-ded pronunciations ("pron"), homograph numbers ("hnum"), part-of-speech ("pos"), translations ("spel" under "xlat"), collocations and examples in the source language ("gloss" under "xmp"), and translations of collocations and examples ("tran" under "xmp").

Figure 2 shows how the above-mentioned combina­tion of the multiple entries for quack and other words brings about the creation of a "superhom" node. The superhom ("superscript homograph") node serves to separate the multiple entries and preserve the original sense numbers. This is important because cross refer­ences in other entries may contain superhomograph numbers as well as sense numbers. Even in entries where there is only one sense, homograph, or superho-

[gloss=to ~ sb (to sth)] tran=avvisare qn (di qc) tran=avvertire qn (di qc) [gloss=to ~ sb to the dangers of sth] [tran=mettere qn in guardia contro qc]]]]]

Figure 1. Normalized Collins English-Italian entry for alert.

mograph, these attributes are retained to make access logic consistent with other entries.

The design of the dictionary entry parser began with a familiar model for parsers of English sentences: a set of rules supported by a parsing engine. Two such models were available to us: a bottom-up, all-paths strategy offered by PLNLP (Langendoen and Barnett(to appear)), and a top-down depth-first approach offered by logic grammars written in Prolog (McCord(1987)). We have versions in PLNLP and VM/Prolog, both of which are basically adequate. The version which sup­ported the work reported here is the Prolog version; however, we are concerned about future problems with the top-down approach when we have to process input that is corrupt. Usually for lexicographical processing, one should not have to worry about large amounts of ill-formed input, but the complex history of some of our tapes forces us to consider possible recovery strategies for text with missing or corrupt font-codes. At that point, it may be necessary to use the bottom-up ap­proach and some recovery strategy analogous to parse-fitting (cf. Jensen, et al.(1983)).

, Nicolctta Calzolari, Martin S. Chodorow, Judith L. Klavans, Mary S. Neff, Omneya A. Rizk

Figure 2. Normalized Collins English-Italian entry for quack.

Grammars for dictionary entries and grammars for natural language sentences differ in three important ways: (1) entries can be huge: some are longer than 5000 bytes; (2) there is a different definition of a token, and (3) a dictionary grammar can be to a large extent deterministic. A consequence of (1) is that it takes a large amount of storage to parse an entry. The Prolog version now runs on a 4-megabyte virtual machine under VM/CMS. The motivation for (2) comes from the fact that the entries consist of text interspersed with font-change codes, which are not required to be delim­ited with blanks. Whereas for ordinary language, the tokens are strings between blanks, for dictionary entries they are the strings of characters between a defined set of delimiters, usually font-change codes, but occasion­ally also semicolons, periods, and complex character strings. The delimiters are tokens as well. In view of (3), one might well ask why anyone would use a powerful tool like PLNLP or a logic grammar to accomplish something that does not need such powerful support. The answer is that we now have a dictionary parsing engine, a formalism for rules, and an easily modifiable set of rules, all achieved in a shorter development time than would have been accomplished with a conventional programming language. Furthermore, stage two of the acquisition of new dictionaries now requires only writing, testing, and running a new set of grammar rules.

The parsing engine and support code are written in VM/Prolog. The rules, which may be different for each dictionary, are written using a formalism in the spirit of the "definite clause grammars" (DCG's) of Pereira and Warren (1980) and "modular logic grammars" (MLG's) of McCord (1986). A rule compiler translates the rules into Prolog code; this compiler owes much to the groundwork laid by McCord. Significant to McCord's work and necessary for ours here is the distinction between two different kinds of rules, one which pro­duces a structural node (strong non-terminal), and one which produces an intermediate list of nodes (weak non-terminal). The dictionary entry rule compiler sports several new features required by dictionary entry pars­ing. These include an 'ignore' operator, which causes tokens in the input to be ignored by the automatic structure builder, an 'insert' operator which causes new elements to be inserted, and the efficient implementa­tion of rules consisting entirely of optional elements. This generalized rule-driven parsing engine will be used to help develop entry grammars for all our dictionary resources; a small tool kit for this is already available, and other tools will be added as development progresses.

We illustrate the rule formalism with the following two rules from the grammar for the Collins Italian-English dictionary:

senslist(num) = = > sens(num) : (senslist(num) | nil).

The first rule says that an unnumbered homograph consists of a number of optional elements, of which at least one must be present: pronunciation; choice of part-of-speech or verb-reflexive or none; morphological information; selection of perfect tense auxiliary; a cer­tain kind of cross reference (marked with V); a gloss; or a list of senses. The horn constituent is declared elsewhere as a strong nonterminal; a node is created for it, with daughter nodes as defined to the right of the arrow.

Prolog backtracking conventions apply; the first suc­cessful parse of an entry is used, even if there might be more than one possibility. In the case of most optional constituents, the existence of the constituent is tried before its absence. In the case of the gloss in the first rule, however, the correct result is obtained if the absence of the gloss is tried first.

, Nicoletta Calzolari, Martin S. Chodorow, Judith L. Klavans, Marys.Neff, Omneya A. Rizk

The second rule says that a list of numbered senses consists of a numbered sense and a numbered sense list (or nothing). The recursive property of this rule allows the assembling of any number of numbered senses. The senslist constituent is declared as a weak non-terminal, so that all the sub-constituents defined by the rule will appear as sister nodes at the same level, immediately dominated not by senslist, but by its parent, horn.

The present version of the grammar has been run on parts of the Collins Italian-English and English-Italian dictionaries with a success rate of about 95%. Some of the entries were quite large. The high success rate is partly due to the consistency and discipline in the formatting of the Collins Italian dictionaries. The rules are still being improved. There are several reasons for the remaining 5% — and, indeed, also for some of the ad hoc devices we have used to successfully parse some entries. Sometimes the primary clues to parsing, the font change codes, can be missing because two discrete data segments are contiguous and appear in the same font. Alternatively, the font changes can actually occur in the middle of a coherent data segment. For example, the translation for English queen bee is given as "ape f regina inv". Here, "f ' is the feminine gender marking for ape, and "inv" (i.e., "invariable") indicates that the word regina remains invariable in the plural form.

In this case, there is grammatical information about each of the pieces of the collocation. In other cases, the first piece of grammatical information attaches to the first word, and the second to the entire collocation. To take a second example, the locution "x or y" is often used for phrases in all languages. The problem is that both "x" and "y" can be gapped, or otherwise abbre­viated, by deletion of words identical to those found on the other side of the "or". An example is in the translation given for Italian si sta facendo buio: "it is growing o getting dark" (shorthand for "it is growing dark o it is getting dark").

For cases like these, we would ideally like to identify and extract all the information immediately. It would be nice to separate the grammatical information from ape regina, in the first case, and to reconstitute the transla­tions it is growing dark and it is getting dark, in the second. However, we recognize that solving all such problems is a never-ending process and would indefi­nitely delay the time that any information is available for lexicological analysis. Consequently, we make a practical distinction between the acquisition and analy­sis phases of our work: information that is not easily decipherable using our entry parsing tools is left in its undeciphered form in the initial lexical data base that we create. Thus, for our examples, we should have the following attributes.

Later work can be devoted to the detailed analysis of such complex attributes. The results of such analysis will be used to create more refined versions of the initial lexical data bases.

2.2. Lexical Data Bases and the Lexical Query Language.

The lexical data base (LDB) facility is designed to store dictionary information and to provide fast, flexible, and convenient access to it. LDBs differ from DAM files in that the entries are structured into hierarchies of at­tribute-value pairs, such as those that result from the entry parsing described in the previous section. The lexical query language (LQL) allows users to query, modify, and format those entries. A prototype LDB/LQL system has been built as a testing ground for various concepts and algorithms and is currently in use, as described in sections 4 and 5. The final version of the facility is still under development.

Dictionary entries are typically organized as shallow hierarchies of attribute-value pairs with a variable num­ber of copies of certain nodes at each level. The hierarchical nature of entries stems from the fact that values can be entire subtrees. Thus, after processing by the dictionary entry parser described in the previous section, the entry for quack from the Collins English-Italian dictionary can be stored in an LDB as shown in Figure 3. In this hierarchy, the "hdw" terminal node contains the headword of the entry. Dictionary workers are familiar with the notion of hierarchically structured entries of this type.

It is important to distinguish between the type of hierarchies found in dictionary entries and those, for example, that characterize the syntactic structure of English sentences. Both types of hierarchy are, in principle, of unbounded size. Dictionary entries achieve their unboundedness by iteration: a bilingual dictionary entry may have any number of homographs, which may, in turn, have an arbitrary number of senses, and so forth. Syntax trees achieve their unboundedness both by iteration (e.g., coordination) and by recursion (e.g., self-embedding). The observation that dictionary entry hierarchies may only iterate is useful for defining lexical data bases. All entries in an LDB can be characterized by a "design" which is stored once for the data base. The design is a single hierarchy which serves as a "grammar" for any instance of an entry by naming the possible parent, siblings, and children of each node type that may occur. Actual LDBs are created by storing hierarchically formatted entries in a DAM file with the design stored as control information. Each entry is stored with its headword as key. Alternate access paths can be established by building indexes on other at­tributes.

, Nicoletta Calzolari, Martins.Chodorow, Judithl.Klavans, Marys.Neff, Omneyaa.Rizkl. s. a.

+-superhom +-hum 1 +-hyph 0.

+-superhom +-hum 2 +-hyph 0.

+-xlat I +-note pej I +-spei ciarlatano/a +-note fam: doctor +-spei dottore/essa

Figure 3. LDB representation for the Collins English-Italian entry for quack.

LQL allows the user to specify conditions on the attributes of LDB entries. Only those entries which satisfy all conditions become part of the query answer. Further, the user specifies which attributes of the suc­cessful entries remain in the answer and what their output format will be. The query is stated as entries in the nodes of a two-dimensional representation of an LDB's design. The query syntax is reminiscent of the syntax of the Query-by-Example (QBE) data base query language (Zloof(1974)). Example-elements, denoted by a leading underscore, are used to relate values of attributes in a query tree to conditions in a "condition box", to display positions in an "output box", and to attribute values in other dictionary entry trees.

Part (a) of Figure 4 shows a query which will list all words which are both nouns and verbs in English together with their translations in the Collins English-Italian dictionary. The condition on the noun part-of-speech attribute is simple (it must be "n" for this data base) and is entered directly in the tree. The condition on the verb part of speech is more complex, and the example-element _vpos is used to relate those attribute values to the condition box, where they are tested for equality with either "vi" or "vt". The example-ele­ments _word, _ntran, and .vtran are used to map an­swers from the hierarchy into the output box where their eventual output format is schematically repre­sented. Part (b) shows sample entries from the answer to this query when applied to the Collins English-Italian dictionary. Such a query might be useful, for example, in a study of the relation between homography in English and derivational morphology in Italian.

Figure 4. An LQL query.

LQL conditions specify tests to be performed on the values of attributes in the data tree to which nodes in the query tree may be mapped during query processing.

, Nicoletta Calzolari, Martin S. Chodorow, Judith L. Klavans, Mary S. Neff, Omneya A. Rizk

Terminal nodes may be tested using a variety of string and arithmetic operations. The current prototype imple­mentation includes the built-in functions of the REXX programming language (IBM (1984)). Non-terminal nodes may only be tested for equality or inequality with other nodes of the same type. All nodes may have aggregate functions (e.g., count, maximum, minimum, etc.) applied to them and the results may either be tested in conditions or be output as part of the query answer. Nodes may also be tested for a null value (i.e., non-occurrence in a particular entry).

As in the query tree itself, the output box may contain both constants and example-elements. The con­stants define boilerplate material used to label the variable instantiations of the example-elements, as in Figure 4. Such labelling is perhaps more useful when dictionary entries are presented one at a time, as in the WordSmith on-line dictionary system (Neff and Byrd(1987)).

A realistic data base system must provide facilities for creating and maintaining data bases. LDB creation is usually a bulk operation and has been discussed in the previous section. LDB maintenance, on the other hand, can benefit from the flexibility provided by combining a powerful query processor with the capability to insert, delete, and update parts of the LDB entries. LQL offers this flexibility by providing the operators "i." (for "insert"), "d." (for "delete), and "u." (for "update"). These are also familiar Query-by-Example concepts and are described in a relational context in IBM(1978), and in the context of LDBs in Neff, et al.(1988). LDB modification will not be discussed further in this paper.

We envision that LQL will be available in multiple environments. Naturally as a stand-alone processor, it can be used to specify batch query and modification operations against entire lexical data bases. This pro­cessor and the flexibility with which it can produce results tailored to a specific problem make LQL a valuable tool for lexicological research. We also plan to use LQL as an entry formatting and filtering mechanism for the WordSmith on-line dictionary system.

3. Tools and Methodologies. 3.1. tools.

In this subsection, we give an alphabetical listing of the major computational tools, other than the dictionary entry parser and LDB/LQL, at our disposal for doing lexicological analysis of our MRDs. The tools are described in more detail in the sections of the paper where their use in a certain application is presented. Further detail is available in the references. Minor tools used for individual applications are not shown here.

The tools are in the form of a variety of programs written by different members of the Lexical Systems project. The programming languages used include IBM's EXEC2, REXX, PL/I, and VM/Prolog lan­guages. What makes them coherent are the fact that we use the Dictionary Access Method and its utilities as a common interface to the data and that we share a common set of methods and goals when we apply them.

DAM. The Dictionary Access Method (Byrd, et al.(1986b)) provides random and sequential access to dictionary entries associated with words which serve as search keys. The sequential access can be modified to observe "correct" alphabetical orders in any of several languages. An associated set of utilities makes the management of DAM files (including creation, updating, and compression) especially convenient.

DICTUTIL. DICTUTIL is a menu-driven interface to the prototype implementation of the lexical data base system. It allows the analyst to create LDBs using output from the parsing of Collins bilingual dictionary entries and to execute simple queries against those LDBs. The output from a DICTUTIL query is stored as an index to the original LDB (i.e., as a set of value-headword pairs).

Filtering. Filtering, described in Chodorow, et al.(1985), is a method for expanding a set of words each of which has a given property. It uses the hypernym relationship, produced by Head Finding, or the synonym relationship to find new words that are related only to words already in the set. Such new words are added to the set and the process repeats until convergence. See section 3.2, below.

Head Finding. Head finding, also described in Chodo­row, et al.(1985) and in Calzolari( 1984a), is a method for automatically discovering the hypernym (or genus term) for a word by analyzing its dictionary definition. It is based on the observation that, at least for nouns and verbs, the syntactic head(s) of a definition is(are) usu­ally the hypernym(s) of the word being defined. See section 3.2, below.

Matrix Building. The Matrix Building program takes ordered pairs of words that bear a given relationship to one another (e.g., X is a synonym of Y) and constructs a matrix in which each X word is represented by a row, each Y word is represented by a column, and the relationship between them is indicated in the cell that is formed by the XY intersection. Synonym matrices are used for clustering synonyms into senses. Translation matrices (where X is a translation of Y) are used for transferring and analyzing sense information using bi­lingual dictionaries. See sections 5.1 and 5.2, below.

Sprouting. Sprouting, described in Chodorow, et al.(1985), is a method for exploring the hypernym, hyponym, and synonym relations. It finds words related to some initial seed by computing the transitive closure over the relation (i.e., "sprouting a tree") beginning from the seed. See 4.1, below.

TUPLES. TUPLES (Byrd(1986b)) is a text analysis system for finding frequent words and phrases in ordi­nary text. It uses UDICT to provide lemma normaliza­tion and base normalization as it compares strings from

, Nicoletta Calzolari, Martin S. Chodorow, Judith L. Klavans, Mary S. Neff, Omneya A. Rizk

the text. TUPLES is used in the analysis of dictionary definitions for locating and counting defining formulae and presenting them in a key-word-in-context format.

UDICT. UDICT (Byrd(1986a)) is a computerized lexi­con system. It supplies a rich variety of lexical infor­mation to natural language processing applications. Of particular interest for lexicography is the fact that UDICT performs morphological analysis of words hav­ing inflectional and derivational affixes as well as for compound words and multi-word entries. It also has a facility for producing correctly inflected form of English words. UDICT is a component of TUPLES and Word-Smith.

WordSmith. WordSmith is an on-line dictionary system, described in Neff and Byrd(1987), which allows flexible access to dictionaries stored as DAM files and lexical data bases. Although not explicitly mentioned in the remainder of the paper, it is an important reference tool for all of our research. Among its capabilities, Word-Smith provides access to:

• definitions, synonyms, and etymologies from Web­ster's Seventh (Merriam(1963)),

• pronunciations from Webster's Seventh and rhymes based on them,

• definitions and grammatical information from LDOCE (Longman(1978)),

• synonyms from the Collins Thesaurus (Collins(1984)),

• entries from the Collins bilingual dictionaries for English/Italian, English/French, English/Spanish, and English/German (Collins(1971, 1978, 1980, 1981)).

3.2. Methodologies.

At the beginning of our work, we dealt with words as undifferentiated wholes. That is, we only stored lexical information for a word when it applied to all of the word's senses for a given part of speech. Even though this practice made us somewhat uncomfortable, it al­lowed us to make significant progress in dictionary analysis. The primary reason, as shown in Figure 5, is that most words in the dictionaries we analyze are monosemous; that is, they have just one sense. Of the approximately 60,000 headwords in Webster's Seventh having fewer than ten senses, almost two-thirds of them (over 38,000) are monosemous. Further analysis, not given here, shows that, even among words that have multiple parts-of-speech, each part-of-speech is also typically monosemous. This further increases the pro­portion of the dictionary entries that we can handle with our earliest techniques. Analysis of Zingarelli(1971) reveals that the facts for Italian are analogous to those for English: over all parts-of-speech, 61.7 percent of the Italian entries were monosemous; 38.3 percent were polysemous. Further details of the Italian study are presented in Calzolari(1980).

These statistics explain why MRD-based methodol­ogies such as head-finding, sprouting, and filtering are as productive as they are, even though they don't 59934 Total words Figure 5. Number of senses per word in Webster's Seventh.

handle polysemy correctly. They are described in sec­tion 3.2.1. Despite these encouraging statistics, how­ever, it is ultimately necessary to be able to deal correctly with the multiple senses of polysemous words. The need is especially acute because the most fre­quently used words tend to also be the most polyse­mous. Two specific problems associated with polysemy in MRDs are the mapping problem and the addenda problem. Section 3.3 describes these problems and offers solutions for them.

3.2.1. head finding, sprouting, and filtering.

One goal of our research is to extract semantic and syntactic information from standard dictionary defini­tions for use in constructing lexicons for natural lan­guage processing systems. Dictionaries are rich sources of detailed information, but in order to use the informa­tion for natural language processing, it must be orga­nized systematically.

Amsler(1980) demonstrates that additional structure can be imposed upon a dictionary by making certain assumptions about the ways in which definitions are constructed. Foremost among these assumptions is that definitions consist of one or more "genus" terms, which identify superordinate categories of the defined word, and "differentia" which distinguish this instance of the superordinate categories from other instances. By man­ually extracting and disambiguating genus terms for a pocket dictionary, Amsler demonstrated the feasibility of generating semantic hierarchies.

It has been our goal to automate the genus extraction and disambiguation processes so that hierarchies can be generated from full-sized dictionaries, such as Web­ster's Seventh New Collegiate Dictionary. These efforts are described in detail in Chodorow, et al. (1985). They begin with Head Finding.

In the definition of car as "a vehicle moving on wheels", the word vehicle serves as the genus term, while "moving on wheels" differentiates cars from some other types of vehicles. Taken as a group, all of the word/genus pairs contained in a normal dictionary for words of a given part-of-speech form what Amsler(1980) calls a "tangled hierarchy". In this hier­archy, each word constitutes a node whose subordinate

, Nicoletta Calzolari, Martins.Chodorow, Judithl.Klavans, Marys.Neff, Omneyaa.Rizkl. s. a.

nodes are words for which it serves as a genus term. The words at those subordinate nodes are called the word's "hyponyms". Similarly, the words at the supe­rordinate nodes for a given word are the genus terms for the various sense definitions of that word. These are called the given word's "hypernyms". Because words are polysemous any word may have multiple hy­pernyms; hence the hierarchy is "tangled".

Our automated approach to finding the genus terms in definitions is based on the observation that the genus term for verb and noun definitions is typically, though not always, the syntactic head of the defining phrase. This reduces the task to that of finding the heads of verb phrases and noun phrases. The syntax of the verb phrase used in verb definitions makes it possible to always locate its head with a simple heuristic: the head is the single verb following the word to, or if there is a conjunction of verbs following to, then they are all heads.

Head finding is much more complex for noun defini­tions because of their greater variety, but we have taken advantage of the fact that dictionary definitions are written in a special and predictable style to develop a heuristic procedure for finding the head. The procedure can be described briefly as follows. First the substring which must contain the head is found. This substring is bounded on the left by a word which obligatorily appears in prenominal position: a, an, the, its, two, three, . . ., twelve, first, second, and so forth. It is bounded on the right by a word or sequence that can only appear in postnominal position, such as a relative pronoun, a preposition, or a present participle following a noun.

Once the substring is isolated, the search for the head begins. Typically, but not always, it is the rightmost noun in the substring. If however, the substring con­tains a conjunction, each conjunct is processed sepa­rately, and multiple heads may result. If the word found belongs to a small class of "empty heads" (words like one, any, kind, class, manner, family, race, group, complex, etc.) and is followed by of, then the string following of is reprocessed in an effort to locate addi­tional heads. The accuracy of head-finding for nouns was approximately 98 percent, based on a random sample of the output.

The word defined and the head of its definition are the raw materials for two semi-automatic processes which make explicit an important part of the lexical organization and featural information that are held implicit in dictionaries. The first of these is sprouting.

Sprouting, which derives its name from the action of growing a semantic tree from a specified root, uses the results of head-finding organized into a "hyponym index", in which each word that was used as a genus term is associated with all of its hyponyms. Thus, vehicle would have an entry in the index which reads (in part):

vehicle: ambulance . . . bicycle . . . car . . . tanker . . .

For a given part-of-speech, the hyponym index needs to be built only once.

When invoking the SPROUT program, the user se­lects a root from which a semantic tree is to be grown. The system then computes the transitive closure over the hyponym index, beginning at the chosen root. In effect, for each new word (including the root), all of its hyponyms are added to the tree. This operation is applied recursively, until no further new words are found.

The interactiveness of the sprouting process results from the fact that the user is consulted for each new word. If the user decides that the word does not belong to the tree being grown, it can be pruned. These pruning decisions result in the disambiguation of the tree, though clearly not by means of an automatic process.

The output of a sprouting session, then, is a disam­biguated tree extracted from the tangled hierarchy rep­resented by the hyponym index. The words it contains all have at least one sense which bears the property for which the root was originally selected. It is important to note that any serious use of sprouting to locate all words bearing a particular semantic feature must involve the careful selection and use of several roots, because of the variety of genus terms employed by the Webster's lexicographers. This will be quite obvious in the case studies cited below.

Filtering, like sprouting, results in lists of words bearing a certain property (e.g., [+human]). Unlike sprouting, however, filtering only picks up words all of whose senses have the property. It is based on a hypernym index (the inversion of the hyponym index), in which each word is listed with its hypernyms, as in the example given here:

vehicle: agent equipment means medium The filtering process begins with a "seed filter" consist­ing of an initial set of words all of whose senses bear some required property. The seed filter may be obtained in any manner that is convenient. Several approaches to setting up a seed filter are discussed in the case studies that follow. Given the filter, the system uses it to evaluate all of the words in the hypernym index. Any words, all of whose hypernyms are already in the filter, become candidates for inclusion in the filter during the next pass. The user is consulted for each candidate, and may accept or reject it. Finally, all accepted words are added to the filter, and the process is repeated until it converges.

Like sprouting, filtering produces a list of words having some desired property. In the case of filtering, however, the resulting words have the property in all of their senses.

3.3. Multi-dictionary Lexicology.

When lexicographers create dictionaries, they identify varying numbers of senses for words based on such

, Nicoletta Calzolari, Martins.Chodorow, Judithl.Klavans, Marys.Neff, Omneyaa.Rizkl. s. a.

Collins's sense 2. Here the binary mappings are sepa­rated by semicolons. Each mapping consists of two lists of senses separated by colons. The leftmost list is for Webster's Seventh, the rightmost for Collins English-Italian. Since the mappings are symmetric, the assign­ment of left or right to a particular dictionary is arbi­trary. Each list, in turn, is a comma-delimited set of sense numbers. A list may be null if one dictionary excludes a sense contained in the other. (It is important to note, however, that the success of the mapping strategy depends on the MRDs involved having fairly complete coverage of the language. Arbitrary omission of word senses will, in general, lead to incorrect results.)

With respect to the number of non-null word senses in the two entries for a given word, there are four possible types of mappings, as shown in the table in Figure 7.

good as having a single translation. Likewise, if a single translation is polysemous in Italian but all senses bear the relevant feature, then the observations still hold. Further generalizations should yield a useful procedure for instantiating (part of) the desired mapping.

Many such procedures, possibly augmented by some manual labor, will be required to build an entire mapping. In particular, we expect that for highly polysemous and common words (such as run, go, turn, thing, make, be, etc., in English) the mappings will have to be done by hand. Once obtained, however, the mapping will be per­manent and can be used for a variety of purposes, some of which will be discussed in sections 4 and 5 of this paper.

3.3.2. The Addenda Problem.

Sense mapping is not the only aspect of work with computerized dictionaries where the ability to identify

Figure 7. Types of mappings.

We can imagine very simple strategies for populating the mapping file for mappings of types 1,2 and 3. We simply build the mapping entry shown in the fourth column of the figure. Given our previous observation that most entries in a typical dictionary are monose­mous, these strategies will account for a significant portion of the mapping file entries.

Mappings of type 4 will require more involved strat­egies. As an example of one such strategy, consider the example in Figure 6. If II is [+human] and is the only translation given for the first sense of El in Collins English-Italian, then the first two senses of El in Web­ster's may be mapped to the first sense of El in Collins. Likewise, if 12 is [+instr] and is the only translation given for the second sense of El in Collins, then we may assume that the third sense of El in Webster's (which is [+instr]) can be mapped to the second sense of El in Collins. A procedure based on these observations will yield the mapping shown before, namely:

Note that the procedure will include generalizations of some aspects of the observations made above. For example, if Collins gives multiple translations for a single sense of a word, and if all are [+human] that is as word senses is critical. Activities which develop dictio­nary information—either by dictionary and text analysis or by transferring the information from other dictionar­ies—must ultimately have word senses as targets for that information. The information to be associated with the target dictionary is generally represented either as features, or as relations (i.e., attribute-value pairs).

The dictionary to which the information is to be attached may either be a published dictionary, like Webster's Seventh, or a dictionary intended for com­puter use, such as UDICT. In the former case, the senses are those established by the lexicographers. In the latter case, the senses are those which meet the needs of the application(s) which the dictionary serves.

We will also need to decide whether the new infor­mation should actually be added to the existing file for the target dictionary or, rather, whether it should be kept in a separate "addendum" to the base file, leaving the base file unchanged. Ignoring, for the moment, the first possibility, we propose the following strategy for adding new information as an addendum to an existing base dictionary.

The information must be addressable by word and sense number, where the sense numbers are those given in the base dictionary. However, the need to access

, Nicoletta Calzolari, Martin S. Chodorow, Judithl.Klavans, Mary S. Neff, Omneyaa.Rizka.

Figure 8. Results of processing "act" and "state" hyponyms.

Once we derived the verbs, we were then ready to test the original hypothesis. If the hypothesis is correct, verbs underlying nouns which have state as a hypernym are likely to be stative. Similarly, verbs underlying nouns which have act as a hypernym are likely to be active.

We constructed the following syntactic tests from Joos (1968) and Quirk and Greenbaum (1972) which are said to characterize stativity: (1) the simple present progressive, (2) the present progressive with the adver­bial modifier "more and more each day" in order to test a continual or process reading, and (3) the imperative. We built a sentence frame generating tool which takes as input each verb to be tested. Test sentences are constructed and the user is asked to respond whether a sentence is acceptable, unacceptable, or questionable. There is also opportunity for comments. The results are placed in a matrix containing the collected judgements for a list of verbs.

Before running our lists of derived verbs through the sentence frame tool, we decided to test it with a verbs which have been said to be undeniably stative and active. We took seventy stative verbs and forty action verbs from Joos (1968) and Quirk and Greenbaum (1972).

To illustrate, given the sample list of three verbs in (24), frame tests were automatically generated for each verb in turn, as shown in (25)-(27). The list in (24) has three verbs: one said to be stative (resemble), one active (throw), and one process (feel). Results are tabulated for each response, and given in (28).

(24) resemble throw feel

TEST 1: He is resembling his mother today. TEST 2: She is resembling his mother more and

more (each day). TEST 3: Resemble your mother now!

TEST 1: He is throwing the ball today.

TEST 2: She is throwing the ball more and more

TEST 1: He is feeling happy today.

TEST 2: She is feeling happier more and more

The table given in (28) shows the results of applying the frame tests to the verbs given in (24): The expected result if the distinction were as it is said to be is given in (29).

(28) Results of Testing Verbs for Stativity:

Notice that the patterns in (28) do not conform to the expected patterns in (29).

A more complete set of responses using a random

, Nicoletta Calzolari, Martin S. Chodorow, Judith L. Klavans, Mary S. Neff, Omneya A. Rizk

subset of the verbs taken from Joos (1968) and Quirk and Greenbaum (1972), again failed to show any clear patterns or any clear distinctions among sets of verbs which are supposedly active and stative.

Despite the fact that textbooks set out explicit dis­tinctions for verb types, there appear to be no signifi­cant clear-cut core set of verbs in each category. Rather, there appear to be situations in which a given verb has a stative reading and other situations in which the same verb has an active reading. What is worse, these "situations" can be either semantically or syntac­tically conditioned.

Choice of Auxiliary Verb in Italian. We next explored the possibility of a correspondence between verbs which tend to be active in English and those which select for the auxiliary avere in Italian, and similarly for the stative verbs in English and those requiring essere. Grammar books often state that in Romance languages, the choice of auxiliary verb depends on whether the verb is stative or active. We performed two tests of this hypothesis. The first was to extract some verbs from the Collins English-Italian dictionary which were labelled as taking essere "to be". Verbs are not marked for avere "to have" since it is the unmarked case. We then passed these verbs through a frame test, identical to the one described above (except in Italian, of course.) Results showed that less than 10% of verbs with auxil­iary essere "to be" pass the test for stativity. We also took the translations of the core set of [+active] and [+stative] verbs from the sample verbs given in Joos(1968) and in Quirk and Greenbaum(1972). We then examined the corresponding Italian verbs taken from the Collins English-Italian dictionary to determine which took essere "to be" and avere "to have". Results showed that almost all of them took the auxil­iary avere "to have". Thus, we were unable to correlate activity and stativity with translation of verbs taking essere "to be" and avere "to have".

Conclusion. The active-stative distinction appears to be an aspectual feature related to discourse. It is not a lexical feature, although certain verbs occur more nat­urally whereas others are more tightly constrained. This means that the properties active and stative are quali­tatively different from the inherent features [+human] or [+male] for nouns, or from the selectional features [+thatcomp] (takes a that complementizer), or [requires a human subject] for verbs. For example, a verb which indicates a true state, such as "resemble", is more likely to occur in a sentence where stative aspect is present, but not necessarily. This being the case, then it is strictly speaking not the job of the lexicographer alone to identify such features. Rather, it is the job of the syntactician and semanticist to identify such con­texts, along with the lexicographer who should identify verbs that are potentially active or stative in a given context.

4.2. Synonymy and the Problem of Sense Discrimination.

As described above, sense mapping between dictionar­ies is one of the most important and challenging prob­lems facing computational lexicography. A related problem is sense disambiguation within a single dictio­nary. Webster's Seventh Collegiate Dictionary (W7) defines car as a kind of 'vehicle' but does not indicate that it is sense 4 of vehicle which is intended and not senses 1, 2, or 3. This lack of an explicit indication of the hypernym's intended sense limits the usefulness of the hypernym relation as we have currently extracted it because hypernymy is actually a relation between word senses, not a relation between words. As a conse­quence, our sprouts from hypernyms require human intervention to insure that the proper sense of each node is followed. Without automatic sense disambiguation, taxonomic sprouting is thus relegated to the status of a semi-automatic (rather than a fully automatic) process­ing tool.

It has been suggested (Lesk(1986)) that sense disam­biguation of hypernyms might be achieved by compar­ing the differentia (the words which appear with the hypernym, e.g., "moving on wheels" in the definition of car as a "vehicle moving on wheels") to the words that appear in the definitions of the hypernym's senses. The sense definition which most closely matches the differentia would then be selected as the intended sense of the hypernym. The measure of closeness might be something as simple as the number of content words found in the intersection of words from the differentia and the sense definition. However, to the best of our knowledge, this has not yet been carried out automati­cally on a large scale. Our disambiguation of synonym senses has followed a somewhat different course based on the assumption that synonymy is a symmetric rela­tion between word senses.

Our procedures for sense disambiguation have been used to process The Collins Thesaurus (CT), which is stored as a DAM file with 16,700 keyed records con­taining a total of 278,000 synonym tokens. Each key is a headword or a boldface run-on word from a main entry. The record associated with each key holds the synonym tokens organized by part-of-speech of the key and, within each part-of-speech, by sense number. Many of the entries on the original tape and the printed version of CT are not marked for part-of-speech, so it was necessary to use UDICT to analyze the parts-of-speech of the synonyms listed under each sense num­ber. From this it could then be inferred under which part-of-speech the sense number belonged (Chodorow and Ravin (1987)).

In a dictionary-style thesaurus such as CT, an entry A may have word B listed as a synonym of its nth sense, and entry B may have word A listed as a synonym of its mth sense. Based on the assumption that synonyms are symmetric, we can mark B in entry A as the mth sense

, Nicoletta Calzolari, Martins.Chodorow, Judithl.Klavans, Marys.Neff, Omneyaa.Rizkl. s. a.

the cells contain either a "1" or a "0" to indicate the presence or absence of a synonym relationship between the row word and the column word. Finally, an algo­rithm (such as the one reported in Marcotorchino(1986)) is applied to the matrix to produce the clusters.

There are several advantages to organizing syn­onyms into clusters, as suggested by Warnesson (1985). (1) Space can be saved by having all members of the cluster point to a single location which represents the sense they share, rather than listing in the entry for each member all the other members. (2) The word having the largest number of connections within the cluster and the fewest connections outside it might be offered to the user as the synonym which is most representative of the sense. (3) It is possible to compute the strength of association between clusters based on connections be­tween member and nonmember words. Strong associa­tions should exist between closely related senses, and weak associations between more distant ones. In this way, it might be possible to generate a hierarchical structure of senses from the bottom up, i.e., based on the actual patterns of synonym connections found in the thesaurus. (4) An on-line thesaurus in which sense-disambiguated words have been structured into clusters provides the best of both forms of organization.

5. Applied Computational Lexicology: Bilingual Dictionaries. 5.1. transfer of lexical information through bilingual dictionaries.

The goal of the experiments described in this section is to take information available from a monolingual dictio­nary for a given language and to use a bilingual dictio­nary as a transfer device to enable the same information to be transferred to a monolingual dictionary in a second language. The clearest and most successful attempt at this is reported in the first section below on the semantic feature [+human]. Other more speculative attempts with synonyms and with the syntactic feature [+thatcomp] resulted in partial successes and are re­ported in later sections.

5.1.1. [+human] nouns.

Using sprouting and filtering, we extracted a set of English nouns with the feature [+Human] in all senses. Next, we obtained the set of Italian translations for those nouns from the Collins English-Italian dictionary. Finally, we re-translated the Italian nouns back into English, using the Collins Italian-English dictionary.

Our results were very positive. Of the 321 original English words, 296 Italian words were obtained in the forward (E-I) direction; 63 multi-word translations were also found. Re-translating these 296 Italian words in the backward (I-E) direction yielded 373 English words plus 45 English multi-word translations. Examining the two lists of English words showed that out of the 321 original, 157 were not among the final re-translations. Furthermore, out of the 373 re-translations, 167 were new words not in the original.

An analysis of the words obtained gave two clear results. First, only six out of the 296 Italian words obtained in the forward (E-I) direction did not have the feature [+Human] in all of their senses. This feature can therefore be very well transferred (given our constraint on the polysemy of the source data) to the other monolingual dictionary. Second, well over 90% of the 167 new English re-translations were also [+Human]. This demonstrates the effectiveness of this procedure in the task of enlarging the set of words having a given feature.

The procedures used for transferring the [+human] feature from English to Italian can be viewed as in­stances of a special kind of sprouting called "bilingual sprouting". In this case, the binary relation through which we propagate features is the translation relation, rather than synonymy, hypernymy, or hyponymy. As with other forms of sprouting, we can view the transla­tion relation as transitive, and therefore we are able to propagate lexical information not only from one lan­guage to another, but also back to the original language. Of course, again as with normal sprouting, we must take care to control for the effects of polysemy. These considerations lead to the definition of two types of bilingual sprouting: unconstrained and constrained.

Unconstrained bilingual sprouting. This form of bilin­gual sprouting assumes that if a word has a particular feature in all of its senses, then any translations of that word that are monosemous will also bear that feature. This, of course, excepts grammatical features which only apply in one of the two languages involved. So, for example, we could not propagate grammatical gender from Italian to English.

A reasonable procedure that implements this kind of sprouting would begin with a list of words in the first language (LI) bearing the feature to be propagated. For each word, assign the feature to any of its monosemous translations, given by the bilingual dictionary. This accomplishes the transfer of the feature to the second language (L2). We can continue by inspecting the bilingual dictionary for the other direction (from L2 to LI). Since the words in L2 are all monosemous, any translations from them back into LI are also likely candidates for the feature.

Despite the care taken by this procedure to deal only with monosemous words in L2, polysemy in LI and in the bilingual dictionary causes the procedure to yield lists which must be manually checked. However, the labor involved is much less than would be required to do the work entirely by hand. As reported above, when the procedure was used to propagate the [+Human] feature from English to Italian, we obtained a list of Italian words which were correctly marked in much more than 95 percent of the cases. Furthermore, when we reflected the feature back onto English (with the Italian-English dictionary) we obtained a new set of English words not on our original list. More than 90 percent of those were indeed [+Human] words.

Royj., Nicoletta Calzolari, Martins.Chodorow, Judithl.Klavans, Marys.Neff, Omneyaa.Rizkj. l. s. a.

The results we present should be viewed as prelimi­nary because we took only a small set of some ten English verbs and some twenty-five Italian verbs. The first observation is that some of the translations do not belong to the same semantic field as the word in LI. An example is add and addizionare. Addizionare means to "to add" in the mathematical sense, but does not mean "to add" in the sense of "comment". When we have lexical information applied to senses, disambiguated references in bilingual dictionaries (see section 5.2.1) and sense mappings (see section 5.2.2) such problems should disappear.

Second, there are the cases of a single entry in LI with multiple translations in L2. An example of that is anticipate:

There is a possibility of having a that clause for at least one of the translations in each group of translations. Further, it appears that the pattern could serve as a clue to pick which of the many translations is the one that corresponds semantically to the English word sense taking that. This applies to the first two test frames, i.e. columns one and two in the table. For example in the case of anticipate, aspettarsi and prevedere are the two translations with the meaning closest to anticipate when anticipate is used in the sense taking a that clause. Our conclusion is that this process is not feasible completely automatically, but that such a procedure could reduce much searching and guessing. Further, we believe we could profit by utilizing information on that clauses found in example sentences in the bilingual dictionary. This could provide an additional mechanism to elimi­nate non-corresponding translations.

5.2. Cross-References and Sense Disambiguation in Bilingual Dictionaries.

Since the Collins bilingual dictionaries play such an important role in our research, both as a conduit for lexical information transfer and as reference material for researchers, it is desirable to make them as accurate and useful as possible. This section discusses two methods of enhancing the contents of those bilingual dictionaries. The first method supplies sense numbers to the cross-references between the two sides of the dic­tionary. The second supplies "missing information" in the form of cross-references which should be there but are not.

5.2.1. disambiguating references in collins bilingual dictionaries.

The English-Italian side of Collins assigns sense num­bers to the English words. Similarly, the Italian-English side assigns sense numbers to the Italian words. A desirable enhancement to the entries in either side is to assign sense numbers to the translations so that they accurately point to the other side. Pictorially, we want to instantiate sense references represented by the ar­rows in Figure 9. This result is essentially equivalent to that discussed in section 4.2 for disambiguating the cross-references in the Collins Thesaurus (CT).

In terms of LDBs, we are proposing that the struc­ture of the current translation attribute "xlat" be en­hanced by the addition of a new "senseno" (sub)at-tribute identifying the sense number on the other side of the dictionary for the word given as the value of the "spel" (sub)attribute. The resulting "xlat" attribute will look like:

I E I I I I I I E I I_I_I I_l_I

+-spel +-senseno +- . . .

As with the mapping files, several procedures will be required in order to determine the correct values for these new "senseno" attributes. Some of these proce­dures will be trivial, as when the words involved are monosemous. Recall that, in the case of sense mapping with Webster's Seventh, the majority of words are monosemous. We expect to find the same situation in Collins. A version of the procedure given in Section 4.2 should supply many values. Specifically, if any sense of an English word, El, gives a particular Italian word, II, as its translation, and if one sense of II gives El as its translation, then we can mark the II translation of El (on the English-Italian side) with the sense number of II (on the Italian-English side) that had the reciprocal

Royj., Nicoletta Calzolari, Martins.Chodorow, Judithl.Klavans, Marys.Neff, Omneyaa.Rizkj. l. s. a.