<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="155"/><title>Selection Preference Basede Verb Sense Disambiguation Using WordNet</title><author surname="Ye" givenname="Patrick"><org  name="University of Melbourne" country="Australia" city="Melbourne"/></author></firstpageheader><frontmatter><p><b>Selectional Preference Based Verb Sense Disambiguation Using WordNet</b></p><p><b>Patrick </b><b>Ye</b></p><p>Department of Computer Science and Software Engineering University of Melbourne, VIC 3010, Australia jingy@cs.mu.oz.au</p></frontmatter><abstract>Selectional preferences are a source of linguistic infor­mation commonly applied to the task of Word Sense Dis­ambiguation (WSD). To date, WSD systems using se-lectional preferences as the main disambiguation mech­anism have achieved limited success. One possible rea­son for this limitation is the limited number of semantic roles used in the construction of selectional preferences. This study investigates whether better performance can be achieved using the current state-of-art semantic role labelling systems, and explores alternative ways of ap­plying selectional preferences for WSD. In this study, WordNet noun synonym sets and hypernym sets were used in the construction of selectional preferences; Sem-cor2.0 data was used for the training and evaluation of a support vector machine classifier and a Naive Bayes classifier. </abstract></header><body><section number="1" title="Introduction"><p>Word Sense Disambiguation (WSD) is the process of examining word tokens in a given context and specifying exactly which sense of each word is in­tended in that context. It has many applications in natural language processing related areas such as document retrieval, question answering, and com­positional sentence analysis (Jurafsky and Martin, 2000), to name a few.</p><p>WSD systems can be roughly divided into two categories based on how the disambiguation infor­mation is obtained and applied: knowledge based and corpus based. Knowledge based systems in general require certain existing linguistic informa­tion repositories which provides all the information that can be used by the disambiguation system to distinguish different senses of the same polysemous words based on the context. Examples of knowl­edge based systems include dictionary based sys­tems (Lesk, 1986) and selectional preference based systems (Resnik, 1997).</p><p>Corpus based systems in general do not require any linguistic information, instead, they require a certain amount of training data (labelled or unlabelled), and a set of predefined disambiguation fea­tures which can be used by a statistical method to train a classifier which then is used in the disam­biguation of previously unseen data. A corpus based system is described in (Yarowsky, 1995).</p><p>Selectional preferences between predicating words (verbs and adjectives) and their arguments (nouns) are a type of linguistic information which has previously been combined with statistical methods to perform word sense disambiguation, ((Resnik, 1997) and (McCarthy and Carroll, 2003)). A selectional preference is a function mapping semantic-role to noun type. The basic assumption made by all selectional preference based WSD systems is that the different senses of the same predicating word would have different selectional preferences with their arguments.</p><p>As will be discussed in section 2.2, selectional preference based WSD systems developed so far are limited in terms of coverage and accuracy. In my opinion, the most important cause of this lim­itation is these systems' inability to extract a suffi­cient number of semantic roles to be used in the con­struction of selectional preferences. For example, if a WSD system uses only the subject of the verbs in the selectional preferences, then it cannot be ex­pected to correctly identify the appropriate sense of the verb "run" in "John ran a race" and "John ran a restaurant", since the distinguishing feature of these two senses of "run" comes from the objects they take.</p><p>Given the difficulty of semantic role labelling, it is not surprising that only a small set of semantic roles have been used in the literature on selectional preference based WSD. However, recent develop­ments in semantic role labelling makes it possible to extract a much richer set of semantic roles from un­restricted text, thereby enabling more complex se-lectional preferences to be constructed.</p><p>The main objective of this study is to investigate whether the performance of selectional preference based WSD can be improved by using the current state-of-art semantic role labelling systems. This paper is organised as follows:<page local="2" global="156"/> section 2 will give a formal description of the research problem pre­sented in this paper; section 3 will provide a review of some related work; section 4 will discuss the sta­tistical methods investigated in this study and how they are combined with selectional preferences; the results of this study will be presented in section 5; and section 6 gives a conclusion of this study and some avenues for further research.</p></section><section number="2" title="Background"><subsection number="2.1" title="Selectional Preference"><p>Selectional preferences (p) are verb-sense specific. It is possible for a particular sense of a verb to have more than one selectional preference. A se-lectional preference of a verb-sense (s) refers to the predicate-argument structure relationship between s and its arguments. Formally, a selectional prefer­ence is a function whose domain is the finite set of semantic roles (r) and whose range is a finite set of noun types (t):</p><doubt alpha="62.5" length="8" tooSmall="False" monospace="0.0">p(ri)=tj</doubt><p>For example, the first sense of the verb "eat" (eati : <i>take in solid food) </i>in WordNet (Miller, 1995) would have a selectional preference that requires the <b>subject </b>of the verb to be nouns of the <b>animate </b>type and the <b>object </b>of the verb to be nouns of the <b>food </b>type; whereas the fourth sense of "eat" <i>(eat4 </i>: <i>use up (resources or materials)) </i>would have a selectional preference which allows the subject of the verb to be of both animate type and inanimate type.</p><p>Since there does not exist a set of commonly ac­cepted noun types, it is common for different selec-tional preference based WSD systems to invent their noun types.</p><p>One can draw a parallel between verb selectional preferences of natural language and function over­loading of the programming language Java. In Java, two or more functions can be declared with the same name, each of these functions will have a different argument list which the Java interpreter uses at run­time to select (disambiguate) the correct function. The argument list of Java functions is an ordered list of Java object <i>types. </i>Similarly, one can treat the dif­ferent senses of any verb as different functions shar­ing the same name, and distinguish between them based on which type of nouns are used in which se­mantic role of the verb.</p></subsection><subsection number="2.2" title="Related Work"><p>Resnik (1997) describes a WSD system which uses selectional preferences to train an entropy based probabilistic model classifier.  Resnik defines the <i>prior </i>distribution <i>Prp </i>(t) as the probability of the noun-type t occurring in a particular selectional preference p. From the prior distribution, Resnik defines the <i>selectional preference strength </i>of a par­ticular verb sense s with respect to a particular se-lectional preference p over a finite set of noun types T as:</p><doubt alpha="48.3" length="29" tooSmall="False" monospace="0.0">Stp(s)   =D(Prp(t\s)\\Prp(t))</doubt><doubt alpha="83.3" length="6" tooSmall="False" monospace="0.0">teTpv1</doubt><p>From the above equation, it is obvious that the selectional preference strength ofa verb sense s de­pends on how much mutual information the noun types of its arguments share. In other words, verb senses which take a small set ofnouns as arguments are easier to disambiguate.</p><p>With the selection preference strength, Resnik further defined the <i>selectional association </i>value be­tween a verb sense s and a noun-type t as:</p><p>The disambiguation of a polysemous verb v using Resnik's system is therefore achieved in the follow­ing way: Suppose the noun n is an argument to a polysemous verb v; Let <i>[s1,s2,sn</i>] be v's senses; let <i>[ns1,ns2,...,nsk</i>] be n's senses; and for each nsj, let <i>H</i><i>j </i>be the set of WordNet synsets which are hypernyms of <i>ns</i><i>j; </i>compute the following for each <i>s</i><i>i:</i></p><p><i>VA(s</i><i>i)</i><i> </i><i>=</i><i> </i>max <i>Ap(si,ns</i><i>j)</i></p><p>Then the verb sense(s) which maximise(s) the function <i>VA </i>will be chosen as the most appropriate sense(s) for <i>v. </i>Since Resnik's system is trained and evaluated on WordNet, he used a subset of WordNet noun synsets as the noun-types of his selectional preferences. Therefore, each <i>ns</i><i>i </i>is a noun type.</p><p>I believe this method of choosing noun types is a weakness of Resnik's system. It is not clear from his description whether this subset of noun synsets were hand picked or computed from the available data. If these synsets were hand picked (which is the likely scenario), then the resulting sys­tem could suffer from poor coverage because it was highly unlikely that the hand picked set of noun types were complete or compatible with the Word­Net noun hypernym hierarchy.   To illustrate this problem, consider the verb-object relationship be­tween <i>drink1 </i><i>(take in liquids </i>and its objects:<page local="3" global="157"/> if the noun-type <b>beverage </b><i>(beverage1 </i>) is chosen as a noun type (as it was in Resnik's paper), and the sentences "John drank wine" and "Joe drank cof­fee" are in the training data, since <i>coffee</i><i>1 </i>and <i>wine</i><i>1 </i>both have <b>beverage </b>as hypernym then the proba­bility of <i>Pr(beverage\drink1 ) </i>is very likely to be high. However, if in the testing data, the system encounters the sentence "John drank some water", then because "water" does not have <b>beverage </b>as a hypernym in WordNet, it would be unlikely for the system to identify the correct sense of "drink".</p><p>On the other hand, if the subset of noun types are computed from the training data, then all the hyper-nyms of the nouns in the training data would also be taken into account in the estimation of Prp(t\s). Furthermore, since the hypernym of a noun <i>n </i>would always describe a more general concept than <i>n, </i>then it is natural that the noun types describing the most general concepts would produce the highest value for the estimation of Prp(t\s). However, the more general the noun type is, the less distinguishing fea­ture it would be able to provide, therefore such noun types would not be effective for the WSD task.</p><p>Another weakness of Resnik's system is that the selectional preferences used in this system were constructed with only a single semantic role, e.g. the object of the verb or the subject of the verb. Therefore, these selectional preferences could only provide limited features useful for sense disam­biguation.</p></subsection></section><section number="3" title="Methodology"><subsection number="3.1" title="System Architecture"><p>The system developed in this study takes semantic-role-labelled sentences as inputs and trains a clas­sifier which can be used for the disambiguation of verbs.</p><p>The system consists of two major components: the selectional preference construction module and the classifier training and disambiguation module. When the system is given a semantic-role-labelled sentence, it first constructs the selectional prefer­ences from the labelled semantic roles and their head nouns. These selectional preferences are then passed to the statistical classifier for the training or the disambiguation of the verb.  In this study, two types of statistical classifiers were investigated: a Support Vector Machine (SVM) classifier and a Bayesian classifier.</p><p>A state-of-the-art semantic role labelling system, "ASSERT" (Pradhan et al., 2004), was used for the task of semantic role labelling. The influence of the</p><p>ASSERT will be discussed in section 4. In the re­mainder ofthis section, I will give details ofthe two main modules of the WSD system.</p></subsection><subsection number="3.2" title="Selectional Preference Construction"><p>In this study, the WordNet <b>noun hypernym hierar­chy </b>(NHH) is used to generate the noun-types used to construct the selectional preferences. A noun synset <i>nsa </i>is the hypernym of another noun synset <i>nsb </i>if <i>nsa </i>denotes a more general concept than nsb. A hypernym hierarchy for a noun synset <i>ns </i>is the tree structure which includes all the direct and indi­rect hypernyms of <i>ns.</i></p><p>Each path from the most specific node in the NHH to the most general node is treated as a sep­arate noun-type. For example, the NHH of the first sense of "apple" <i>(apple1 </i>: <i>fruit with red or yellow or green skin and sweet to tart crisp whitish flesh) </i>would generate the following noun-types:</p><p><i>t</i><i>i: </i><i>(entity</i><i>i,</i><i> </i><i>substance</i><i>i,</i><i> </i><i>solid</i><i>i,</i><i> </i><i>food</i><i>2,</i><i> </i><i>produce</i><i>i,</i><i> </i><i>edible-fruiti, applet)</i></p><p><i>ti'. (entityi, objecti, natural jobjecti, plant jparti, plantjorgani, reproductivestructurei, fruiti, edible-fruiti, applet)</i></p><p><i>ts'. (entityi, objecto, naturaljobjecti, plantjparti, plantjorgani, reproductivestructurei, fruiti, pome</i><i>i,</i><i> </i><i>apple</i><i>i)</i></p><p>There are two advantages ofusing paths extracted from WordNet NHH as the noun-types. First, it eliminates the need for a set of hand generated noun-types which is most likely to be not as com­prehensive as WordNet. Second, since the noun-types are set of noun synsets of varying degrees of generality, it is possible to compute partial equality between them, this partial equality will then be ap­plicable to the comparison between selectional pref­erences, thereby increasing the coverage and poten­tially the accuracy of the system.</p><p>To illustrate how selectional preferences are constructed from semantic-role-labelled sentences, suppose we have the following sentence:</p><p><b>E1 </b>[The monkey]argo [ate]target [an apple]</p><p>The head nouns for <b>arg0 </b>and <b>arg1 </b>are "monkey" and "apple" respectively. Since the system does not know which senses of these words are being used here, it will have to consider all senses of both words. In WordNet, "monkey" and "apple" corre­spond to the NHHs shown in figure 1.</p><p>Each <b>path </b>in the above NHHs is a noun-type. As we can see, there are 3 potential noun-types for "monkey" <b>(arg0), </b>and 4 potential noun-types for "apple" <b>(arg1).</b><page local="4" global="158"/><b> </b>Therefore, the sentence E1 gives rise to 12 potential selectional preferences.</p><doubt alpha="75.0" length="8" tooSmall="True" monospace="0.0">monkey_2</doubt><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">I ~</doubt><doubt alpha="71.4" length="7" tooSmall="True" monospace="0.0">child_1</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">1</doubt><doubt alpha="80.0" length="10" tooSmall="True" monospace="0.0">juvenile_1</doubt><doubt alpha="75.0" length="8" tooSmall="True" monospace="0.0">person_1</doubt><doubt alpha="80.0" length="10" tooSmall="True" monospace="0.0">chordate_1</doubt><doubt alpha="75.0" length="8" tooSmall="True" monospace="0.0">animal_1</doubt><doubt alpha="80.0" length="10" tooSmall="True" monospace="0.0">organism_1</doubt><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">I "</doubt><doubt alpha="78.6" length="14" tooSmall="True" monospace="0.0">living_thing_1</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">I</doubt><doubt alpha="75.0" length="8" tooSmall="True" monospace="0.0">object_1</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">r</doubt><doubt alpha="75.0" length="8" tooSmall="True" monospace="0.0">entity_1</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">f</doubt><doubt alpha="71.4" length="7" tooSmall="True" monospace="0.0">apple_2</doubt><doubt alpha="75.0" length="12" tooSmall="True" monospace="0.0">apple_tree_1</doubt><doubt alpha="75.0" length="12" tooSmall="True" monospace="0.0">fruit_tree_1</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">1 ~</doubt><doubt alpha="85.0" length="20" tooSmall="True" monospace="0.0">angiospermous_tree_1</doubt><doubt alpha="80.0" length="10" tooSmall="True" monospace="0.0">organism 1</doubt></subsection><subsection number="3.3" title="Training of the SVM Classifier"><p>The SVM classifier is the first of the two clas­sifiers investigated in this study. Since most verbs have more than two senses, the SVM clas­sifier was trained to be multi-class, and each sense was treated as an independent class. Two types of multi-class classification were experi­mented: One-class-Against-the-Rest-of-the-classes and One-class-Against-One-other-class.</p><p>The attributes used in the classification are com­binations of semantic role and WordNet noun synset. Recall that a selectional preference is a func­tion mapping semantic roles to noun types; and each noun-type is a set of WordNet noun synsets. Each noun synset will be combined with its respective se­mantic role to form a feature. Therefore, if the total number of semantic roles is <i>Nr </i>and the total number of WordNet noun synsets is Nns, the total number of dimensions or features is then <i>Nf = Nr </i>xNns. Dur­ing the training and the classification, all the selec-tional preferences generated for the <b>same </b>instance of a verb are used to create a single feature vector. If a synset <i>nsi </i>appears in the noun-type for a partic­ular semantic role <i>rj, </i>then the feature corresponding to the <i>(rj ,nsi ) </i>tuple will have the value of 1.0, otherwise this feature will have the value of 0.0. Fur­thermore, all the selectional preference generated are always stored in the same feature vector. The total number of features may seem excessive, how­ever, since the training data is unlikely to contain all the relevant selectional preferences, it is therefore necessary to include all the possible features during training and classification.</p><p>Two types of SVM kernels were experimented with in this study, linear and degree 2 polynomial.</p></subsection><subsection number="3.4" title="Training of the Probabilistic Classifier"><p>Since theoretically it is possible for a verb to take a large number of nouns for any of its semantic roles, the training of the probabilistic classifier would suf­fer from the data sparseness problem if no prepro­cessing is performed on the training data.</p><p>The preprocessing performed in this study is based on the theory of argument fusion (Jackendoff, 1990). Its main purpose is to extract common fea­tures from the noun types and give them appropriate mass in the probabilistic distribution. For example, suppose the training data consists of the following sentences for <i>eat</i><i>1 </i><i>(take in solid food).</i></p><p>51 [The monkey] argO [ate]target [anapple]0r9i</p><p>52 John's dietitian allowed [him]arg0 to [eat]targetonly [one slice of the <i>cake3]argi </i>at his birthday party.</p><p>S1 would generate the following selectional pref­erences:</p><p>S 1_1 <b>argO </b><i>(entityi,objecti,livingjthingi,</i></p><p><i>organism</i><i>i</i><i>, animal</i><i>i</i><i>, chordate</i><i>i</i><i>, verebrate</i><i>i,</i><i> </i><i>mammal</i><i>i</i><i>,placental</i><i>i</i><i>, primate</i><i>2</i><i>, monkey</i><i>i ) </i><b>argl </b><i>(entityi, objecti, natural jobjecti, plantjparti, plantjorgani, reproductionstructurei, fruiti, edible-fruiti, applei)</i></p><p>Sl_2 <b>argO </b><i>(entityi, objecti, living Jthingi,</i></p><p><i>organism</i><i>i</i><i>, animal</i><i>i</i><i>, chordate</i><i>i</i><i>, verebrate</i><i>i,</i><i> </i><i>mammal</i><i>i</i><i>,placental</i><i>i</i><i>, primate</i><i>2</i><i>, monkey</i><i>i )</i></p><p><b>argl </b><i>(entityi, objecti, natural </i><i>job</i><i> jecti, plantjparti, plantjorgani, reproductionstructurei, fruiti, pome</i><i>i</i><i> </i><i>, apple</i><i>i)</i></p><p>S 1_3 <b>argO </b><i>(entityi, objecti, living-thing </i><i>i,</i></p><p><i>organism</i><i>i,</i><i> </i><i>animal</i><i>i,</i><i> </i><i>chordate</i><i>i,</i><i> </i><i>verebrate</i><i>i,</i><i> </i><i>mammal</i><i>i,</i><i> </i><i>placental</i><i>i,</i><i> </i><i>primate</i><i>2</i><i>, monkey</i><i>i ) </i><b>argl </b><i>(entity</i><i>i</i><i>, substance</i><i>i,</i><i> </i><i>solid</i><i>i</i><i>,</i></p><p><i>food2</i><i>, producei, edible-fruiti, apple</i><i>i)</i><page local="5" global="159"/></p><doubt alpha="75.0" length="8" tooSmall="True" monospace="0.0">monkey 1</doubt><doubt alpha="77.8" length="9" tooSmall="True" monospace="0.0">primate_2</doubt><doubt alpha="81.8" length="11" tooSmall="True" monospace="0.0">placental_1</doubt><doubt alpha="75.0" length="8" tooSmall="True" monospace="0.0">mammal_1</doubt><doubt alpha="78.6" length="14" tooSmall="True" monospace="0.0">causal agent 1</doubt><doubt alpha="83.3" length="12" tooSmall="True" monospace="0.0">vertebrate 1</doubt><doubt alpha="75.0" length="8" tooSmall="True" monospace="0.0">entity 1</doubt><doubt alpha="71.4" length="7" tooSmall="True" monospace="0.0">apple 1</doubt><table caption="Figure 1: Example NHHs" class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>edible_fruit_1</b></p></td><td class="cell"><p><b>pome 1</b></p><p>1 "</p></td><td class="cell"><p><b>i</b></p><p><b>tree_1</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>produce 1</b></p><p>1 "</p></td><td class="cell"><p><b>fruit 1</b></p></td><td class="cell"><p><b>fruit 1</b></p></td><td class="cell"><p><b>f</b></p><p><b>woody_plant 1</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>rood 2</b></p><p>1</p></td><td class="cell"><p><b>reproductive structure 1</b></p><p>|"</p></td><td class="cell"><p><b>reproductive structure 1</b></p><p>|"</p></td><td class="cell"><p>1</p><p><b>vascular_plant_1</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>solid 1 </b>1</p></td><td class="cell"><p><b>plant organ 1</b></p><p>"l</p></td><td class="cell"><p><b>plant organ 1</b></p><p>"I</p></td><td class="cell"><p>1</p><p><b>plant_2</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>substance 1</b></p><p>1 "</p></td><td class="cell"><p><b>plant_part 1</b></p><p>1 "</p></td><td class="cell"><p><b>plant_part 1</b></p><p>1 "</p></td><td class="cell"><p>1</p><p><b>organism 1</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>entity_1</p></td><td class="cell"><p><b>natural object 1</b></p><p><b>~|</b></p></td><td class="cell"><p><b>natural object 1</b></p><p>"|</p></td><td class="cell"><p>1 "</p><p><b>living_thing_1</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>object 1</b></p><p>1 "</p></td><td class="cell"><p><b>object 1</b></p><p>1 "</p></td><td class="cell"><p>1</p><p><b>object_1</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>entity_1</p></td><td class="cell"><p>entity_1</p></td><td class="cell"><p>I "</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>S2 would generate the following selectional pref­erences:</p><p>S2_l <b>argO </b><i>(entityi, casualjagenti,person\, male</i><i>2</i><i>, man\, John)</i></p><p><b>argl </b><i>(entity</i><i>i</i><i>, substance</i><i>i</i><i>, solid</i><i>i</i><i>, food</i><i>2 </i><i>baked.goodsi, cake$)</i></p><p>S2_2 <b>argO </b><i>(entityi, ob jecti,living Ihingi,</i></p><p><i>organism</i><i>i</i><i>, person</i><i>i</i><i>, male</i><i>2</i><i>, man</i><i>i</i><i>, John)</i></p><p><b>argl </b><i>(entity</i><i>i,</i><i> </i><i>substance</i><i>i,</i><i> </i><i>solid</i><i>i,</i><i> </i><i>food</i><i>2</i><i> </i><i>baked.goodsi, cake$)</i></p><p>It can be observed that some of these selectional preferences have partial overlappings among the noun-types of the same semantic roles. These over-lappings capture what is in common between the ex­amples from the training data. Intuitively, the over-lappings are more suitable to be the selectional pref­erences than the individual training examples. For example, consider the selectional preferences gener­ated by S1 and S2 for <i>eat1</i>, one of the overlappings between them is:</p><p>S12_l <b>argO </b><i>(entityi, ob jecti, living Jchingi, organism</i><i>i)</i> <b>argl </b><i>(entity</i><i>i</i><i> </i><i>, substance</i><i>i,</i><i> </i><i>solid</i><i>i,</i><i> </i><i>food</i><i>2)</i></p><p>It is obvious that S 12_1 captures almost exactly what eaii's selectional preference really should be, namely that the subject of the verb has to be some living organism and the object of the verb has to be some kind of food. In the remainder of this pa­per, selectional preferences constructed through the process of argument fusion will be referred to as <b>fused selectional preferences, </b>and selectional pref­erences directly constructed from the training data will be referred to as <b>raw selectional preferences. </b>Since fused selectional preferences are more proto­typical than the raw ones, it would make sense to give them greater mass in the final probability dis­tribution.</p><p>Formally, the frequency of the selectional prefer­ences are estimated in the following way:</p><p>Let <i>rp</i><i>i </i>be a raw selectional preference, its fre­quency <i>(C (rpi)) </i>is the number of times <i>rpi </i>appears in the training examples.</p><p>Let <i>fp</i><i>j </i>be a fused selectional preference, and let <i>[rp1,rp2, ■ ■■,rpk </i>] be the set of raw selectional pref­erences from which <i>fp</i><i>j </i>was derived, then <i>fp</i><i>j</i>'s fre­quency is calculated as:</p><doubt alpha="47.4" length="19" tooSmall="False" monospace="0.0">C (fPj) = E C (rPi)</doubt><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">i=1</doubt><p>Similarly, the conditional frequency of the se-lectional preference pi given the verb sense <i>S</i><i>j </i><i>(C(pi\s</i><i>j</i>)) is estimated as the number of times pico-occurs with <i>S</i><i>j.</i></p><p>The two frequency distributions are then used to construct the corresponding probability distribu­tions which are then smoothed to allow far unseen data.</p><p>The classification of previously unseen data is not as simple as finding the verb sense si which maximises the probability of <i>Pr(si\p</i><i>j</i>). Firstly, let Pc be the set of candidate selectional prefer­ences [pi<i>pcn</i>] extracted with respect to an am­biguous verb <i>v </i>in a given context. Let P* be the set of selectional preferences [pi, <i>■ ■■■,pm</i>] from the training data. Suppose the set of senses of <i>v </i>is <i>S = [s1, sk</i>], then the most suitable sense(s) of v will be chosen in the following equation:</p><p><i>s</i><i>max </i><i>= </i>argmax <i>(max(Pr(s</i><i>i</i><i>\pj)) </i><i>s</i><i>i</i><i>€S,p</i><i>Ç</i><i>e</i><i>P</i><i>c</i></p><p>From Bayes' rule, <i>Pr(si\pc) </i>is calculated as fol­lows:</p><doubt alpha="66.7" length="9" tooSmall="False" monospace="0.0">Pr(si\pc)</doubt><doubt alpha="66.7" length="24" tooSmall="False" monospace="0.0">Pr(pcj\si)Pr(si) Pr(pjC)</doubt><p>However, since it is very likely that pc has not previously been seen in the training data, <i>Pr(si\pc ) </i>is therefore estimated as follows:</p><p>max <i>(Pr(si\ptk</i>)</p><doubt alpha="63.6" length="11" tooSmall="False" monospace="0.0">sim(pc,pk))</doubt><p>The function <i>sim(pa,pb) </i>calculates the similar­ity between two given selectional preferences. Let <i>dom(pa) </i>and <i>dom(pb) </i>be the sets of semantic roles applicable to <i>pa </i>and <i>pb </i>respectively. Recall that in this study, a noun-type <i>t </i>is a <b>set </b>of WordNet noun synsets, then the function <i>sim </i>works in the follow­ing way:</p><doubt alpha="46.7" length="15" tooSmall="False" monospace="0.0">sim(pa,pb ) = {</doubt></subsection></section><section number="4" title="Results"><doubt alpha="80.0" length="10" tooSmall="True" monospace="0.0">riEdom(pa)</doubt><doubt alpha="61.1" length="18" tooSmall="True" monospace="0.0">cos(pa(ri),pb(ri))</doubt><doubt alpha="67.9" length="28" tooSmall="True" monospace="0.0">dom(pa) = dom(pb ) otherwise</doubt><p>The system developed in this study was evaluated using the Semcor2.0 data and the Propbank data. Two types of baseline performances were used in the evaluation: the majority sense baseline (baseline 1) and the bag-of-synsets baseline (baseline 2).</p><p>The bag-of-synsets baseline works by first col­lecting the 10 nouns closest to and before the verb, and the 10 nouns closest to and after the verb; then extracting the synsets from their WordNet noun hy-pernym hierarchy; and finally using these synsets as features to training a support vector machine classi­fier.<page local="6" global="160"/> The purpose of this baseline is to see whether the additional information provided by the semantic roles can indeed improve the performance of WSD.</p><doubt alpha="0.0" length="1" tooSmall="True" monospace="0.0">0</doubt><p>Because of the diverse natural of verb selectional preferences and the different availabilities of the verb specific training data, the evaluation of the two classifiers was performed in a verb-by-verb fashion. The verbs selected for evaluation are: "bear", "eat", "kick", "look", "run", and "serve". As shown in ta­ble 1, these verbs are chosen because they represent a variety of transitivities, semantic role combina­tions, and different degrees of similarities between the senses. The senses of these verbs are defined in</p><p>WordNet2.0.</p><p>The following classifiers were trained and evalu­ated:</p><p>C1 SVM classifier with a linear kernel</p><p>C2 SVM classifier with a degree 2 polynomial kernel</p><p>C3 Naive Bayes classifier using thematic role tag set</p><table caption="Table 2 shows the number of senses and the ma­jority class baselines of the above verbs:"></table><p>'Intransitive <footnote anchor="2"/> Transitive</p><footnote label="3">Require Prepositional Complement</footnote><footnote label="4">Number of applicable semantic roles according to Prop-bank</footnote><p>Tables 3 to 5 show the results (Accuracy) of the above classifiers trained on 30%, 50%, and 80% of the training data:</p><p>The most significant feature of the results is that the three classifiers all performed below the major­ity class baseline. These poor results were caused by a combination of the following factors: complex sentence, poor semantic role labelling, inconsistent data, too finely defined verb senses and inadequate smoothing of the probability distributions.</p><p>The sentences used in the evaluation are gener­ally longer than 20 words and contain embedded clauses, metaphors and ellipses. For instance, one of the examples for "eat" <i>(eat</i>1 ) is the sentence: <i>The dialogue is sharp witty and candid typical don't </i><b>eat </b><i>the daisies materi al which has stamped the author throughout her books and plays and it was obvious that the Theatre-by-the-Sea audience liked it". </i>In this sentence, there is no subject/AGENT for "eat".   Another example for "eat" (eat3) is:</p><table caption="Table 3: Classifiers accuracy(%) when 30% data was used in the training" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Verbs</p></td><td class="cell"><p>baseline 2</p></td><td class="cell"><p>CI</p></td><td class="cell"><p>C2</p></td><td class="cell"><p>C3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>bear</p></td><td class="cell"><p>30.23</p></td><td class="cell"><p>37.9</p></td><td class="cell"><p>31.62</p></td><td class="cell"><p>20.23</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>eat</p></td><td class="cell"><p>75</p></td><td class="cell"><p>61.5</p></td><td class="cell"><p>64.5</p></td><td class="cell"><p>70.25</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>kick</p></td><td class="cell"><p>43.75</p></td><td class="cell"><p>42.5</p></td><td class="cell"><p>46.25</p></td><td class="cell"><p>43.16</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>look</p></td><td class="cell"><p>5.97</p></td><td class="cell"><p>49.25</p></td><td class="cell"><p>50.14</p></td><td class="cell"><p>26.27</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>run</p></td><td class="cell"><p>4.31</p></td><td class="cell"><p>4.83</p></td><td class="cell"><p>3.97</p></td><td class="cell"><p>5.69</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>serve</p></td><td class="cell"><p>12.5</p></td><td class="cell"><p>36.5</p></td><td class="cell"><p>38</p></td><td class="cell"><p>16.58</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 4: Classifiers accuracy(%) when 50% data was used in the training" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Verbs</p></td><td class="cell"><p>baseline 2</p></td><td class="cell"><p>CI</p></td><td class="cell"><p>C2</p></td><td class="cell"><p>C3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>bear</p></td><td class="cell"><p>6.67</p></td><td class="cell"><p>41.33</p></td><td class="cell"><p>37.33</p></td><td class="cell"><p>20.33</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>eat</p></td><td class="cell"><p>7.14</p></td><td class="cell"><p>60.36</p></td><td class="cell"><p>63.57</p></td><td class="cell"><p>65.36</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>kick</p></td><td class="cell"><p>18.18</p></td><td class="cell"><p>54.54</p></td><td class="cell"><p>50.9</p></td><td class="cell"><p>45.45</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>look</p></td><td class="cell"><p>57.63</p></td><td class="cell"><p>50.69</p></td><td class="cell"><p>52.5</p></td><td class="cell"><p>34.79</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>run</p></td><td class="cell"><p>1.19</p></td><td class="cell"><p>5</p></td><td class="cell"><p>9.52</p></td><td class="cell"><p>6.91</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>serve</p></td><td class="cell"><p>12.79</p></td><td class="cell"><p>4.88</p></td><td class="cell"><p>4.88</p></td><td class="cell"><p>14.42</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 1: Semantic Properties ofthe verbs" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Verb</p></td><td class="cell"><p>Intran.<footnote anchor="1"/></p></td><td class="cell"><p>Trans.<footnote anchor="2"/></p></td><td class="cell"><p>Compl.<footnote anchor="3"/></p></td><td class="cell"><p>NSR <footnote anchor="4"/></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>bear</p></td><td class="cell"><p>no</p></td><td class="cell"><p>yes</p></td><td class="cell"><p>no</p></td><td class="cell"><p>11</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>eat</p></td><td class="cell"><p>yes</p></td><td class="cell"><p>yes</p></td><td class="cell"><p>no</p></td><td class="cell"><p>2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>kick</p></td><td class="cell"><p>yes</p></td><td class="cell"><p>yes</p></td><td class="cell"><p>no</p></td><td class="cell"><p>9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>look</p></td><td class="cell"><p>yes</p></td><td class="cell"><p>no</p></td><td class="cell"><p>yes</p></td><td class="cell"><p>11</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>run</p></td><td class="cell"><p>yes</p></td><td class="cell"><p>yes</p></td><td class="cell"><p>no</p></td><td class="cell"><p>28</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>serve</p></td><td class="cell"><p>no</p></td><td class="cell"><p>yes</p></td><td class="cell"><p>yes</p></td><td class="cell"><p>9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 5: Classifiers accuracy(%) when 80% data was used in the training" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Verbs</p></td><td class="cell"><p>baseline 2</p></td><td class="cell"><p>CI</p></td><td class="cell"><p>C2</p></td><td class="cell"><p>C3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>bear</p></td><td class="cell"><p>14.29</p></td><td class="cell"><p>40.71</p></td><td class="cell"><p>44.28</p></td><td class="cell"><p>24.29</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>eat</p></td><td class="cell"><p>8.33</p></td><td class="cell"><p>58.33</p></td><td class="cell"><p>63.33</p></td><td class="cell"><p>63.33</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>kick</p></td><td class="cell"><p>20</p></td><td class="cell"><p>35</p></td><td class="cell"><p>56</p></td><td class="cell"><p>40</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>look</p></td><td class="cell"><p>33.89</p></td><td class="cell"><p>47.29</p></td><td class="cell"><p>48.81</p></td><td class="cell"><p>39.32</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>run</p></td><td class="cell"><p>2.56</p></td><td class="cell"><p>4.61</p></td><td class="cell"><p>7.17</p></td><td class="cell"><p>5.64</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>serve</p></td><td class="cell"><p>7.89</p></td><td class="cell"><p>8.95</p></td><td class="cell"><p>7.89</p></td><td class="cell"><p>13.16</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 2: Majority class baseline" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Verb</p></td><td class="cell"><p>Majority Baseline 1</p></td><td class="cell"><p>No. of senses</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>bear</p></td><td class="cell"><p>31.58%</p></td><td class="cell"><p>9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>eat</p></td><td class="cell"><p>76.27%</p></td><td class="cell"><p>3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>kick</p></td><td class="cell"><p>45%</p></td><td class="cell"><p>3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>look</p></td><td class="cell"><p>56.9%</p></td><td class="cell"><p>8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>run</p></td><td class="cell"><p>33.75%</p></td><td class="cell"><p>26</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>serve</p></td><td class="cell"><p>27.81%</p></td><td class="cell"><p>11</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="7" global="161"/><p><i>"No matter that it is his troops who rape Western women and eat Western men" </i>In this sentence, "eat" is clearly used in a metaphoric way therefore should not be interpreted literally. These complex sentences not only increase the amount of noise in the data, but also make semantic role labelling diffi­cult. According to my estimation, less than 30% of the sentences were correctly tagged with semantic roles.</p><p>Another problem with the semantic role labelling is that it only labels noun phrases. The impact of this problem is shown by the very poor re­sult on the verb "serve" most of whose senses re­quire either a propositional phrase or a verb phrase as compliment. For example, the sentence "The tree stump serves as a table" is annotated as "[The tree stump]agerat [serves]target as <i>[a]proposition </i>ta­ble" which is clearly wrong.</p><p>The problem caused by the excessively fine­grained senses is that these senses have very similar (sometimes identical) selectional preferences which cause inconsistency in the training data. Take <i>eat </i>for example, the definitions of its first and second senses are: <i>'take in solid food", </i>and <i>"eat a meal; take a meal"</i>respectively. In the training data, in <i>" She was personally sloppy, and when she had colds would blow her nose in the same handkerchief all dayandkeep itsoakingwetdangling fromherwaist and when she gardened she would </i><b>eat </b><i>dinner with dirt on her calves </i>, <i>eat </i>is labelled as having the first sense, but it is labelled as having the second sense in <i>'Charlie </i><b>ate </b><i>some supper in the kitchen and wentJnto the TV-room to hear the news. </i><i>".</i><i> </i>This type of inconsistency causes the classifiers to some­times behave almost randomly with respect to the relevant senses.</p><p>The problem caused by the inadequate smooth­ing of the probability distributions is more sub­tle. Given a very frequent senses <i>sa </i>and a very infrequent verb sense <i>sh </i>and a candidate selec-tional preference <i>pi </i>, the conditional probabilities of <i>Pr(sa\pc) </i>and <i>Pr(sh\pc) </i>depends on the values of <i>Pr(pc\s</i><i>a) • </i><i>Pr(s</i><i>a) </i>and <i>Pr(pc\s</i><i>h</i>) • <i>Pr(s</i><i>h). </i>It is often the case that there are so many selectional</p><doubt alpha="58.8" length="34" tooSmall="False" monospace="0.0">Pr{pt\sa) Pr(sb)Pr{pï\sb) ^ Pr(sa)</doubt><p>Another interesting feature of the results is that the differences of the accuracy is relatively small with respect to the different amounts of data used in training.  This feature is expected because one of the assumptions made by selectional preference based WSD is that each semantic role of any verb sense should be filled by nouns of similar type, in other words, nouns that have something in common. Therefore even though the amount of training data is different, the common features between the nouns of the same semantic role can still be captured and used for disambiguation.</p><p>Finally, the results also show that verbs with higher order of transitivity are easier to disam-biguate. This is also not surprising because higher transitivity means more semantic roles which in turn provides more disambiguating features.</p></section><section number="5" title="Conclusion and Future work"><p>This paper has presented a study of whether the performance of selectional preference based WSD could be improved by using the current state-of-art semantic role labelling system. Although very little performance improvement was able to be achieved by the systems developed in this study, a few useful observations could be made.</p><p>First, selectional preference based WSD systems do not require a large amount of training data, as demonstrated by the previous section. Therefore, they may be more useful or more effective than cor­pus based WSD systems when the amount of train­ing data is very limited or to act as a bootstrapping mechanism.</p><p>Second, to a very large degree, the performance of a selectional preference based WSD system de­pends on how finely the different senses of a verb are defined and the total number of semantic roles associated with the senses. As demonstrated by the "eat" example, the finer the senses are defined, the less effective selectional preference will be.</p><p>Third, the performance of selectional preference based WSD systems is heavily influenced by the quality of the semantic role identification. More importantly, it is not sufficient to only use seman­tic roles which can only be filled by noun phrases, as "serve" illustrated in the previous section; prepo­sitions and verbal complements are also likely to be useful to selectional preference based WSD sys­tems.</p><p>The results of this study also merit several further research topics. First, the focus of this research was on the disambiguation of verbs. However, the re­sults of the disambiguation also contains the sense information of the nouns which are the arguments to the disambiguated verbs. Therefore, the next step of the current research is to assess how well the system developed in this work would perform on the nouns.</p><p>A further extension to the current WSD system preferences applicable to <i>s</i><i>a </i>that thereby making the Bayes classifier assign s to in­stances of <i>s</i><i>a.</i><i> </i>Currently, the Lidstone probability distribution with <i>7 </i>of 0.0001 is used by the Bayes classifier; further study is required to select a more suitable probability distribution.</p><page local="8" global="162"/><p>would be to incorporate extra information such as the prepositions and other open class words in the disambiguation. This extension may require a hy­brid WSD system incorporating selectional prefer­ence based mechanisms and corpus based mecha­nisms.</p><p>Finally, as it was observed that the performance of a selectional preference based WSD system was heavily influenced by the quality of semantic role labelling; it might also be possible to use selec-tional preference as a crude measure of the perfor­mances of semantic role labelling systems on unla-belled data. This is because it is likely for a par­ticular semantic role to be filled by nouns of simi­lar types, therefore nouns correctly labelled for the same semantic role should exhibit a greater similar­ity than if incorrectly labelled.</p></section><section title="Acknowledgements"><p>I would like thank my supervisors, Professor Steven Bird and Dr. Adrian Pearce, for reviewing the paper. This research was sponsored by Australian Post­graduate Awards (APA) from the Australian Re­search Council (ARC).</p></section><references><p>Ray Jackendoff, 1990. <i>Semantic Structures, </i>chap­ter 2. The MIT Press.</p><p>Daniel Jurafsky and James H. Martin, 2000. <i>Speech and Language Processing, </i>chapter 17. Prentice</p><p>Hall, 1 edition. Michael Lesk. 1986. Automatic sense disambigua­tion using machine readable dictionaries: how to tell a pine cone from an ice cream cone. In <i>Proceedings of the 5th annual international con­ference on Systems documentation, </i>pages 24-26.</p><p>ACM Press.</p><p>Diana McCarthy and John Carroll. 2003. Disam-biguating nouns, verbs, and adjectives using auto­matically acquired selectional preferences. <i>Com­putational Linguistics, </i>29(4):639-654, Decem­ber.</p><doubt alpha="58.3" length="48" tooSmall="False" monospace="0.0">George A. Miller.   1995.   Wordnet:   a lexical</doubt><p>database for english.   <i>Communications of the</i> and D. Jurafsky. 2004. Shallow semantic pars­ing using support vector machines. In <i>Proceed­ings ofthe Human Language Technology Con­ference/North American chapter ofthe Associa­tion for Computational Linguistics annual meet­ing (HLT/NAACL-2004), Boston, MA </i>May 2-7. Philip Resnik.   1997.  Selectional preference and sense disambiguation. In <i>Proceedings ofACL Siglex Workshop on Tagging Text with Lexical</i> <i>Semantics, Why, What and How?</i><i>, Washington,</i> disambiguation rivaling supervised methods. In <i>Meeting </i><i>ofthe</i><i> Association for Computational Linguistics, </i>pages 189-196.</p><doubt alpha="51.6" length="62" tooSmall="False" monospace="0.0">ACM,38(11):39-41. S. Pradhan, W. Ward, K. Hacioglu, J. Martin,</doubt><doubt alpha="63.9" length="61" tooSmall="False" monospace="0.0">April 4-5, 1997.David Yarowsky. 1995. Unsupervised word sense</doubt></references></body></article>