<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="59"/><title>Discriminative Learning of Selectional Preference from Unlabeled Text</title><author surname="Bergsma" givenname="Shane"><org  name="University of Alberta" country="Canada" city="Edmonton"/></author><author surname="Lin" givenname="Dekang"><org  name="University of Alberta" country="Canada" city="Edmonton"/></author><author surname="Goebel" givenname="Randy"><org  name="University of Alberta" country="Canada" city="Edmonton"/></author></firstpageheader><frontmatter><p><b>Discriminative Learning of Selectional Preference from Unlabeled Text</b></p><p><b>Shane Bergsma Dekang Lin Randy Goebel</b></p><p>Department of Computing Science Google, Inc. Department of Computing Science</p><p>University of Alberta 1600 Amphitheatre Parkway        University of Alberta</p><p>Edmonton, Alberta Mountain View Edmonton, Alberta</p><p>Canada, T6G 2E8 California, 94301 Canada, T6G 2E8</p><p>bergsma@cs.ualberta.ca lindek@google.com goebel@cs.ualberta.ca</p></frontmatter><abstract>We present a discriminative method for learn­ing selectional preferences from unlabeled text. Positive examples are taken from ob­served predicate-argument pairs, while nega­tives are constructed from unobserved combi­nations. We train a Support Vector Machine classifier to distinguish the positive from the negative instances. We show how to parti­tion the examples for efficient training with 57 thousand features and 6.5 million training instances. The model outperforms other re­cent approaches, achieving excellent correla­tion with human plausibility judgments. Com­pared to Mutual Information, it identifies 66% more verb-object pairs in unseen text, and re­solves 37% more pronouns correctly in a pro­noun resolution experiment. </abstract></header><body><section number="1" title="Introduction"><p>Selectional preferences (SPs) tell us which argu­ments are plausible for a particular predicate. For example, Table 2 (Section 4.4) lists plausible and implausible direct objects (arguments) for particu­lar verbs (predicates). SPs can help resolve syntac­tic, word sense, and reference ambiguity (Clark and Weir, 2002), and so gathering them has received a lot of attention in the NLP community.</p><p>One way to determine SPs is from co-occurrences of predicates and arguments in text. Unfortunately, no matter how much text we use, many acceptable pairs will be missing. Bikel (2004) found that only 1.49% of the bilexical dependencies considered by Collins' parser during decoding were observed dur­ing training.  In our parsed corpus (Section 4.1), for example, we find <i>eat </i>with <i>nachos, burritos, </i>and <i>tacos, </i>but not with the equally tasty <i>quesadillas, chimichangas, </i>or <i>tostadas.</i><i> </i>Rather than solely re­lying on co-occurrence counts, we would like to use them to generalize to unseen pairs.</p><p>In particular, we would like to exploit a number of arbitrary and potentially overlapping properties of predicates and arguments when we assign SPs. We do this by representing these properties as fea­tures in a linear classifier, and training the weights using discriminative learning. Positive examples are taken from observed predicate-argument pairs, while pseudo-negatives are constructed from unob­served combinations. We train a Support Vector Ma­chine (SVM) classifier to distinguish the positives from the negatives. We refer to our model's scores as Discriminative Selectional Preference (Dsp). By creating training vectors automatically, Dsp enjoys all the advantages of supervised learning, but with­out the need for manual annotation of examples.</p><p>We evaluate DSP on the task of assigning verb-object selectional preference. We encode a noun's textual distribution as feature information. The learned feature weights are linguistically interesting, yielding high-quality similar-word lists as latent in­formation. Despite its representational power, DSP scales to real-world data sizes: examples are parti­tioned by predicate, and a separate SVM is trained for each partition. This allows us to efficiently learn with over 57 thousand features and 6.5 million ex­amples. DSP outperforms recently proposed alterna­tives in a range of experiments, and better correlates with human plausibility judgments. It also shows strong gains over a Mutual Information-based co-</p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">59</doubt><p><i>Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, </i>pages 59-68, Honolulu, October 2008. ©2008 Association for Computational Linguistics<page local="2" global="60"/></p><p>occurrence model on two tasks: identifying objects of verbs in an unseen corpus and finding pronominal antecedents in coreference data.</p></section><section number="2" title="Related Work"><p>Most approaches to SPs generalize from observed predicate-argument pairs to semantically similar ones by modeling the semantic class of the argu­ment, following Resnik (1996). For example, we might have a class <i>Mexican Food </i>and learn that the entire class is suitable for eating. Usually, the classes are from WordNet (Miller et al., 1990), although they can also be inferred from clustering (Rooth et al., 1999). Brockmann and Lapata (2003) compare a number of WordNet-based approaches, including</p><doubt alpha="53.2" length="47" tooSmall="False" monospace="0.0">Resnik (1996), Li and Abe (1998), and Clark and</doubt><p>Weir (2002), and found that the more sophisticated class-based approaches do not always outperform simple frequency-based models.</p><p>Another line of research generalizes using simi­lar words. Suppose we are calculating the proba­bility of a particular noun, n, occurring as the ob­ject argument of a given verbal predicate, v. Let Pr(n|v) be the empirical maximum-likelihood esti­mate from observed text. Dagan et al. (1999) define the similarity-weighted probability, PrSIM, to be:</p><doubt alpha="51.6" length="31" tooSmall="False" monospace="0.0">PrsiM(n|v)=Sim(v',v)Pr(n|v')(1)</doubt><p>v'esiMs(v)</p><p>where <i>Sim(v', </i><i>v)</i><i> </i>returns a real-valued similarity be­tween two verbs v' and v (normalized over all pair similarities in the sum). In contrast, Erk (2007) generalizes by substituting similar <i>arguments, </i>while Wang et al. (2005) use the cross-product of simi­lar pairs. One key issue is how to define the set of similar words, SlMs(w). Erk (2007) compared a number of techniques for creating similar-word sets and found that both the Jaccard coefficient and Lin (1998a)'s information-theoretic metric work best. Similarity-smoothed models are simple to compute, potentially adaptable to new domains, and require no manually-compiled resources such as WordNet.</p><p>Selectional Preferences have also been a recent focus of researchers investigating the learning of paraphrases and inference rules (Pantel et al., 2007; Roberto et al., 2007). Inferences such as <i>"[X </i>wins <i>Y]</i><i> </i>=&gt; [X plays Y]" are only valid for certain arguments <i>X</i>and <i>Y.</i><i> </i>We follow Pantel et al. (2007) in us­ing automatically-extracted semantic classes to help characterize plausible arguments.</p><p>Discriminative techniques are widely used in NLP and have been applied to the related tasks of word prediction and language modeling. Even-Zohar and Roth (2000) use a classifier to predict the most likely word to fill a position in a sentence (in their ex­periments: a verb) from a set of candidates (sets of verbs), by inspecting the context of the target token (e.g., the presence or absence of a particu­lar nearby word in the sentence). This approach can therefore learn which specific arguments occur with a particular predicate. In comparison, our fea­tures are second-order: we learn what <i>kinds </i>of argu­ments occur with a predicate by encoding features of the arguments. Recent distributed and latent­variable models also represent words with feature vectors (Bengio et al., 2003; Blitzer et al., 2005).</p><p>Many of these approaches learn both the feature weights and the feature representation. Vectors must be kept low-dimensional for tractability, while learn­ing and inference on larger scales is impractical. By partitioning our examples by predicate, we can effi­ciently use high-dimensional, sparse vectors.</p><p>Our technique of generating negative examples is similar to the approach of Okanohara and Tsujii (2007). They learn a classifier to disambiguate ac­tual sentences from pseudo-negative examples sam­pled from an N-gram language model. Smith and Eisner (2005) also automatically generate negative examples. They perturb their input sequence (e.g. the sentence word order) to create a neighborhood of <i>implicit </i>negative evidence. We create negatives by substitution rather than perturbation, and use corpus-wide statistics to choose our negative instances.</p></section><section number="3" title="Methodology"><subsection number="3.1" title="Creating Examples"><p>To learn a discriminative model of selectional pref­erence, we create positive and negative training ex­amples automatically from raw text. To create the positives, we automatically parse a large corpus, and then extract the predicate-argument pairs that have a statistical association in this data. We measure this association using pointwise Mutual Information (MI) (Church and Hanks, 1990). The MI between a verb predicate, v, and its object argument, n, is:<page local="3" global="61"/></p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">60</doubt><doubt alpha="43.4" length="76" tooSmall="False" monospace="0.0">„tt/x,     Pr(v,n), Pr(n|v)MI(v, n)= logp\ p /= log-P-1^-(2) Pr(v)Pr(n)Pr(n)</doubt><p>If MI&gt;0, the probability v and n occur together is greater than if they were independently distributed.</p><p>We create sets of positive and negative examples separately for each predicate, v. First, we extract all pairs where MI(v, n)&gt;T as positives. For each pos­itive, we create pseudo-negative examples, (v,n'), by pairing v with a new argument, n', that either has MI below the threshold or did not occur with v in the corpus. We require each negative n' to have a similar frequency to its corresponding n. This prevents our learning algorithm from focusing on any accidental frequency-based bias. We mix in K negatives for each positive, sampling without replacement to cre­ate all the negatives for a particular predicate. For each v, k+i of its examples will be positive. The threshold t represents a trade-off between capturing a large number of positive pairs and ensuring these pairs have good association. Similarly, K is a trade­off between the number of examples and the com­putational efficiency. Ultimately, these parameters should be optimized for task performance.</p><p>Of course, some negatives will actually be plau­sible arguments that were unobserved due to sparse-ness. Fortunately, modern discriminative methods like soft-margin SVMs can learn in the face of label error by allowing slack, subject to a tunable regular-ization penalty (Cortes and Vapnik, 1995).</p><p>If MI is a sparse and imperfect model of SP, what can Dsp gain by training on MI's scores? We can regard DSP as learning a view of SP that is or­thogonal to MI, in a co-training sense (Blum and</p><p>Mitchell, 1998). MI labels the data based solely on co-occurrence; DSP uses these labels to iden­tify other regularities - ones that extend beyond co-occurring words. For example, many instances of n where MI (eat, n)&gt;T also have MI (buy, n)&gt;T and MI (cook, n)&gt;T. Also, compared to other nouns, a disproportionate number of <i>eat</i>-nouns are lower­case, single-token words, and they rarely contain digits, hyphens, or begin with a human first name like <i>Bob. </i>DSP encodes these interdependent prop­erties as features in a linear classifier. This classi­fier can score any noun as a plausible argument of <i>eat </i>if indicative features are present; MI can only assign high plausibility to observed <i>(eat</i>,n) pairs. Similarity-smoothed models can make use of the regularities across similar verbs, but not the finer-grained string- and token-based features.</p><p>Our training examples are similar to the data cre­ated for <i>pseudodisambiguation, </i>the usual evalua­tion task for SP models (Erk, 2007; Keller and La-pata, 2003; Rooth et al., 1999). This data con­sists of triples (v, n, n') where v, n is a predicate-argument pair observed in the corpus and v, n' has not been observed. The models score correctly if they rank observed (and thus plausible) argu­ments above corresponding unobserved (and thus likely implausible) ones. We refer to this as <i>Pair-wise Disambiguation. </i>Unlike this task, we classify each predicate-argument pair independently as plau­sible/implausible. We also use MI rather than fre­quency to define the positive pairs, ensuring that the positive pairs truly have a statistical association, and are not simply the result of parser error or noise. <footnote anchor="1"/></p></subsection><subsection number="3.2" title="Partitioning for Efficient Training"><p>After creating our positive and negative training pairs, we must select a feature representation for our examples. Let $ be a mapping from a predicate-argument pair (v,n) to a feature vector, $ : (v,n) — ).  Predictions are made based on a weighted combination of the features, y = A • $(v, n), where A is our learned weight vector.</p><p>We can make training significantly more efficient by using a special form of attribute-value features. Let every feature 0j be of the form 0j(v, n) = (v = v A f (n)). That is, every feature is an intersection of the occurrence of a particular predicate, v, and some feature of the argument f(n). For example, a fea­ture for a verb-object pair might be, "the verb is <i>eat </i>and the object is lower-case." In this representation, features for one predicate will be completely inde­pendent from those for every other predicate. Thus rather than a single training procedure, we can actu­ally partition the examples by predicate, and train a classifier for each predicate independently.<page local="4" global="62"/> The pre­diction becomes y" <i>= X" • </i>3&gt;" (n), where <i>X" </i>are the learned weights corresponding to predicate v and all features 3&gt;" (n)=f (n) depend on the argument only.</p><footnote label="1">For a fixed verb, MI is proportional to Keller and Lapata (2003)'s conditional probability scores for pseudodisambigua-tion of (v, n, n') triples: Pr(v|n) = Pr(v, n)/Pr(n), which was shown to be a better measure of association than co-occurrence frequency f (v,n). Normalizing by Pr(v) (yielding MI) allows us to use a constant threshold across all verbs. MI was also recently used for inference-rule SPs by Pantel et al. (2007).</footnote><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">61</doubt><p>Some predicate partitions may have insuficient examples for training. Also, a predicate may oc­cur in test data that was unseen during training. To handle these instances, we decided to cluster low-frequency predicates. In our experiments assigning SP to verb-object pairs, we cluster all verbs that have less than 250 positive examples, using clusters gen­erated by the CBC algorithm (Pantel and Lin, 2002). For example, the low-frequency verbs <i>incarcerate, parole, </i>and <i>court-martial </i>are all mapped to the same partition, while more-frequent verbs like <i>arrest </i>and <i>execute </i>each have their own partition. About 5.5% of examples are clustered, corresponding to 30% of the 7367 total verbs. 40% of verbs (but only 0.6% of examples) were not in any CBC cluster; these were mapped to a single backoff partition.</p><p>The parameters for each partition, <i>X"</i>, can be trained with any supervised learning technique. We use SVM (Section 4.1) because it is effective in simi­lar high-dimensional, sparse-vector settings, and has an efficient implementation (Joachims, 1999). In SVM, the sign of<i>y" </i>gives the classiication. We can also use the scalar <i>y" </i>as our DSP score (i.e. the posi­tive distance from the separating SVM hyperplane).</p></subsection><subsection number="3.3" title="Features"><p>This section details our argument features, f<i>(n),</i><i> </i>for assigning verb-object selectional preference. For a verb predicate (or partition) <i>v </i>and object argument n, the form of our classifier is y" =    <i>% </i>A" <i>f</i><i>i</i><i>(n).</i></p><subsubsection number="3.3.1" title="Verb co-occurrence"><p>We provide features for the empirical probability of the noun occurring as the object argument of other verbs, Pr(njv'). If we were to only use these features (indexing the feature weights by each verb v'), the form of our classiier would be:</p><doubt alpha="35.0" length="20" tooSmall="False" monospace="0.0">y" =£A",Pr(njv') (3)</doubt><doubt alpha="50.0" length="2" tooSmall="False" monospace="0.0">v'</doubt><p>Note the similarity between Equation (3) and Equa­tion (1). Now the feature weights, A""', take the role of the similarity function, <i>Sim(v',v).</i><i> </i>Unlike Equa­tion (1), however, these weights are not set by an external similarity algorithm, but are optimized to discriminate the positive and negative training ex­amples. We need not restrict ourselves to a short list of similar verbs; we include Probj(njv') features for every verb that occurs more than 10 times in our cor­pus. A""' may be positive or negative, depending on the relation between v' and v. We also include fea­tures for the probability of the noun occurring as the <i>subject </i>of other verbs, <i>Pr</i><i>subj</i>(njv'). For example, nouns that can be the object of <i>eat </i>will also occur as the subject of <i>taste </i>and <i>contain. </i>Other contexts, such as adjectival and nominal predicates, could also aid the prediction, but have not yet been investigated.</p><p>The advantage of tuning similarity to the appli­cation of interest has been shown previously by Weeds and Weir (2005). They optimize a few meta­parameters separately for the tasks of thesaurus gen­eration and pseudodisambiguation. Our approach, on the other hand, discriminatively sets millions of individual similarity values. Like Weeds and Weir (2005), our similarity values are asymmetric.</p></subsubsection><subsubsection number="3.3.2" title="String-based"><p>We include several simple character-based fea­tures of the noun string: the number of tokens, the case, and whether it contains digits, hyphens, an apostrophe, or other punctuation. We also include a feature for the irst and last token, and ire indicator features if any token in the noun occurs on in-house lists of given names, family names, cities, provinces, countries, corporations, languages, etc. We also ire a feature if a token is a corporate designation (like <i>inc. </i>or <i>ltd.) </i>or a human one (like <i>Mr. </i>or <i>Sheik).</i></p></subsubsection><subsubsection number="3.3.3" title="Semantic classes"><p>Motivated by previous SP models that make use of semantic classes, we generated word clusters us­ing CBC (Pantel and Lin, 2002) on a 10 GB corpus, giving 3620 clusters. If a noun belongs in a cluster, a corresponding feature ires. If a noun is in none of the clusters, a <i>no-class </i>feature ires.</p><p>As an example, CBC cluster 1891 contains:</p><p>sidewalk, driveway, roadway, footpath, bridge, highway, road, runway, street, alley, path, Interstate,...</p><p>In our training data, we have examples like <i>widen highway, widen road </i>and <i>widen motorway. </i>If we see that we can widen a highway, we learn that we can also widen a sidewalk, bridge, runway, etc.<page local="5" global="63"/></p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">62</doubt><p>We also made use of the person-name/instance pairs automatically extracted by Fleischman et al. (2003).<footnote anchor="2"/> This data provides counts for pairs such as "Edwin Moses, <i>hurdler" </i>and "William Farley, <i>in­dustrialist" </i>We have features for all <i>concepts </i>and therefore learn their association with each verb.</p></subsubsection></subsection></section><section number="4" title="Experiments and Results 4.1   Set up"><p>We parsed the 3 GB AQUAINT corpus (Voorhees, 2002) using Minipar (Lin, 1998b), and collected verb-object and verb-subject frequencies, building an empirical MI model from this data. Verbs and nouns were converted to their (possibly multi-token) <i>root, </i>and string case was preserved. Passive sub­jects <i>(the car was bought) </i>were converted to objects <i>(bought car). </i>We set the MI-threshold, t, to be 0, and the negative-to-positive ratio, K, to be 2.</p><p>Numerous previous pseudodisambiguation evalu­ations only include arguments that occur between 30 and 3000 times (Erk, 2007; Keller and Lapata, 2003; Rooth et al., 1999). Presumably the lower bound is to help ensure the negative argument is unobserved because it is unsuitable, not because of data sparse-ness. We wish to use our model on arguments of any frequency, including those that never occurred in the training corpus (and therefore have empty co­occurrence features (Section 3.3.1)). We proceed as follows: irst, we exclude pairs whenever the noun occurs less than 3 times in our corpus, removing many misspellings and other noun noise. Next, we omit verb co-occurrence features for nouns that oc­cur less than 10 times, and instead ire a low-count feature. When we move to a new corpus, previously-unseen nouns are treated like these low-count train­ing nouns.</p><p>This processing results in a set of 6.8 million pairs, divided into 2318 partitions (192 of which are verb clusters (Section 3.2)). For each parti­tion, we take 95% of the examples for training, 2.5% for development and 2.5% for a inal unseen test set. We provide full results for two models: DsPcooc which only uses the verb co-occurrence fea­tures, and <i>Dsp</i><i>aa </i>which uses all the features mentioned in Section 3.3. Feature values are normalized within each feature type. We train our (linear kernel) discriminative models using</p><footnote label="2">Available at http://www.mit.edu/~mbf/instances.txt.gz</footnote><p>SVM%h* (Joachims, 1999) on each partition, but set meta-parameters C (regularization) and j (cost of positive vs. nega­tive misclassifications: max at j=2) on the macro-averaged score across all development partitions. Note that we can not use the development set to op­timize t and K because the development examples are obtained <i>after </i>setting these values.</p><subsection number="4.2" title="Feature weights"><p>It is interesting to inspect the feature weights re­turned by our system. In particular, the weights on the verb co-occurrence features (Section 3.3.1) provide a high-quality, argument-speciic similarity-ranking of other verb contexts. The Dsp parameters for <i>eat, </i>for example, place high weight on features like Pr(njbraise), Pr(nj<i>ration), </i>and Pr(njgarnish). Lin (1998a)'s similar word list for <i>eat </i>misses these but includes <i>sleep </i>(ranked 6) and <i>sit </i>(ranked 14), be­cause these have similar <i>subjects </i>to eat. Discrimina­tive, context-speciic training seems to yield a bet­ter set of similar predicates, e.g. the highest-ranked contexts for DsPcooc on the verb join,<footnote anchor="3"/> give a better SIMS(join) for Equation (1) than the top similarities returned by (Lin, 1998a):</p><doubt alpha="45.0" length="129" tooSmall="False" monospace="0.0">lead 1.42, rejoin 1.39, form 1.34, belong to 1.31, found 1.31, quit 1.29, guide 1.19, induct 1.19, launch(subj)1.18, work at 1.14</doubt><doubt alpha="41.5" length="135" tooSmall="False" monospace="0.0">participate 0.164, lead 0.150, return to 0.148, say 0.143, rejoin 0.142, sign 0.142, meet 0.142, include 0.141, leave 0.140, work 0.137</doubt><p>Other features are also weighted intuitively. Note that case is a strong indicator for some arguments, for example the weight on being lower-case is high for <i>become </i>(0.972) and <i>eat </i>(0.505), but highly nega­tive for <i>accuse </i>(-0.675) and <i>embroil </i>(-0.573) which often take names of people and organizations.</p></subsection><subsection number="4.3" title="Pseudodisambiguation"><p>We first evaluate Dsp on disambiguating posi­tives from pseudo-negatives, comparing to recently-<page local="6" global="64"/></p><footnote label="3">Which all correspond to nouns occurring in the object po­sition of the verb (e.g. Pr obj (n|lead)), except "launch (subj)" which corresponds to Pr sub j (n l launch).</footnote><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">63</doubt><p>Table 1: Pseudodisambiguation results averaged across each example <i>(MacroAvg), </i>weighted by word frequency <i>(Mi-croAvg), </i>plus coverage and accuracy of pairwise competition <i>(Pairwise).</i></p><p>proposed systems that also require no manually-compiled resources like WordNet. We convert Da-gan et al. (1999)'s similarity-smoothed probability to MI by replacing the empirical Pr(njv) in Equa­tion (2) with the smoothed PrSIM from Equation (1). We also test an MI model inspired by Erk (2007):</p><doubt alpha="66.7" length="15" tooSmall="False" monospace="0.0">MIsim(n,v)= log</doubt><doubt alpha="55.6" length="9" tooSmall="False" monospace="0.0">Sim(n',n)</doubt><p>ra'GslMS(ra)</p><doubt alpha="50.0" length="20" tooSmall="False" monospace="0.0">Pr(v, n )Pr(v)Pr(n')</doubt><p>We gather similar words using Lin (1998a), mining similar verbs from a comparable-sized parsed cor­pus, and collecting similar nouns from a broader 10 GB corpus of English text.<footnote anchor="4"/></p><p>We also use Keller and Lapata (2003)'s approach to obtaining web-counts. Rather than mining parse trees, this technique retrieves counts for the pattern "V Det N" in raw online text, where V is any in­flection of the verb, Det is the, a, or the empty string, and N is the singular or plural form of the noun. We compute a web-based MI by collecting Pr(n, v), Pr(n), and Pr(v) using all inflections, ex­cept we only use the root form of the noun. Rather than using a search engine, we obtain counts from the Google Web 5-gram Corpus.<footnote anchor="5"/></p><p>All systems are thresholded at zero to make a clas­sification. Unlike Dsp, the comparison systems may not be able to provide a score for each example. The similarity-smoothed examples will be undeined if SiMs(w) is empty. Also, the Keller and Lapata (2003) approach will be undeined if the pair is un­observed on the web. As a reasonable default for these cases, we assign them a negative decision.</p><footnote label="4">For both the similar-noun and similar-verb smoothing, we only smooth over similar pairs that occurred in the corpus. While averaging over all similar pairs tends to underestimate the probability, averaging over only the observed pairs tends to overestimate it. We tested both and adopt the latter because it resulted in better performance on our development set.</footnote><footnote label="5">Available from the LDC as LDC2006T13. This collection was generated from approximately 1 trillion tokens of online text. Unfortunately, tokens appearing less than 200 times have been mapped to the (UNK) symbol, and only N-grams appear­ing more than 40 times are included. Unlike results from search engines, however, experiments with this corpus are replicable.</footnote><p>We evaluate disambiguation using precision (P), recall (R), and their harmonic mean, F-Score (F). Table 1 gives the results of our comparison. In the <i>MacroAvg </i>results, we weight each example equally. For <i>MicroAvg, </i>we weight each example by the fre­quency of the noun. To more directly compare with previous work, we also reproduced <i>Pairwise Disam­biguation </i>by randomly pairing each positive with one of the negatives and then evaluating each system by the percentage it ranks correctly (Acc). For the comparison approaches, if one score is undeined, we choose the other one. If both are undeined, we abstain from a decision. Coverage (Cov) is the per­cent of pairs where a decision was made.<footnote anchor="6"/></p><p>Our simple system with only verb co-occurrence features, Dspcooc, outperforms all comparison ap­proaches. Using the richer feature set in DsPallresults in a statistically signiicant gain in perfor­mance, up to an F-Score of 0.65 and a pairwise disambiguation accuracy of 0.81.<footnote anchor="7"/> Dspall has both broader coverage and better accuracy than all com­peting approaches. In the remainder of the experi­ments, we use Dspall and refer to it simply as Dsp.</p><p>Some errors are because of plausible but unseen arguments being used as test-set pseudo-negatives. For example, for the verb <i>damage, </i>Dsp's three most high-scoring false positives are the nouns <i>jetliner, carpet, </i>and <i>gear. </i>While none occur with <i>damage </i>in<page local="7" global="65"/></p><footnote label="6">I.e. we use the "half coverage" condition from Erk (2007). 7 The differences between DsP a ;; and all comparison sys­tems are statistically significant (McNemar's test, p&lt;0.01).</footnote><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">64</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>System</p></td><td class="cell"><p><i>MacroAvg </i>P       R F</p></td><td class="cell"><p><i>MicroAvg </i>P       R F</p></td><td class="cell"><p><i>Pairwise </i>Acc Cov</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Daganetal. (1999) Erk (2007)</p><p>Keller and Lapata (2003)</p><p>D s <i>^cooc</i></p><p>Dsp<i>aii</i></p></td><td class="cell"><p>0.36   <b>0.90 </b>0.51 0.49   0.66 0.56 <b>0.72   </b>0.34 0.46 0.53   0.72 0.61 0.60   0.71 <b>0.65</b></p></td><td class="cell"><p>0.68   <b>0.92 </b>0.78 0.70   0.82 0.76 <b>0.80   </b>0.50 0.62 0.73   0.94 <b>0.82 </b>0.77   0.90 <b>0.83</b></p></td><td class="cell"><p>0.58 0.98 0.72 0.83 0.80 0.57 0.77 <b>1.00 0.81 1.00</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><doubt alpha="0.0" length="35" tooSmall="False" monospace="0.0">0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">0</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">DSP</doubt><doubt alpha="100.0" length="3" tooSmall="True" monospace="0.0">all</doubt><doubt alpha="51.4" length="35" tooSmall="False" monospace="0.0">Erk (2007) Keller and Lapata (2003)</doubt><doubt alpha="2.7" length="37" tooSmall="False" monospace="0.0">10     100   1000  10000 100000 1e+06</doubt><p>Noun Frequency</p><figure caption="Figure 1: Disambiguation results by noun frequency."></figure><p>our corpus, all intuitively satisfy the verb's SPs.</p><p><i>MacroAvg </i>performance is worse than <i>MicroAvg </i>because all systems perform better on frequent nouns. When we plot F-Score by noun frequency (Figure 1), we see that Dsp outperforms comparison approaches across all frequencies, but achieves its biggest gains on the low-frequency nouns. A richer feature set allows Dsp to make correct inferences on examples that provide minimal co-occurrence data. These are also the examples for which we would ex­pect co-occurrence models like MI to fail.</p><p>As a further experiment, we re-trained Dsp but with only the string-based features removed. Overall macro-averaged F-score dropped from 0.65 to 0.64 (a statistically signiicant reduction in performance). The system scored nearly identically to Dsp on the high-frequency nouns, but performed roughly 15% worse on the nouns that occurred less than ten times. This shows that the string-based features are impor­tant for selectional preference, and particularly help­ful for low-frequency nouns.</p></subsection><subsection number="4.4" title="Human Plausibility"><p>Table 2 compares some of our systems on data used by Resnik (1996) (also Appendix 2 in Holmes et al. (1989)). The plausibility of these pairs was initially judged based on the experimenters' intuitions, and later conirmed in a human experiment. We include the scores of Resnik's system, and note that its errors were attributed to sense ambiguity and other limi­tations of class-based approaches (Resnik, 1996).<footnote anchor="8"/></p><footnote label="8">For example, warn-engine scores highly because engines are in the class entity, and physical entities (e.g. people) are often objects of warn. Unlike D sp , Resnik's approach cannot learn that for warn, "the property of being a person is more</footnote><p>Seen Criteria</p><doubt alpha="52.9" length="17" tooSmall="False" monospace="0.0">MI&gt;0 Freq.&gt;0Dsp&gt;0</doubt><p>Unseen Verb-Object Freq.</p><p>Table 3: Recall on identification of Verb-Object pairs from an unseen corpus (divided by pair frequency).</p><p>The other comparison approaches also make a num­ber of mistakes, which can often be traced to a mis­guided choice of similar word to smooth with.</p><p>We also compare to our empirical MI model, trained on our parsed corpus. Although Resnik (1996) reported that 10 of the 16 plausible pairs did not occur in his training corpus, all of them occurred in ours and hence MI gives very reasonable scores on the plausible objects. It has no statistics, however, for many of the implausible ones. Dsp can make iner decisions than MI, recognizing that "warning an engine" is more absurd than "judging a climate."</p></subsection><subsection number="4.5" title="Unseen Verb-Object Identification"><p>We next compare MI and Dsp on a much larger set of plausible examples, and also test how well the models generalize across data sets. We took the MI and Dsp systems trained on AQUAINT and asked them to rate observed (and thus likely plausible) verb-object pairs taken from an unseen corpus. We extracted the pairs by parsing the San Jose Mercury News (SJM) section of the TIPSTER corpus (Har-man, 1992). Each unique verb-object pair is a single instance in this evaluation.</p><p>Table 3 gives recall across all pairs <b>(All) </b>and grouped by pair-frequency in the unseen corpus (1, 2, 3, &gt;3). Dsp accepts far more pairs than MI (73% vs. 44%), even far more than a system that accepts any previously observed verb-object combi­nation as plausible (57%). Recall is higher on more frequent verb-object pairs, but 70% of the pairs oc­curred only once in the corpus. Even if we smooth MI by smoothing Pr(njv) in Equation 2 using modi­fied KN-smoothing (Chen and Goodman, 1998), the recall of MI&gt;0 on SJM only increases from 44.1% to 44.9%, still far below Dsp. Frequency-based models have fundamentally low coverage. As fur-</p><p>important than the property ofbeing an entity" (Resnik, 1996).</p><doubt alpha="66.7" length="12" tooSmall="True" monospace="0.0">Si oo CO LL.</doubt><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">65</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>All</b></p></td><td class="cell"><p>= 1</p></td><td class="cell"><p>= 2</p></td><td class="cell"><p>= 3</p></td><td class="cell"><p>&gt; 3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>0.44</p></td><td class="cell"><p>0.33</p></td><td class="cell"><p>0.57</p></td><td class="cell"><p>0.70</p></td><td class="cell"><p>0.82</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>0.57</p></td><td class="cell"><p>0.45</p></td><td class="cell"><p>0.76</p></td><td class="cell"><p>0.89</p></td><td class="cell"><p>0.96</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>0.73</p></td><td class="cell"><p>0.69</p></td><td class="cell"><p>0.80</p></td><td class="cell"><p>0.85</p></td><td class="cell"><p>0.88</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="8" global="66"/><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">0</doubt><p>Table 2: Selectional ratings for plausible/implausible direct objects (Holmes et al., 1989). Mistakes are marked with an asterisk (*), undefined scores are marked with a dash (—). Only Dsp is completely defined and completely correct.</p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">P</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">dte</doubt><doubt alpha="100.0" length="5" tooSmall="False" monospace="0.0">latol</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">rp</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">ter</doubt><doubt alpha="0.0" length="33" tooSmall="False" monospace="0.0">0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">0.2</doubt><doubt alpha="0.0" length="7" tooSmall="False" monospace="0.0">0.4 0.6</doubt><doubt alpha="100.0" length="6" tooSmall="False" monospace="0.0">Recall</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">0.8</doubt><figure caption="Figure 2: Pronoun resolution precision-recall on MUC."></figure><p>ther evidence, if we build a model of MI on the SJM corpus and use it in our pseudodisambiguation ex­periment (Section 4.3), MI&gt;0 gets a <i>MacroAvg </i>pre­cision of 86% but a <i>MacroAvg </i>recall of only 12%.<footnote anchor="9"/></p></subsection><subsection number="4.6" title="Pronoun Resolution"><p>Finally, we evaluate Dsp on a common application of selectional preferences: choosing the correct an­tecedent for pronouns in text (Dagan and Itai, 1990; Kehler et al., 2004). We study the cases where a pronoun is the direct object of a verb predicate, v. A pronoun's antecedent must obey v's selectional pref­erences. If we have a better model of SP, we should be able to better select pronoun antecedents.</p><footnote label="9">Recall that even the Keller and Lapata (2003) system, built on the world's largest corpus, achieves only 34% recall (Table 1) (with only 48% of positives and 27% of all pairs previously observed, but see Footnote 5).</footnote><p>We parsed the MUC-7 (1997) coreference corpus and extracted all pronouns in a direct object rela­tion. For each pronoun, p, modified by a verb, v, we extracted all preceding nouns within the current or previous sentence. Thirty-nine anaphoric pronouns had an antecedent in this window and are used in the evaluation. For each p, let N(p)+ by the set of preceding nouns coreferent with p, and let N(p)-be the remaining non-coreferent nouns. We take all (v,n+) where n+ G N(p)+ as positive, and all other pairs (v,n-), n- G N(p)- as negative.</p><p>We compare MI and Dsp on this set, classifying every (v,n) with MI&gt;T (or Dsp&gt;T) as positive. By varying T, we get a precision-recall curve (Fig­ure 2). Precision is low because, of course, there are many nouns that satisfy the predicate's SPs that are not coreferent. Dsp&gt;0 has both a higher recall and higher precision than accepting every pair pre­viously seen in text (the right-most point on MI&gt;T). The Dsp&gt;T system achieves higher precision than MI&gt;T for points where recall is greater than 60% (where MI&lt;0). Interestingly, the recall of MI&gt;0 is<page local="9" global="67"/></p><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">1</doubt><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">66</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Verb</p></td><td class="cell"><p>Plaus./Implaus.</p></td><td class="cell"><p>Resnik</p></td><td class="cell"><p>Dagan et al.</p></td><td class="cell"><p>Erk</p></td><td class="cell"><p>MI</p></td><td class="cell"><p>Dsp</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>see</p></td><td class="cell"><p>friend/method</p></td><td class="cell"><p>5.79/-0.01</p></td><td class="cell"><p><i>0.20/1.40*</i></p></td><td class="cell"><p>0.46/-0.07</p></td><td class="cell"><p>1.11/-0.57</p></td><td class="cell"><p>0.98/0.02</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>read</p></td><td class="cell"><p>article/fashion</p></td><td class="cell"><p>6.80/-0.20</p></td><td class="cell"><p>3.00/0.11</p></td><td class="cell"><p>3.80/1.90</p></td><td class="cell"><p>4.00/—</p></td><td class="cell"><p>2.12/-0.65</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>find</p></td><td class="cell"><p>label/fever</p></td><td class="cell"><p>1.10/0.22</p></td><td class="cell"><p><i>1.50/2.20*</i></p></td><td class="cell"><p>0.59/0.01</p></td><td class="cell"><p>0.42/0.07</p></td><td class="cell"><p>1.61/0.81</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>hear</p></td><td class="cell"><p>story/issue</p></td><td class="cell"><p><i>1.89/1.89*</i></p></td><td class="cell"><p><i>0.66/1.50*</i></p></td><td class="cell"><p><i>2.00/2.60*</i></p></td><td class="cell"><p>2.99/-1.03</p></td><td class="cell"><p>1.66/0.67</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>write</p></td><td class="cell"><p>letter/market</p></td><td class="cell"><p>7.26/0.00</p></td><td class="cell"><p>2.50/-0.43</p></td><td class="cell"><p>3.60/-0.24</p></td><td class="cell"><p>5.06/-4.12</p></td><td class="cell"><p>3.08/-1.31</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>urge</p></td><td class="cell"><p>daughter/contrast</p></td><td class="cell"><p><i>1.14/1.86*</i></p></td><td class="cell"><p><i>0.14/1.60*</i></p></td><td class="cell"><p><i>1.10/3.60*</i></p></td><td class="cell"><p>-0.95/—</p></td><td class="cell"><p>-0.34/-0.62</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>warn</p></td><td class="cell"><p>driver/engine</p></td><td class="cell"><p>4.73/3.61</p></td><td class="cell"><p>1.20/0.05</p></td><td class="cell"><p>2.30/0.62</p></td><td class="cell"><p>2.87/—</p></td><td class="cell"><p>2.00/-0.99</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>judge</p></td><td class="cell"><p>contest/climate</p></td><td class="cell"><p>1.30/0.28</p></td><td class="cell"><p><i>1.50/1.90*</i></p></td><td class="cell"><p><i>1.70/1.70*</i></p></td><td class="cell"><p>3.90/—</p></td><td class="cell"><p>1.00/0.51</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>teach</p></td><td class="cell"><p>language/distance</p></td><td class="cell"><p><i>1.87/1.86</i></p></td><td class="cell"><p>2.50/1.30</p></td><td class="cell"><p>3.60/2.70</p></td><td class="cell"><p>3.53/—</p></td><td class="cell"><p>1.86/0.19</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>show</p></td><td class="cell"><p>sample/travel</p></td><td class="cell"><p>1.44/0.41</p></td><td class="cell"><p>1.60/0.14</p></td><td class="cell"><p>0.40/-0.82</p></td><td class="cell"><p>0.53/-0.49</p></td><td class="cell"><p>1.00/-0.83</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>expect</p></td><td class="cell"><p>visit/mouth</p></td><td class="cell"><p><i>0.59/5.93*</i></p></td><td class="cell"><p><i>1.40/1.50*</i></p></td><td class="cell"><p>1.40/0.37</p></td><td class="cell"><p>1.05/-0.65</p></td><td class="cell"><p>1.44/-0.15</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>answer</p></td><td class="cell"><p>request/tragedy</p></td><td class="cell"><p>4.49/3.88</p></td><td class="cell"><p>2.70/1.50</p></td><td class="cell"><p>3.10/-0.64</p></td><td class="cell"><p>2.93/—</p></td><td class="cell"><p>1.00/0.01</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>recognize</p></td><td class="cell"><p>author/pocket</p></td><td class="cell"><p><i>0.50/0.50*</i></p></td><td class="cell"><p><i>0.03/0.37*</i></p></td><td class="cell"><p><i>0.77/1.30*</i></p></td><td class="cell"><p>0.48/—</p></td><td class="cell"><p>1.00/0.00</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>repeat</p></td><td class="cell"><p>comment/journal</p></td><td class="cell"><p><i>1.23/1.23*</i></p></td><td class="cell"><p>2.30/1.40</p></td><td class="cell"><p>2.90/—</p></td><td class="cell"><p>2.59/—</p></td><td class="cell"><p>1.00/-0.48</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>understand</p></td><td class="cell"><p>concept/session</p></td><td class="cell"><p><i>1.52/1.51</i></p></td><td class="cell"><p>2.70/0.25</p></td><td class="cell"><p>2.00/-0.28</p></td><td class="cell"><p>3.96/—</p></td><td class="cell"><p>2.23/-0.46</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>remember</p></td><td class="cell"><p>reply/smoke</p></td><td class="cell"><p>1.31/0.20</p></td><td class="cell"><p>2.10/1.20</p></td><td class="cell"><p><i>0.54/2.60*</i></p></td><td class="cell"><p>1.13/-0.06</p></td><td class="cell"><p>1.00/-0.42</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>System Acc higher here than it is for general verb-objects (Sec­tion 4.5). On the subset of pairs with strong empir­ical association (MI&gt;0), MI generally outperforms Dsp at equivalent recall values.</p><doubt alpha="61.8" length="55" tooSmall="False" monospace="0.0">Most-Recent NounI17.9% Maximum MI 28.2% MaximumDsp38.5%</doubt><table caption="Table 4: Pronoun resolution accuracy on nouns in current or previous sentence in MUC."></table><p>We next compare MI and Dsp as stand-alone pro­noun resolution systems (Table 4). As a standard baseline, for each pronoun, we choose the most recent noun in text as the pronoun's antecedent, achieving 17.9% resolution accuracy. This baseline is quite low because many of the most-recent nouns are subjects of the pronoun's verb phrase, and there­fore resolution violates syntactic coreference con­straints. If instead we choose the previous noun with the highest MI as antecedent, we get an accuracy of 28.2%, while choosing the previous noun with the highest Dsp achieves 38.5%. Dsp resolves 37% more pronouns correctly than MI. We leave as fu­ture work a full-scale pronoun resolution system that incorporates both MI and Dsp as backed-off, inter­polated, or separate semantic features.</p></subsection></section><section number="5" title="Conclusions and Future Work"><p>We have presented a simple, effective model of se-lectional preference based on discriminative train­ing. Supervised techniques typically achieve higher performance than unsupervised models, and we du­plicate these gains with Dsp. Here, however, these gains come at no additional labeling cost, as train­ing examples are generated automatically from un-labeled text. Dsp allows an arbitrary combination of features, including verb co-occurrence features that yield high-quality similar-word lists as latent output. This work only scratches the surface of possible fea­ture mining; information from WordNet relations, Wikipedia categories, or parallel corpora could also provide valuable clues to SP. Also, if any other sys­tem were to exceed Dsp's performance, it could also be included as one of Dsp's features.</p><p>It would be interesting to expand our cooccurrence features, including co-occurrence counts across more grammatical relations and using counts from external, unparsed corpora like the world wide web. We could also reverse the role of noun and verb in our training, having verb-speciic features and discriminating separately for each argument noun. The latent information would then be lists of similar nouns.</p><p>Finally, note that while we focused on word-word co-occurrences, sense-sense SPs can also be learned with our algorithm. If our training corpus was sense­labeled, we could run our algorithm over the senses rather than the words. The resulting model would then require sense-tagged input if it were to be used within an application like parsing or coreference res­olution. Also, like other models of SP, our technique can also be used for sense disambiguations: the weightings on our semantic class features indicate, for a particular noun, which of its senses (classes) is most compatible with each verb.</p></section><section title="Acknowledgments"><p>We gratefully acknowledge support from the Natu­ral Sciences and Engineering Research Council of Canada, the Alberta Ingenuity Fund, and the Alberta Informatics Circle of Research Excellence.</p></section><references><p>Yoshua Bengio, Rejean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic lan­guage model. <i>Journal of Machine Learning Research, </i>3:1137-1155.</p><p>Daniel M. Bikel. 2004. Intricacies of Collins' parsing model. <i>Computational Linguistics, </i>30(4):479-511.</p><p>John Blitzer, Amir Globerson, and Fernando Pereira. 2005. Distributed latent variable models of lexical co­occurrences. In <i>AISTATS.</i></p><p>Avrim Blum and Tom Mitchell. 1998. Combining la­beled and unlabeled data with co-training. In <i>Proceed­ings ofCOLT, </i>pages 92-100.</p><p>Carsten Brockmann and Mirella Lapata. 2003. Evalu­ating and combining approaches to selectional prefer­ence acquisition. In <i>EACL, </i>pages 27-34.</p><p>Stanley F. Chen and Joshua Goodman. 1998. An empir­ical study of smoothing techniques for language mod­eling. TR-10-98, Harvard University.</p><p>Kenneth Ward Church and Patrick Hanks. 1990. Word association norms, mutual information, and lexicogra­phy. <i>Computational Linguistics, </i>16(1):22-29.</p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">67</doubt><page local="10" global="68"/><p>Stephen Clark and David Weir. 2002. Class-based prob­ability estimation using a semantic hierarchy. <i>Compu­tational Linguistics, </i>28(2):187-206.</p><p>Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. <i>Machine Learning, </i>20(3):273-297.</p><p>Ido Dagan and Alan Itai. 1990. Automatic processing of large corpora forthe resolutionofanaphorareferences.</p><p>In <i>COLING, </i>volume 3, pages 330-332.</p><p>Ido Dagan, Lillian Lee, and Fernando C. N. Pereira. 1999. Similarity-based models of word cooccurrence probabilities. <i>Machine Learning, </i>34(1-3):43-69.</p><p>Katrin Erk. 2007. A simple, similarity-based model for selectional preference. In <i>ACL, </i>pages 216-223.</p><p>Yair Even-Zohar and Dan Roth. 2000. A classification approach to word prediction. In <i>NAACL, </i>pages 124­131.</p><p>Michael Fleischman, Eduard Hovy, and Abdessamad Echihabi. 2003. Offline strategies for online question answering: answering questions before they are asked. In <i>ACL, </i>pages 1-7.</p><p>Donna Harman. 1992. The DARPA TIPSTER project.</p><p><i>ACMSIGIR Forum, </i>26(2):26-28.</p><p>Virginia M. Holmes, Laurie Stowe, and Linda Cupples. 1989. Lexical expectations in parsing complement-verb sentences.  <i>Journal ofMemory and Language,</i> 28:668-689.</p><p>Thorsten Joachims. 1999. Making large-scale Support Vector Machine learning practical. In B. Scholkopf and C. Burges, editors, <i>Advances in Kernel Methods: Support Vector Machines, </i>pages 169-184. MIT-Press.</p><p>Andrew Kehler, Douglas Appelt, Lara Taylor, and Alek-sandr Simma. 2004. The (non)utility of predicate-argument frequencies for pronoun interpretation. In <i>HLT/NAACL, </i>pages 289-296.</p><p>Frank Keller and Mirella Lapata. 2003. Using the web to obtain frequencies for unseen bigrams. <i>Computational Linguistics, </i>29(3):459-484.</p><p>Hang Li and Naoki Abe. 1998. Generalizing case frames using a thesaurus and the MDL principle. <i>Computa­tional Linguistics, </i>24(2):217-244.</p><p>Dekang Lin. 1998a. Automatic retrieval and clustering of similar words. In <i>COLING-ACL, </i>pages 768-773.</p><p>Dekang Lin. 1998b. Dependency-based evaluation of MINIPAR. In <i>LREC Workshop on the Evaluation of Parsing Systems.</i></p><p>George A. Miller, Richard Beckwith, Christiane Fell­baum, Derek Gross, and Katherine J. Miller. 1990. Introduction to WordNet: an on-line lexical database. <i>International Journal of Lexicography, </i>3(4):235-244.</p><p>MUC-7. 1997. Coreference task definition (v3.0, 13 Jul 97). In <i>Proceedings ofthe Seventh Message Under­standing Conference (MUC-7).</i></p><p>Daisuke Okanohara and Jun'ichi Tsujii. 2007. A discriminative language model with pseudo-negative samples. In <i>ACL, </i>pages 73-80.</p><p>Patrick Pantel and Dekang Lin. 2002. Discovering word senses from text. In <i>KDD, </i>pages 613-619.</p><p>Patrick Pantel, Rahul Bhagat, Bonaventura Coppola, Timothy Chklovski, and Eduard Hovy. 2007. ISP: Learning inferential selectional preferences. In <i>NAACL-HLT, </i>pages 564-571.</p><p>Philip Resnik. 1996. Selectional constraints: An information-theoretic model and its computational re­alization. <i>Cognition, </i>61:127-159.</p><p>Basili Roberto, Diego De Cao, Paolo Marocco, and Marco Pennacchiotti. 2007. Learning selectional preferences for entailment or paraphrasing rules. In <i>RANLP.</i><i></i></p><p>Mats Rooth, Stefan Riezler, Detlef Prescher, Glenn Car­roll, and Franz Beil. 1999. Inducing a semantically annotated lexicon via EM-based clustering. In <i>ACL, </i>pages 104-111.</p><p>Noah A. Smith and Jason Eisner. 2005. Contrastive esti­mation: training log-linear models on unlabeled data.</p><doubt alpha="58.8" length="68" tooSmall="False" monospace="0.0">InACL,pages 354-362. Ellen Voorhees. 2002. Overview of the TREC 2002</doubt><p>question answering track. In <i>Proceedings of the Eleventh Text REtrieval Conference (TREC).</i></p><p>Qin Iris Wang, Dale Schuurmans, and Dekang Lin. 2005. Strictly lexical dependency parsing. In <i>International Workshop on Parsing Technologies, </i>pages 152-159.</p><p>Julie Weeds and David Weir. 2005. Co-occurrence re­trieval: a flexible framework for lexical distributional similarity. <i>Computational Linguistics, </i>31(4):439-475.</p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">68</doubt></references></body></article>