<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="783"/><title>Active Learning for Word Sense Disambiguation with Methods for Addressing the Class Imbalance Problem</title><pubinfo>Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning,pp. 783-790, Prague, June 2007. ©2007 Association for Computational Linguistics</pubinfo><author surname="Zhu" givenname="Jingbo"><org  name="University of Southern California" country="USA" city="Marina del Rey"/></author><author surname="Hovy" givenname="Eduard"><org  name="University of Southern California" country="USA" city="Marina del Rey"/></author></firstpageheader><frontmatter><p><b>Active Learning for Word Sense Disambiguation with Methods for Addressing the Class Imbalance Problem</b></p><p><b>Jingbo Zhu</b></p><p>University of Southern California</p><p>Information Sciences Institute Northeastern University, P.R.China Natural Language Processing Lab</p><p>Zhujingbo@mail.neu.edu.cn</p><p><b>Eduard Hovy</b></p><p>University of Southern California Information Sciences Institute 4676 Admiralty Way Marina del Rey, CA 90292-6695</p><p>hovy@isi.edu</p></frontmatter><abstract>In this paper, we analyze the effect of resampling techniques, including under-sampling and over-sampling used in active learning for word sense disambiguation (WSD). Experimental results show that un­der-sampling causes negative effects on ac­tive learning, but over-sampling is a rela­tively good choice. To alleviate the within-class imbalance problem of over-sampling, we propose a bootstrap-based over-sampling (BootOS) method that works bet­ter than ordinary over-sampling in active learning for WSD. Finally, we investigate when to stop active learning, and adopt two strategies, max-confidence and min-error, as stopping conditions for active learning. According to experimental results, we sug­gest a prediction solution by considering max-confidence as the upper bound and min-error as the lower bound for stopping conditions. </abstract></header><body><section number="1" title="Introduction"><p>Word sense ambiguity is a major obstacle to accu­rate information extraction, summarization, and machine translation (Ide and Veronis, 1998). In recent years, a variety of techniques for machine learning algorithms have demonstrated remarkable performance for automated word sense disam­biguation (WSD) (Chan and Ng, 2006; Dagan et. al., 2006; Xue <i>et. al., </i>2006; Kohomban and Lee. 2005; Dang and Palmer, 2005), when enough la­beled training data is available. However, creating a large sense-tagged corpus is very expensive and time-consuming, because these data have to be an­notated by human experts.</p><p>Among the techniques to solve the knowledge bottleneck problem, active learning is a promising way (Lewis and Gale, 1994; McCallum and Ni-gram, 1998). The purpose of active learning is to minimize the amount of human labeling effort by having the system automatically select for human annotation the most informative unannotated case.</p><p>In real-world data, the distribution of the senses of a word is often very skewed. Some studies re­ported that simply selecting the predominant sense provides superior performance, when a highly skewed sense distribution and insufficient context exist (Hoste <i>et al., </i>2001; McCarthy <i>et. al., </i>2004). The data set is <i>imbalanced </i>when at least one of the senses is heavily underrepresented compared to the other senses. In general, a WSD classifier is de­signed to optimize overall accuracy without taking into account the class imbalance distribution in a real-world data set. The result is that the classifier induced from imbalanced data tends to over-fit the predominant class and to ignore small classes (Japkowicz and Stephen, 2002). Recently, much work has been done in addressing the class imbalance problem, reporting that resampling methods such as <i>over-sampling </i>and <i>under-sampling </i>are useful in supervised learning with imbalanced data sets to induce more effective classifiers (Estabrooks <i>et al., </i>2004; Zhou and Liu, 2006).</p><p>In general framework of active learning, the learner (i.e. supervised classifier) is formed by us­ing supervised learning algorithms. To date, how­ever, no-one has studied the effects of over-sampling and under-sampling on active learning methods.<page local="2" global="784"/> In this paper, we study active learning with resampling methods addressing the class im­balance problem for WSD. It is noteworthy that neither of these techniques need modify the architecture or learning algorithm, making them very easy to use and extend to other domains.</p><p>Another problem in active learning is knowing when to stop the process. We address this problem in this paper, and discuss how to form the final classifier for use. This is a problem of estimation of classifier effectiveness (Lewis and Gale, 1994). Because it is difficult to know when the classifier reaches maximum effectiveness, previous work used a simple stopping condition when the training set reaches desirable size. However, in fact it is almost impossible to predefine an appropriate size of desirable training data for inducing the most effective classifier. To solve the problem, we consider the problem of estimation of classifier effectiveness as a second task of estimating classifier confidence. This paper adopts two strategies: <i>max-confidence </i>and <i>min-error, </i>and sug­gests a prediction solution by considering max-confidence as the upper bound and min-error as the lower bound for the stopping conditions.</p></section><section number="2" title="Related Work"><p>The ability of the active learner can be referred to as <i>selective sampling, </i>of which two major schemes exist: <i>uncertainty sampling </i>and <i>committee-based sampling. </i>The former method, for example pro­posed by Lewis and Gale (1994), is to use only one classifier to identify unlabeled examples on which the classifier is least confident. The latter method (McCallum and Nigam, 1998) generates a commit­tee of classifiers (always more than two classifiers) and selects the next unlabeled example by the prin­ciple of maximal disagreement among these classi­fiers. With selective sampling, the size of the train­ing data can be significantly reduced for text classification (Lewis and Gale, 1994; McCallum and Nigam, 1998), and word sense disambiguation (Chen, <i>et al. </i>2006).</p><p>A method similar to committee-based sampling is <i>co-testing </i>proposed by Muslea et al. (2000), which trains two learners individually on two compatible and uncorrelated views that should be able to reach the same classification accuracy. In practice, however, these conditions of view selection are difficult to meet in real-world word sense disambiguation tasks.</p><p>Recently, much work has been done on the class imbalance problem. The well-known approach is <i>resampling, </i>in which some training material is du­plicated. Two types of popular resampling methods exist for addressing the class imbalance problem: <i>over-sampling </i>and <i>under-sampling. </i>The basic idea of resampling methods is to change the training data distribution and make the data more balanced. It works ok in supervised learning, but has not been tested in active learning. Previous work re­ports that cost-sensitive learning is a good solution to the class imbalance problem (Weiss, 2004). In practice, for WSD, the costs of various senses of a disambiguated word are unequal and unknown, and they are difficult to evaluate in the process of learning.</p><p>In recent years, there have been attempts to ap­ply active learning for word sense disambiguation (Chen <i>et al., </i>2006). However, to our best knowl­edge, there has been no such attempt to consider the class imbalance problem in the process of ac­tive learning for WSD tasks.</p></section><section number="3" title="Resampling Methods 3.1 Under-sampling"><p>Under-sampling is a popular method in addressing the class imbalance problem by changing the train­ing data distribution by removing some exemplars of the majority class at random. Some previous work reported that under-sampling is effective in learning on large imbalanced data sets (Japkowicz and Stephen, 2002). However, as under-sampling removes some potentially useful training samples, it could cause negative effects on the classifier per­formance.</p><p>One-sided sampling is a method similar to un­der-sampling, in which redundant and borderline training examples are identified and removed from training data (Kubat and Matwin, 1997). Kuban and Matwin reported that one-sided sampling is effective in learning with two-class large imbal-anced data sets. However, the relative computa­tional cost of one-sided sampling in active learning is very high, because sampling computations must be implemented for each learning iteration. Our primitive experimental results show that, in the multi-class problem of WSD, one-sided sampling degrades the performance of active learning. And due to the high computation complexity of one­sided sampling, we use random under-sampling in our comparison experiments instead.<page local="3" global="785"/></p><p>To control the degree of change of the training data distribution, the ratio of examples from the majority and the minority class after removal from the majority class is called the <i>removal rate </i>(Jo and Japkowicz, 2004). If the removal rate is 1.0, then under-sampling methods build data sets with complete class balance. However, it was reported previously that perfect balance is not always the optimal rate (Estabrooks <i>et al., </i>2004). In our com­parison experiments, we set the removal rate for under-sampling to 0.8, since some cases have 0.8 as the optimal rate reported in (Estabrooks <i>et al.,</i> 2004).</p><subsection number="3.2" title="Over-sampling"><p>Over-sampling is also a popular method in ad­dressing the class imbalance problem by resam­pling the small class until it contains as many ex­amples as the large one. In contrast to under-sampling, over-sampling is the process of adding examples to the minority class, and is accom­plished by random sampling and duplication. Be­cause the process of over-sampling involves making exact copies of examples, it usually in­creases the training cost and may lead to overfit-ting. There is a recent variant of over-sampling named SMOTE (Chawla <i>et al., </i>2002) which is a synthetic minority over-sampling technique. The authors reported that a combination of SMOTE and under-sampling can achieve better classifier performance in ROC space than only under-sampling the majority class.</p><p>In our comparison experiments, we use over-sampling, measured by a resampling rate called the <i>addition rate </i>(Jo and Japkowicz, 2004) that indi­cates the number of examples that should be added into the minority class. The addition rate for over-sampling is also set to 0.8 in our experiments.</p></subsection><subsection number="3.3" title="Bootstrap-based Over-sampling"><p>While over-sampling decreases the between-class imbalance, it increases the within-class imbalance (Jo and Japkowicz, 2004) because of the increase of exact copies of examples at random. To allevi­ate this within-class imbalance problem, we pro­pose a <i>bootstrap-based over-sampling </i>method (BootOS) that uses a bootstrap resampling tech­nique in the process of over-sampling. Bootstrapping, explained below, is a resampling technique similar to <i>jackknifing.</i><i></i></p><p>There are two reasons for choosing a bootstrap method as resampling technique in the process of over-sampling. First, using a bootstrap set can avoid exactly copying samples in the minority class. Second, the bootstrap method may give a smoothing of the distribution of the training sam­ples (Hamamoto <i>et al. </i>, 1997), which can alleviate the within-class imbalance problem cased by over-sampling.</p><p>To generate the bootstrap set, we use a well-known bootstrap technique proposed by Hama-moto <i>et al. </i>(1997) that does not select samples ran­domly, allowing all samples in the minority <u>class(es) an equal chance to be selected.</u><u></u>_ <b>Algorithm </b><i>BootOS(X, </i>N, r, k) <b>Input:</b><b> </b>Minority class sample set X={x1; x2,     xn} of size n; Difference in number of examples between the majority and the minority class = N; Addition rate = r (&lt; 1.0); Number of nearest neighbors = k. <b>Output: </b>bootstrap sample  set XB of size N*r</p><doubt alpha="45.5" length="22" tooSmall="False" monospace="0.0">=XU(xb1, xb2,xB(N-r))-</doubt><doubt alpha="53.3" length="15" tooSmall="False" monospace="0.0">1.Fori = 1ToN*r</doubt><p>2. <b>If </b>i == n then (*all samples in minority class sample set have been used*)</p></subsection></section><section number="3." title="j = 1; //the first sample is selected again"><doubt alpha="57.1" length="7" tooSmall="False" monospace="0.0">4. Else</doubt><doubt alpha="60.0" length="40" tooSmall="False" monospace="0.0">5. j = i; // the i-th sample is selected</doubt></section><section number="6." title="Endif"><p>7. Select j-th sample xj (also as xj,0) from X</p><p>8. Find the k nearest neighbor samples xj,1, xj,2, ., xj,k using similarity functions.</p></section><section number="9." title="Compute a bootstrap sample x Bi:"><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">1</doubt><doubt alpha="25.0" length="4" tooSmall="False" monospace="0.0">k +1</doubt><doubt alpha="57.1" length="7" tooSmall="False" monospace="0.0">l=0xj,l</doubt></section><section number="10." title="Endfor"></section><section number="11." title="return"><figure caption="Figure 1. The BootOS algorithm"></figure><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">4</doubt><p><b>Active Learning with Resampling</b></p><p>In this work, we are interested in selective sam­pling for pool-based active learning, and focus on uncertainty sampling (Lewis and Gale, 1994). The key point is how to measure the uncertainty of an unlabeled exemplar, and select a new exemplar with maximum uncertainty to augment the training data. The maximum uncertainty implies that the current classifier has the least confidence in its classification of this exemplar. The well-known <i>entropy </i>is a good uncertainty measurement widely where <i>U </i>is the uncertainty measurement function <i>H </i>represents the entropy function.<page local="4" global="786"/> In the WSD task, <i>p(sj\wi) </i>is the predicted probability of sense <i>s</i><i>j </i>outputted by the current classifier, when given a <u>sample <i>i </i>containing a disambiguated word <i>w,.</i></u><u><i></i></u><i>_</i> <b>Algorithm </b><i>Active-Learning-with-Resampling(L, U,m) </i><b>Input:</b><b> </b>Let L be initial small training data set; U the pool of unlabeled exemplars <b>Output: </b>labeled training data set L</p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">k</doubt><doubt alpha="62.5" length="64" tooSmall="False" monospace="0.0">used in active learning (zhang and Chen, 2002; Chenet al.,2006):</doubt><doubt alpha="44.4" length="36" tooSmall="False" monospace="0.0">U(i)=H(P)=-]Tp(s;|wt)logp(s,|wt) (1)</doubt><p>1. Resample L to generate new training data set L* using resampling techniques such as under-sampling, over-sampling or BootOS, and then use L* to train the initial classifier</p></section><section number="2." title="Loop while adding new instances into L"><p>a. use the current classifier to probabilistically la­bel all unlabeled exemplars in U</p><p>b. Based on active learning rules, present <i>m </i>top-ranked exemplars to <i>oracle </i>for labeling</p><p>c. Augment L with the <i>m </i>new exemplars, and re­move them from U</p><p>d. Resample L to generate new training data set L* using resampling techniques such as under-sampling, over-sampling, or BootOS, and use L* to retrain the current classifier 3. <b>Until </b>the predefined stopping condition is met.</p></section><section number="4." title="return _"><figure caption="Figure 2. Active learning with resampling"></figure><p>In step 1 and 2(d) in Fig. 2, if we do not gener­ate L*, and L is used directly to train the current classifier, we call it <i>ordinary active learning. </i>In the process of active learning, we used the entropy-based uncertainty measurement for all active learn­ing frameworks in our comparison experiments. Actually our active learning with resampling is a heterogeneous approach in which the classifier used to select new instances is different from the resulting classifier (Lewis and Catlett, 1994).</p><p>We utilize a maximum entropy (ME) model (Berger <i>et al., </i>1996) to design the basic classifier used in active learning for WSD. The advantage of the ME model is the ability to freely incorporate features from diverse sources into a single, well-grounded statistical model. A publicly available ME toolkit (Zhang <i>et. al., </i>2004) was used in our experiments. In order to extract the linguistic fea­tures necessary for the ME model, all sentences containing the target word were automatically partof-speech (POS) tagged using the Brill POS tagger (Brill, 1992). Three knowledge sources were used to capture contextual information: unordered single words in topical context, POS of neighboring words with position information, and local colloca­tions. These are same as three of the four knowl­edge sources used in (Lee and Ng, 2002). Their fourth knowledge source (named syntactic rela­tions) was not used in our work.</p></section><section number="5" title="Stopping Conditions"><p>In active learning algorithm, defining the stopping condition for active learning is a critical problem, because it is almost impossible for the human an-notator to label all unlabeled samples. This is a problem of estimation of classifier effectiveness (Lewis and Gale 1994). In fact, it is difficult to know when the classifier reaches maximum effectiveness. In previous work some researchers used a simple stopping condition when the training set reached a predefined desired size. It is almost impossible to predefine an appropriate size of desirable training data for inducing the most effective classifier.</p><p>To solve the problem, we consider the problem of estimating classifier effectiveness as the problem of confidence estimation of classifier on the remaining unlabeled samples. Concretely, if we find that the current classifier already has acceptably strong confidence on its classification results for all remained unlabeled data, we assume the current training data is sufficient to train the classifier with maximum effectiveness. In other words, if a classifier induced from the current training data has strong classification confidence on an unlabeled example, we could consider it as a redundant example.</p><p>Based on above analyses, we adopt here two stopping conditions for active learning:</p><p><b><i>• Max-confidence: </i></b>This strategy is based on uncertainty measurement, considering whether the entropy of each selected unlabeled example is less than a very small predefined threshold close to zero, such as 0.001.</p><p><b><i>• Min-error: </i></b>This strategy is based on feedback from the <i>oracle </i>when the active learner asks for true labels for selected unlabeled examples, considering whether the current trained classifier could correctly predict the labels or the accuracy performance of predictions on</p><page local="5" global="787"/><p>selected unlabeled examples is already larger than a predefined accuacy threshold. Once <i>max-confidence </i>and <i>min-error </i>conditions are met, the current classifier is assumed to have strong enough confidence on the classification results of all remained unlabeled data.</p><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">6</doubt><p><b>Evaluation</b></p><doubt alpha="50.0" length="8" tooSmall="False" monospace="0.0">6.1 Data</doubt><p>The data used for our comparison experiments were developed as part of the OntoNotes project (Hovy <i>et al., </i>2006), which uses the WSJ part of the Penn Treebank (Marcus <i>et al., </i>1993). The senses of noun words occurring in OntoNotes are linked to the Omega ontology. In OntoNotes, at least two humans manually annotate the coarse-grained senses of selected nouns and verbs in their natural sentence context. To date, OntoNotes has annotated several tens of thousands of examples, covering several hundred nouns and verbs, with an inter-annotator agreement rate of at least 90%.</p><p>Those 38 random chosen ambiguous nouns used in all following experiments are shown in Table 1. It is apparent that the sense distributions of most nouns are very skewed (frequencies shown in the</p><p>In the following active learning comparison experiments, we tested with five resampling methods including <i>random sampling </i>(Random), <i>uncertainty sampling </i>(Ordinary), <i>under-sampling, over-sampling, </i>and <i>BootOS. </i>The 1-NN technique was used for bootstrap-based resampling of BootOS in our experiments. A 5 by 5-fold cross-validation was performed on each noun's data.</p><p>We used 20% randomly chosen data for held-out evaluation and the other 80% as the pool of unlabeled data for each round of the active learning. For all words, we started with a randomly chosen initial training set of 10 examples, and we made 10 queries after each learning iteration.</p><p>In the evaluation, average <i>accuracy </i>and <i>recall </i>are used as measures of performances for each active learning method. Note that the <i>macro-average </i>way is adopted for recall evaluation in each noun WSD task. The accuracy measure indicates the percentage of testing instances correctly identified by the system. The macro-average recall measure indicates how well the system performs on each sense.</p><p><b>Experiment 1: Performance comparison ex­periments on active learning</b><page local="6" global="788"/></p><doubt alpha="87.0" length="23" tooSmall="True" monospace="0.0">Active learning for WSD</doubt><doubt alpha="21.4" length="103" tooSmall="True" monospace="0.0">10    30    50    70    90   110  130  150   170  190  210  230  250  270 290 Number of learned samples</doubt><figure caption="Figure 3. Average accuracy performance com­parison experiments"></figure><doubt alpha="0.0" length="9" tooSmall="True" monospace="0.0">0.44 0.42</doubt><doubt alpha="0.0" length="8" tooSmall="True" monospace="0.0">0.4 0.38</doubt><doubt alpha="20.2" length="109" tooSmall="True" monospace="0.0">10    30    50    70    90   110   130   150   170   190   210   230  250   270 290 Number of learned samples</doubt><figure caption="Figure 4. Average recall performance comparison experiments"></figure><doubt alpha="100.0" length="3" tooSmall="True" monospace="0.0">ccA</doubt><doubt alpha="0.0" length="4" tooSmall="True" monospace="0.0">0.84</doubt><doubt alpha="100.0" length="5" tooSmall="True" monospace="0.0">geaer</doubt><doubt alpha="0.0" length="5" tooSmall="True" monospace="0.0">0. 82</doubt><doubt alpha="0.0" length="3" tooSmall="True" monospace="0.0">0.8</doubt><doubt alpha="0.0" length="4" tooSmall="True" monospace="0.0">0.78</doubt><doubt alpha="0.0" length="3" tooSmall="True" monospace="0.0">0.5</doubt><doubt alpha="0.0" length="4" tooSmall="True" monospace="0.0">0.48</doubt><doubt alpha="100.0" length="2" tooSmall="True" monospace="0.0">re</doubt><doubt alpha="0.0" length="1" tooSmall="True" monospace="0.0">&gt;</doubt><doubt alpha="100.0" length="1" tooSmall="True" monospace="0.0">A</doubt><doubt alpha="0.0" length="4" tooSmall="True" monospace="0.0">0.36</doubt><table caption="Table 1. Data set used in experiments6.2 Results" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Words</p></td><td class="cell"><p>sense distribution</p></td><td class="cell"><p>words</p></td><td class="cell"><p>sense distribution</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Rate</p></td><td class="cell"><p>1025/182</p></td><td class="cell"><p>president</p></td><td class="cell"><p>936/157/17</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>People</p></td><td class="cell"><p>815/67/7/5</p></td><td class="cell"><p>part</p></td><td class="cell"><p>456/102/75/16</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Point</p></td><td class="cell"><p>471/88/37/19/9/6</p></td><td class="cell"><p>director</p></td><td class="cell"><p>517/23</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Revenue</p></td><td class="cell"><p>517/23</p></td><td class="cell"><p>bill</p></td><td class="cell"><p>348/130/40</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Future</p></td><td class="cell"><p>413/82/23</p></td><td class="cell"><p>order</p></td><td class="cell"><p>354/61/54/6/6</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Plant</p></td><td class="cell"><p>376/51</p></td><td class="cell"><p>board</p></td><td class="cell"><p>369/15</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Today</p></td><td class="cell"><p>238/149</p></td><td class="cell"><p>policy</p></td><td class="cell"><p>308/74</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Capital</p></td><td class="cell"><p>325/21/8</p></td><td class="cell"><p>term</p></td><td class="cell"><p>147/137/52/13</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>management</p></td><td class="cell"><p>210/130</p></td><td class="cell"><p>move</p></td><td class="cell"><p>302/13/5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Position</p></td><td class="cell"><p>97/75/67/61/10/7</p></td><td class="cell"><p>amount</p></td><td class="cell"><p>236/57/16</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Home</p></td><td class="cell"><p>267/17/16</p></td><td class="cell"><p>power</p></td><td class="cell"><p>154/134/15</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Leader</p></td><td class="cell"><p>244/38</p></td><td class="cell"><p>return</p></td><td class="cell"><p>191/35/29/12/9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>administration</p></td><td class="cell"><p>266/11</p></td><td class="cell"><p>payment</p></td><td class="cell"><p>201/69</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Account</p></td><td class="cell"><p>233/18/13</p></td><td class="cell"><p>control</p></td><td class="cell"><p>90/66/64/21/12/5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Lot</p></td><td class="cell"><p>221/20</p></td><td class="cell"><p>activity</p></td><td class="cell"><p>218/23</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Drug</p></td><td class="cell"><p>160/74</p></td><td class="cell"><p>building</p></td><td class="cell"><p>177/48/5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Estate</p></td><td class="cell"><p>214/11</p></td><td class="cell"><p>house</p></td><td class="cell"><p>112/71/25</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>development</p></td><td class="cell"><p>165/46/6</p></td><td class="cell"><p>network</p></td><td class="cell"><p>127/53/29</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Strategy</p></td><td class="cell"><p>198/11</p></td><td class="cell"><p>place</p></td><td class="cell"><p>69/63/50/18/5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>As shown in Fig. 3 and Fig. 4, when the number of learned samples for each noun is smaller than 120, the BootOS has the best performance, followed by over-sampling and ordinary method. As the num­ber of learned samples increases, ordinary, over-sampling and BootOS have similar performances on accuracy and recall. Our experiments also ex­hibit that random sampling method is the worst on both accuracy and recall.</p><p>Previous work (Estabrooks <i>et al., </i>2004) reported that under-sampling of the majority class (pre­dominant sense) has been proposed as a good means of increasing the sensitivity of a classifier to the minority class (infrequent sense). However, in our active learning experiments, under-sampling is apparently worse than ordinary, over-sampling and our BootOS. The reason is that in highly imbal-anced data, too many useful training samples of majority class are discarded in under-sampling, causing the performance of active learning to de­grade.</p><p><b>Experiment 2: Effectiveness of learning in­stances for infrequent senses</b></p><p>It is important to enrich the corpora by learning more instances for infrequent senses using active learning with less human labeling. This procedure not only makes the corpora 'richer', but also alleviates the domain dependence problem faced by corpus-based supervised approaches to WSD.</p><p>The objective of this experiment is to evaluate the performance of active learning in learning samples of infrequent senses from an unlabeled corpus. Due to highly skewed word sense distributions in our data set, we consider all senses other than the predominant sense as infrequent senses in this experiment.</p><doubt alpha="80.8" length="26" tooSmall="True" monospace="0.0">u, Active learning for WSD</doubt><doubt alpha="50.0" length="2" tooSmall="True" monospace="0.0">a&gt;</doubt><doubt alpha="0.0" length="36" tooSmall="True" monospace="0.0">» 0.45 .............................</doubt><doubt alpha="2.4" length="42" tooSmall="False" monospace="0.0">3       0.2I-'-'-'-'-'-'-'-'-'-'-'-'-'-'-1</doubt><doubt alpha="0.0" length="85" tooSmall="True" monospace="0.0">&amp; 0     20    40    60    80   100   120   140  160  180  200  220  240  260  280 300</doubt><doubt alpha="88.0" length="25" tooSmall="True" monospace="0.0">Number of learned samples</doubt><figure caption="Figure 5. Comparison experiments on learning in­stances for infrequent senses"></figure><p>Fig. 5 shows that random sampling is the worst in active learning for infrequent senses. The reason is very obvious: the sense distribution of the learned sample set by random sampling is almost identical to that of the original data set.</p><p>Under-sampling is apparently worse than ordi­nary active learning, over-sampling and BootOS methods. When the number of learned samples for each noun is smaller than 80, BootOS achieves slight better performance than ordinary active learning and over-sampling.</p><p>When the number of learned samples is larger than 80 and smaller than 160, these three methods exhibit similar performance. As the number of it­erations increases, ordinary active learning is slightly better than over-sampling and BootOS. In fact, after the 16th iteration (10 samples chosen in each iteration), results indicate that most instances for infrequent senses have been learned.</p><p><b>Experiment 3: Effectiveness of Stopping Condi­tions for active learning</b></p><p>To evaluate the effectiveness of two strategies <i>max-confidence </i>and <i>min-error </i>as stopping condi­tions of active learning, we first construct an ideal stopping condition when the classifier could reach the highest accuracy performance at the first time in the procedure of active learning. When the ideal stopping condition is met, it means that the current classifier has reached maximum effectiveness. In practice, it is impossible to exactly know when the ideal stopping condition is met before all unlabeled data are labeled by a human annotator. We only use this ideal method in our comparison experi­ments to analyze the effectiveness of our two pro­posed stopping conditions.</p><p>For general purpose, we focus on the ordinary active learning to design the basic system, and to evaluate the effectiveness of three stop conditions. In the following experiments, the entropy threshold used in <i>max-confidence </i>strategy is set to 0.001, and the accuracy threshold used in <i>min-error strategy </i>is set to 0.9.</p><p>In Table 2, the column <i>"Size"" </i>stands for the size of unlabeled data set of corresponding noun word used in active learning. There are two columns for each stopping condition: the left column "num" presents number of learned instances and the right column <i>"%" </i>presents its percentage over all data when the corresponding stopping condition is met.</p><page local="7" global="789"/><p>As shown in Table 2, the min-error strategy based on feedback of human annotator is very close to the ideal method. Therefore, when com­paring to ideal stopping condition, min-error strat­egy is a good choice as stopping condition for ac­tive learning. It is important to note that the min-error method does not need more additional computational costs, it only depends upon the feedback of human annotator when labeling the chosen unlabeled samples.</p><p>From experimental results, we can see that max-confidence strategy is worse than min-error method. However, we believe that the entropy of each unlabeled sample is a good signal to stop ac­tive learning. So we suggest that there may be a good prediction solution in which the min-error strategy is used as the lower-bound of stopping condition, and max-confidence strategy as the up­per-bound of stopping condition for active learning.</p></section><section number="7" title="Discussion"><p>As discussed above, finding more instances for infrequent senses at the earlier stages of active learning is very significant in making the corpus richer, meaning less effort for human labeling. In practice, another way to learn more instances for infrequent senses is to first build a training data set by active learning or by human efforts, and then build a supervised classifier to find more instances for infrequent sense. However, it is interesting to know how much initial training data is enough for this task, and how much human labeling efforts could be saved.</p><p>From experimental results, we found that among these chosen unlabeled instances by active learner, some instances are informative samples helpful for improving classification performance, and other instances are borderline samples which are unreli­able because even a small amount of noise can lead the sample to the wrong side of the decision boundary. The removal of these borderline samples might improve the performance of active learning.</p><p>The proposed prediction solution based on max-confidence and min-error strategies is a coarse framework. To predict when to stop active learning procedure, it is logical to consider the changes of accuracy performance of the classifier as a signal to stop the learning iteration. In other words, dur­ing the range predicted by the proposed solution, if the change of accuracy performance of the learner (classifier) is very small, we could assume that the current classifier has reached maximum effective­ness.</p></section><section number="8" title="Conclusion and Future Work"><p>In this paper, we consider the class imbalance problem in WSD tasks, and analyze the effect of resampling techniques including over-sampling and under-sampling in active learning. Experimen­tal results show that over-sampling is a relatively good choice in active learning for WSD in highly imbalanced data. Under-sampling causes negative effect on active learning. A new over-sampling method named BootOS based on bootstrap tech­nique is proposed to alleviate the within-class im­balance problem of over-sampling, and works bet­ter than ordinary over-sampling in active learning for WSD. It is noteworthy that none of these techniques require to modify the architecture or learning algorithm; therefore, they are very easy to use and extend to other applications.<page local="8" global="790"/> To predict when to stop active learning, we adopt two strategies including max-confidence and min-error as stopping conditions. According to our experimental results, we suggest a prediction solution by considering max-confidence as the upper bound and min-error as the lower bound of stopping conditions for active learning.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Words</p></td><td class="cell"><p>Size</p></td><td class="cell"><p>Ideal</p></td><td class="cell"><p>Max-confidence</p></td><td class="cell"><p>Min-error</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>num %</p></td><td class="cell"><p>num %</p></td><td class="cell"><p>num %</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Rate</p></td><td class="cell"><p>966</p></td><td class="cell"><p>200</p></td><td class="cell"><p>.23</p></td><td class="cell"><p>410</p></td><td class="cell"><p>.41</p></td><td class="cell"><p>290</p></td><td class="cell"><p>.29</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>People</p></td><td class="cell"><p>715</p></td><td class="cell"><p>140</p></td><td class="cell"><p>.20</p></td><td class="cell"><p>290</p></td><td class="cell"><p>.41</p></td><td class="cell"><p>200</p></td><td class="cell"><p>.28</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Point</p></td><td class="cell"><p>504</p></td><td class="cell"><p>90</p></td><td class="cell"><p>.18</p></td><td class="cell"><p>220</p></td><td class="cell"><p>.44</p></td><td class="cell"><p>120</p></td><td class="cell"><p>.24</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Revenue</p></td><td class="cell"><p>432</p></td><td class="cell"><p>70</p></td><td class="cell"><p>.16</p></td><td class="cell"><p>110</p></td><td class="cell"><p>.25</p></td><td class="cell"><p>80</p></td><td class="cell"><p>.19</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Future</p></td><td class="cell"><p>414</p></td><td class="cell"><p>120</p></td><td class="cell"><p>.29</p></td><td class="cell"><p>140</p></td><td class="cell"><p>.34</p></td><td class="cell"><p>60</p></td><td class="cell"><p>.14</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Plant</p></td><td class="cell"><p>342</p></td><td class="cell"><p>210</p></td><td class="cell"><p>.61</p></td><td class="cell"><p>180</p></td><td class="cell"><p>.53</p></td><td class="cell"><p>110</p></td><td class="cell"><p>.32</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Today</p></td><td class="cell"><p>382</p></td><td class="cell"><p>250</p></td><td class="cell"><p>.65</p></td><td class="cell"><p>240</p></td><td class="cell"><p>.63</p></td><td class="cell"><p>230</p></td><td class="cell"><p>.60</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Capital</p></td><td class="cell"><p>283</p></td><td class="cell"><p>70</p></td><td class="cell"><p>.25</p></td><td class="cell"><p>180</p></td><td class="cell"><p>.64</p></td><td class="cell"><p>90</p></td><td class="cell"><p>.32</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Management</p></td><td class="cell"><p>272</p></td><td class="cell"><p>200</p></td><td class="cell"><p>.74</p></td><td class="cell"><p>210</p></td><td class="cell"><p>.77</p></td><td class="cell"><p>210</p></td><td class="cell"><p>.77</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Position</p></td><td class="cell"><p>254</p></td><td class="cell"><p>210</p></td><td class="cell"><p>.83</p></td><td class="cell"><p>230</p></td><td class="cell"><p>.91</p></td><td class="cell"><p>220</p></td><td class="cell"><p>.87</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Home</p></td><td class="cell"><p>240</p></td><td class="cell"><p>60</p></td><td class="cell"><p>.25</p></td><td class="cell"><p>160</p></td><td class="cell"><p>.67</p></td><td class="cell"><p>60</p></td><td class="cell"><p>.25</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Leader</p></td><td class="cell"><p>226</p></td><td class="cell"><p>60</p></td><td class="cell"><p>.27</p></td><td class="cell"><p>120</p></td><td class="cell"><p>.53</p></td><td class="cell"><p>70</p></td><td class="cell"><p>.31</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>administration</p></td><td class="cell"><p>222</p></td><td class="cell"><p>30</p></td><td class="cell"><p>.14</p></td><td class="cell"><p>90</p></td><td class="cell"><p>.41</p></td><td class="cell"><p>50</p></td><td class="cell"><p>.23</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Account</p></td><td class="cell"><p>211</p></td><td class="cell"><p>50</p></td><td class="cell"><p>.24</p></td><td class="cell"><p>130</p></td><td class="cell"><p>.62</p></td><td class="cell"><p>70</p></td><td class="cell"><p>.33</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Lot</p></td><td class="cell"><p>185</p></td><td class="cell"><p>30</p></td><td class="cell"><p>.16</p></td><td class="cell"><p>60</p></td><td class="cell"><p>.32</p></td><td class="cell"><p>40</p></td><td class="cell"><p>.22</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Drug</p></td><td class="cell"><p>187</p></td><td class="cell"><p>130</p></td><td class="cell"><p>.70</p></td><td class="cell"><p>140</p></td><td class="cell"><p>.75</p></td><td class="cell"><p>120</p></td><td class="cell"><p>.64</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Estate</p></td><td class="cell"><p>180</p></td><td class="cell"><p>20</p></td><td class="cell"><p>.11</p></td><td class="cell"><p>50</p></td><td class="cell"><p>.28</p></td><td class="cell"><p>30</p></td><td class="cell"><p>.17</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Development</p></td><td class="cell"><p>174</p></td><td class="cell"><p>40</p></td><td class="cell"><p>.23</p></td><td class="cell"><p>150</p></td><td class="cell"><p>.86</p></td><td class="cell"><p>80</p></td><td class="cell"><p>.46</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Strategy</p></td><td class="cell"><p>167</p></td><td class="cell"><p>10</p></td><td class="cell"><p>.06</p></td><td class="cell"><p>100</p></td><td class="cell"><p>.60</p></td><td class="cell"><p>10</p></td><td class="cell"><p>.06</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>President</p></td><td class="cell"><p>888</p></td><td class="cell"><p>120</p></td><td class="cell"><p>.14</p></td><td class="cell"><p>220</p></td><td class="cell"><p>.25</p></td><td class="cell"><p>120</p></td><td class="cell"><p>.14</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Part</p></td><td class="cell"><p>519</p></td><td class="cell"><p>110</p></td><td class="cell"><p>.21</p></td><td class="cell"><p>240</p></td><td class="cell"><p>.46</p></td><td class="cell"><p>130</p></td><td class="cell"><p>.25</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Director</p></td><td class="cell"><p>432</p></td><td class="cell"><p>110</p></td><td class="cell"><p>.25</p></td><td class="cell"><p>130</p></td><td class="cell"><p>.30</p></td><td class="cell"><p>90</p></td><td class="cell"><p>.21</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Bill</p></td><td class="cell"><p>414</p></td><td class="cell"><p>120</p></td><td class="cell"><p>.29</p></td><td class="cell"><p>280</p></td><td class="cell"><p>.68</p></td><td class="cell"><p>150</p></td><td class="cell"><p>.36</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Order</p></td><td class="cell"><p>385</p></td><td class="cell"><p>130</p></td><td class="cell"><p>.34</p></td><td class="cell"><p>220</p></td><td class="cell"><p>.57</p></td><td class="cell"><p>140</p></td><td class="cell"><p>.36</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Board</p></td><td class="cell"><p>307</p></td><td class="cell"><p>40</p></td><td class="cell"><p>.13</p></td><td class="cell"><p>190</p></td><td class="cell"><p>.62</p></td><td class="cell"><p>40</p></td><td class="cell"><p>.13</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Policy</p></td><td class="cell"><p>306</p></td><td class="cell"><p>90</p></td><td class="cell"><p>.29</p></td><td class="cell"><p>200</p></td><td class="cell"><p>.65</p></td><td class="cell"><p>150</p></td><td class="cell"><p>.49</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Term</p></td><td class="cell"><p>279</p></td><td class="cell"><p>120</p></td><td class="cell"><p>.43</p></td><td class="cell"><p>190</p></td><td class="cell"><p>.68</p></td><td class="cell"><p>130</p></td><td class="cell"><p>.47</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Move</p></td><td class="cell"><p>256</p></td><td class="cell"><p>50</p></td><td class="cell"><p>.20</p></td><td class="cell"><p>140</p></td><td class="cell"><p>.55</p></td><td class="cell"><p>50</p></td><td class="cell"><p>.20</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Amount</p></td><td class="cell"><p>247</p></td><td class="cell"><p>210</p></td><td class="cell"><p>.85</p></td><td class="cell"><p>200</p></td><td class="cell"><p>.81</p></td><td class="cell"><p>140</p></td><td class="cell"><p>.57</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Power</p></td><td class="cell"><p>242</p></td><td class="cell"><p>190</p></td><td class="cell"><p>.78</p></td><td class="cell"><p>190</p></td><td class="cell"><p>.78</p></td><td class="cell"><p>190</p></td><td class="cell"><p>.78</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Return</p></td><td class="cell"><p>221</p></td><td class="cell"><p>90</p></td><td class="cell"><p>.41</p></td><td class="cell"><p>160</p></td><td class="cell"><p>.72</p></td><td class="cell"><p>100</p></td><td class="cell"><p>.45</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Payment</p></td><td class="cell"><p>216</p></td><td class="cell"><p>120</p></td><td class="cell"><p>.56</p></td><td class="cell"><p>160</p></td><td class="cell"><p>.74</p></td><td class="cell"><p>150</p></td><td class="cell"><p>.69</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Control</p></td><td class="cell"><p>206</p></td><td class="cell"><p>160</p></td><td class="cell"><p>.78</p></td><td class="cell"><p>200</p></td><td class="cell"><p>.97</p></td><td class="cell"><p>200</p></td><td class="cell"><p>.97</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Activity</p></td><td class="cell"><p>193</p></td><td class="cell"><p>30</p></td><td class="cell"><p>.16</p></td><td class="cell"><p>130</p></td><td class="cell"><p>.67</p></td><td class="cell"><p>70</p></td><td class="cell"><p>.36</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Building</p></td><td class="cell"><p>184</p></td><td class="cell"><p>90</p></td><td class="cell"><p>.49</p></td><td class="cell"><p>130</p></td><td class="cell"><p>.71</p></td><td class="cell"><p>110</p></td><td class="cell"><p>.60</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>House</p></td><td class="cell"><p>166</p></td><td class="cell"><p>100</p></td><td class="cell"><p>.60</p></td><td class="cell"><p>150</p></td><td class="cell"><p>.90</p></td><td class="cell"><p>110</p></td><td class="cell"><p>.66</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Network</p></td><td class="cell"><p>167</p></td><td class="cell"><p>110</p></td><td class="cell"><p>.66</p></td><td class="cell"><p>130</p></td><td class="cell"><p>.78</p></td><td class="cell"><p>100</p></td><td class="cell"><p>.60</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Place</p></td><td class="cell"><p>164</p></td><td class="cell"><p>120</p></td><td class="cell"><p>.73</p></td><td class="cell"><p>150</p></td><td class="cell"><p>.91</p></td><td class="cell"><p>120</p></td><td class="cell"><p>.73</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Table 2 Effectiveness of three stopping conditions</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>In the future work, we will study how to exactly identify these borderline samples thus they are not firstly selected in active learning procedure. The borderline samples have the higher entropy values meaning least confident for the current classifier. The borderline instances can be detected using the concept of <i>Tomek links </i>(Tomek 1976). It is also worth studying cost-sensitive learning for active learning with imbalanced data, and using such techniques for WSD.</p></section><references><p>A. L. Berger, S. A. Della, and V. J Della. 1996. <i>A maximum entropy approach to natural language processing. </i>Compu­tational Linguistics 22(1):39-71.</p><p>E Brill. 1992. <i>A simple rule-based part of speech tagger. </i>In the Proceedings of the Third Conference on Applied Natu­ral Language Processing.</p><p>Y. S. Chan and H. T. Ng. 2006. <i>Estimating class priors in domain adaptation. </i>In Proc. of ACL06.</p><p>N. Chawla, K. Bowyer, L. Hall, W. Kegelmeyer. 2002. <i>SMOTE: synthetic minority over-sampling technique. </i>Jour­nal of Artificial Intelligence Research, 2002(16): 321-357</p><p>J. Chen, A. Schein, L. Ungar, M. Palmer. 2006. <i>An empirical study of the behavior of active learning for word sense dis­ambiguation. </i>In Proc. of HLT-NAACL06</p><p>I. Dagan, O. Glickman, A. Gliozzo, E. Marmorshtein, and C. Strapparava. 2006. <i>Direct Word Sense Matching for Lexi­cal Substitution. </i>In Proc. of ACL'06</p><p>H. T. Dang and M. Palmer. 2005. <i>The Role of Semantic Roles in Disambiguating Verb Senses. </i>In Proc. of ACL'05.</p><p>A. Estabrooks, T. Jo and N. Japkowicz. 2004. <i>A multiple re­sampling method for learning from imbalanced data set. </i>Computational Intelligence, 20(1):18-36</p><p>Y. Hamamoto, S. Uchimura and S. Tomita. 1997. <i>A bootstrap technique for nearest neighbor classifier design. </i>IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(1):73-79</p><p>V. Hoste, A. Kool, and W. Daelemans. 2001. <i>Classifier opti­mization and combination in the English all words task. </i>In Proc. of the SENSEVAL-2 workshop</p><p>E. Hovy, M. Marcus, M. Palmer, L. Ramshaw and R. Weischedel. 2006. <i>Ontonotes: The 90% Solution. </i>In Proc. of HLT-NAACL06.</p><p>N. Ide and J. Veronis. 1998. <i>Introduction to the special issue on word sense disambiguation: the state of the art. </i>Compu­tational Linguistics, 24(1):1-37</p><p>N. Japkowicz and S. Stephen. 2002. <i>The class imbalance problem: a systematic study. </i>Intelligent Data Analysis,</p><doubt alpha="0.0" length="12" tooSmall="False" monospace="0.0">6(5):429-450</doubt><p>T. Jo and N. Japkowicz. 2004. <i>Class imbalances versus small disjuncts. </i>SIGKSS Explorations, 6(1):40-49</p><p>U. S. Kohomban and W. S. Lee. 2005. <i>Learning Semantic Classes for Word Sense Disambiguation. </i>In Proc. of</p><doubt alpha="50.0" length="6" tooSmall="False" monospace="0.0">ACL'05</doubt><p>M. Kubat and S. Matwin. 1997. <i>Addressing the curse of im-balanced training sets: one-sided selection. </i>In Proc. of</p><doubt alpha="66.7" length="6" tooSmall="False" monospace="0.0">ICML97</doubt><p>Y.K. Lee and. H.T. Ng. 2002. <i>An empirical evaluation of knowledge sources and learning algorithm for word sense disambiguation. </i>In Proc. of EMNLP-2002</p><p>D. D. Lewis and W. A. Gale. 1994. <i>A sequential algorithm for training text classifiers. </i>In Proc. of SIGIR-94</p><p>D.D. Lewis and J. Catlett. 1994. <i>Heterogeneous uncertainty sampling for supervised learning. </i>In Proc. of ICML94</p><p>M. Marcus, B. Santorini, and M. A. Marcinkiewicz. 1993. <i>Building a large annotated corpus of English: the Penn Treebank. </i>Computational Linguistics, 19(2):313-330</p><p>A. McCallum and K. Nigram. 1998. <i>Employing EM in pool-based active learning for text classification. </i>In Proc. 15th</p><doubt alpha="100.0" length="4" tooSmall="False" monospace="0.0">ICML</doubt><p>D. McCarthy, R. Koeling, J. Weeds, and J. Carroll. 2004.</p><p><i>Finding predominant senses in untagged text. </i>In Proc. of</p><doubt alpha="60.0" length="5" tooSmall="False" monospace="0.0">ACL04</doubt><p>I. Muslea, S. Minton, and C. A. Knoblock. 2000. <i>Selective sampling with redundant views. </i>In Proc. of National Con­ference on Artificial Intelligence</p><doubt alpha="66.3" length="104" tooSmall="False" monospace="0.0">I. Tomek. 1976.Two modifications of CNN.IEEE Transac­tions on Systems, Man and Cybernetics, 6(6):769-772</doubt><p>G. M. Weiss. 2004. <i>Mining with rarity - problems and solu­tions: a unifying framework. </i>SIGKDD Explorations, 6(1):7-19</p><p>N. Xue, J. Chen and M. Palmer. 2006. <i>Aligning Features with Sense Distinction Dimensions. </i>In Proc. of ACL'06</p><p>Z. Zhou, X. Liu. 2006. <i>Training cost-sensitive neural networks with methods addressing the class imbalance problem. </i>IEEE Transactions on Knowledge and Data Engineering,</p><doubt alpha="0.0" length="11" tooSmall="False" monospace="0.0">18(1):63-77</doubt><p>L. Zhang, J. Zhu, and T. Yao. 2004. <i>An evaluation of statisti­cal spam filtering techniques. </i>ACM Transactions on Asian Language Information Processing, 3(4):243-269.</p><p>C. Zhang and T. Chen. 2002. <i>An active learning frame­work for content-based information retrieval. </i>IEEE Transactions on Multimedia, 4(2):260-268</p></references></body></article>