Bootstrapping, or semisupervised For many language-processing learning, has become an important topic in com- putational of linguistics. tasks, there are an abundance unlabeled making data, but labeled data are lacking and too expensive to create in large quantities, The widely Yarowsky (1995) bootstrapping techniques in was one of desirable. algorithm the In first bootstrapping it of algorithms to The be- come or known computational is linguistics. brief, consists base supervised learning algorithm. Specifically, two loops. “inner loop” learner a a decision list learner that considers rules of the form “If instance Yarowsky uses simple tains f, j” and selects those rules whose on x con- feature is then predict label precision the training The data highest. “outer loop” of is given a seed set of rules to start with. It In each iteration, which rules to assign labels to unlabeled data. selects those instances it uses the current set regarding the from base learner’s is, set them. It predictions then calls the inner are most confident and constructs a labeled (that training of loop to construct a new classifier a new set rules), and the cycle An algorithm, (Blum repeats. alternative more popular, co-training in it and Mitchell proven 1998), has subsequently become (Dasgupta, Littman, perhaps part McAllester because has in amenable Yarowsky to theoretical analysis which is mathematically and 2001), contrast The to the al- gorithm, as of yet increasing poorly understood. current of article Yarowsky aims to rectify this lack understanding, The Yarowsky the attractiveness the algorithm of as an alternative of to co-training. on it algorithm does have Co-training the ad- vantage placing less a restriction into the data sets can be applied to. requires data attributes to be separable Yarowsky two views its the target label; the algorithm makes that are conditionally indepen- dent given no such assumption about data. In previous , work, which I did propose Yarowsky an assumption about the data called precision independence That under is the unsatisfactory, algorithm could however, be shown ney assumption ultimately not only effective (Ab- 2002). because it ∗ 4080 Frieze Bldg., 105 S. State Street, Ann Arbor, MI 48109-1285. E-mail: abney.umich.edu. Submission received: 10 February 26 August 2003; Revised submission received: 21 December 2003; Accepted for publication: 2004 © c 2004 Association for Computational Linguistics Computational Linguistics Volume 30, Number 3 Table The 1 Yarowsky K. algorithm variants. Y-1/DL-EM reduces H; the others reduce [BAR] Y-1/DL-EM- Λ Y-1/DL-EM-X Y-1/DL-1-R Y-1/DL-1-VS YS-P YS-R YS-FS EM EM inner inner loop that uses labeled examples only Near-original loop Yarowsky that uses inner all examples Near-original Yarowsky inner loop, no smoothing Sequential loop, “variable smoothing” Sequential update, “antismoothing” Sequential update, original no smoothing update, Yarowsky smoothing restricts internal the data sets on which the algorithm can would be shown effective, but also for ad- ditional it reasons. A detailed independence discussion is property take us too it would far afield here, but suffice to say that precision from more a properties that not to derive basic of be preferable to assume, but rather which shows Yarowsky that precision independence a data set, and fails to be satisfied in that closer empirical on study is some data This sets it article proposes the algorithm Yarowsky a approach. effective. different optimizing Instead of making objective assumptions the algorithm function. We about data, views the of as optimize the algorithm (though an not the algorithm in will show original that several variants or precisely either log likelihood H alternative objective its form) K, we imposes negative an upper bound on fortunately, like to show that the Yarowsky H. an function, Ideally, that we would But we algorithm minimizes Yarowsky not which able we to do so. Y-1/DL-EM, are able to show that a variant of H. Un- are outer call decreases H in each iteration. It the algorithm, of Yarowsky with inner on com- bines the loop the (EM) algorithm a different loop based the expectation-maximization A second its inner proposed variant is of algorithm. the Yarowsky original algorithm, Yarowsky Y-1/DL-1, Y-1/DL-EM, loop very similar to the inner has the advan- tage that whose inner original. Y-1/DL-1 loop, unlike it loop bears little resemblance the directly H, to the we show that it has disadvantage objective that does not K. reduce but does reduce the alternative in We function it also consider a third variant, (adding YS. It differs in from Y-1/DL-EM iteration), and Y-1/DL-1 that (updating updates sequentially in iteration). a single Besides rule each intrinsic rather parallel rules each having the interest than of in method YS all sequential update, in can original be proven Yarowsky effective when using in exactly the as the algorithm, contrast to Y-1/DL-1, same smoothing used uses or a nonstandard “variable smoothing.” YS is proven which either K. no smoothing to decrease The Yarowsky algorithm variants that we consider of are original summarized in Table extent these the the algorithm, we 1. To the that variants capture from original of its essence a effectiveness. Even if have better formal understanding we the variants obtained are deemed to depart of substantially the algorithm, mathematically have at least a family new bootstrapping algorithms that are understood.

2.1 The The an iterative original Original Yarowsky Algorithm Y-0 algorithm, One which begins with we refer to as Y-0, seed set Λ ( 0 ) of is given in table 2. It is algorithm. a labeled examples and a Abney Understanding the Yarowsky Algorithm Table The 2 generic Yarowsky algorithm (Y-0) (1) Given: examples X, and initial labeling Y ( 0 ) (2) For t ∈{ 0,1, ...} (2.1) Train classifier on labeled examples (Λ ( t ) ,Y ( t ) ) , where Λ ( t ) { x ∈ X | Y ( t ) = (2.2) The For resulting classifier predicts X: label j for example x with probability each example x ∈

(2.2.1) Set ŷ = argmax j π x

 Y ( 0 ) (2.2.2) Set Y ( 1 )  x t + x = ŷ (2.3) If Y t 1 Y ( t ) , ⊥ ( + ) = stop = ⊥} ( t + 1 ) π x ( j ) ( t + 1 ) ( j ) if x ∈ Λ ( 0 ) if ( t + 1 ) otherwise π x ( ŷ ) >ζ set V ( 0 ) of unlabeled examples. At each is iteration, a classifier is constructed from the labeled examples; then the classifier applied to the unlabeled examples to create a new of To labeled set. discuss the algorithm formally, we require some X. The notation. of We assume a F for each x ∈ set examples with first a set is examples f . Note X and feature X f if set and only x if f ∈ F . feature f X that x ∈ We also a series of x require labelings Y ( t ) , where t represents the iteration number. We write Y ( t ) for the label of example x under labeling Y ( t ) x . An unlabeled example is one for which Y ( t ) is undefined, in which case we write Y ( t ) = ⊥ . We write V ( t ) set of x Λ ( the of x the t ) for labeled examples. It will for unlabeled examples and of set with j: also be useful to have a notation for the set examples label Λ ( j t ) ≡{ x ∈ X | Y x ( t ) = j = ⊥} Note that Λ ( t ) is the disjoint union of the sets Λ ( t ) . When t is clear from context, we drop the superscript of ambiguity, ( t ) and write we will simply Λ j ,V, Y x , etc. At the with risk f, also index sometimes write Λ f for the set of trusting the to discriminate between Λ (labeled labeled examples examples with feature f) and Λ (labeled to examples with j). We f feature j represent and j and k to labels. The label reader may always wish use f and g to Table 3, which features represent throughout to refer to In iteration, summarizes Yarowsky notation used the article. on each the Let algorithm uses a supervised learner to train a clas- sifier the . ; is it labeled is examples. function from ( X,Y us call this supervised learner a ( makes to a classifier π from the base algorithm t ) ) drawn a space of learning It assumed the classifier confidence-weighted predictions. That classi- fiers is, Π that is the classifier defines a scoring function π ( x,j ) , and the predicted label for example x ŷ ≡ argmax π ( x,j ) j ( 1 ) Ties maximization are broken arbitrarily. Technically, we assume in a ordering, fixed order in over labels as the first label the of and define the It will returning case in be assume that the scoring function is a tie. over which convenient we to case normalize Henceforward, it to make we π ( x,j nonnegative and bounded, can labels j for a example x. write ) a conditional π x ( j ) instead distribution given of π ( x,j ) , Computational Linguistics Volume 30, Number 3 Table 3 Summary of notation. X Y Λ V x f,g j, F k Y x X x f , , Λ f ,V f m Λ j Λ fj L φ x ( j ) π x ( j ) θ fj ŷ H [[ Φ ]] H ( p ) K H ( p || q ) q f ( j ) q̃ f ( j ) q̂ f ( j ) j † j ∗ u ( · ) understanding π x to be a probability of distribution on over labels x. j. We call this distribution the prediction To distribution Specifically, iteration of the classifier complete an , ŷ is Yarowsky example the algorithm, one assigned to example x if recomputes labels for examples. the label . The the score 1 π x ( ŷ ) exceeds a threshold ζ called the labeling threshold new labeled set Λ ( t + ) which π ŷ >ζ . Relabeling only to examples inV 0 contains for ) applies ( ) 0 ( indelible, original manually . The all ex- amples in x 0 labels for examples opposed Λ ( ) are because Λ ( ) constitutes the to that have been labeled by the learning algorithm itself. labeled data, The as data Yarowsky algorithm continues is until convergence. in The particular base induced learning is algorithm that ministic uses of deterministic, Hence, the sense that the classifier is a deter- whatever function the labeled data. the algorithm known to have converged at We Note point the labeling remains unchanged. that the algorithm as stated leaves the base learning algorithm can generic Yarowsky algorithm Y-0, for which unspecified. distinguish is between open the includes an parameter, the and the specific Yarowsky algorithm , which base learning algorithm outer a specification of the base learner. inner Informally, of we call the The base learner the loop the specific Yarowsky generic algorithm the loop and the base that Yarowsky is decision list induction algorithm. learner of it Section assumes 3. a algorithm. We postpone discussion until set of examples, both labeled and unlabeled the current labeling; Y ( t ) is the labeling at iteration t the (current) (current) set of of labeled examples the index set unlabeled examples an example feature indices label indices the features of of example x the label example x; value is undefined ( ⊥ ) if x is unlabeled examples, with labeled examples, j, unlabeled with examples that have feature f examples of label of examples feature of a given example: | F f and x | (cf. label j the number features equation (12)) the number labels labeling distribution (equation (5)) prediction distribution we (equation (12); except for DL-0, which uses score j; θ the prediction distribution of equation (11)) for rule maximizes f → view f as label of π x is0or1 ( j ) for given x (equation (1) f that truth objective value Φ : value entropy of function, negative log-likelihood (equation (6)) cross entropy: distribution p − objective ∑ p ( x ) log q ( x on ) (cf. H equations (equation (2) (20)) and (3)) x function, of upper (equation bound precision rule f → (equation j (10)) (9)) smoothed precision “peaked” precision the label that maximizes (equation (25)) maximizes precision q f ( j ) for a given feature f (equation (equation label for given feature f (28)) (26)) the that rule score θ fj a uniform distribution Abney Understanding the Yarowsky Algorithm 2.2 An Objective Machine Function learning algorithms are typically measure of designed to optimize The some objective maximum-likelihood func- tion that rion , is represents with most a formal performance. commonly used objective function. Suppose we have a set of crite- the Λ labels probability x x ∈ , of and parametric family of models examples Y for Λ a x, π θ such that π ( j | x; θ is probability of label j to example according to the model, model. ) represents The the of assigning likelihood of θ , the the full data maximum-likelihood set according to the as a θ and the criterion instructs viewed function maximize or equivalently, us to choose the parameter settings θ ˆ that likelihood, log-likelihood: l ( θ ) = log ∏ π ( Y x | x; θ ) = ∑ log π ( Y x | x; θ ) x ∈ Λ = ∑∑ [[j = Y x ]]log π ( j | x; θ ) x ∈ Λ x ∈ Λ j (The zero notation otherwise.) [[ Φ ]] represents the truth value of the proposition Φ ; it is one if Φ is true and Let us define

for x ∈ Λ j: Note Specifically, that φ x satisfies it is the formal requirements with of its a mass probability distribution on over . We labels a point distribution Now we write all concentrated Y x call it the labeling distribution. can φ x ( j )= [[j = Y x ]] l ( θ ) = ∑∑ φ x ( j ) log π ( j | x; θ ) = − ∑ H ( φ x ||π x ) (2) x ∈ Λ x ∈ Λ j implicit. In (2) we We have written π x for the distribution π ( ·| x; θ ) , leaving the . nonstandard notation H ( p || q ) for what dependence is on θ have also It used is the sometimes called cross entropy easy to verify that H ( p || q )= H ( p )+ D ( p || q ) ( 3 ) where is H ( p ) is the entropy of p and a distribution, ( p )= 0 D is Kullback-Leibler divergence. and hence H ( p || q )= D ( p || q ) . In Note that when p point H particular: l ( θ )= − ∑ D ( φ x ||π x ) ( 4 ) x ∈ Λ Thus when, as instructing here, φ x is a point distribution, model we can as choose the that minimizes restate the maximum-likelihood criterion us to model’s the total divergence between . the empirical labeling distributions φ x and the prediction distribu- tions To π x extend ones l ( θ ) to whose unlabeled examples, we provide need only information. observe that Accordingly, unlabeled exam- ples are about labels the data no we Computational Linguistics Volume 30, Number 3 revise is the definition of maximally φ x to treat unlabeled examples which is as ones tion uncertain distribution, to say, whose labeling distribu- the the uniform distribution: ( 5 ) where L is the number of labels. Equivalently: When must we use (2). Since replace Λ with X, expressions (2) and )= H ( φ φ ||π ) , (4) are no and H ( φ ) is longer ||π )+ D ( minimized equivalent; when we minimizing H ( φ x x x one x x x On x is labeled, H ( φ x ||π x ) ,and forces to label is minimized unlabeled φ D ( ||π ) D ( ||π ) when examples. the labels of labeled ex- amples, H with ( x ||π x )= φ x of x model. φ x x examples agree In the we predictions objective the short, adopt as function H ≡ ∑ x ∈ X H ( φ x ||π x )= − l ( φ , θ ) ( 6 ) We seek to minimize H. 2.3 We The Modified Algorithm can Two modifications that a modified Y-1 of H. show version of the Yarowsky algorithm finds a local minimum are necessary: • • The with labeling function Y is recomputed once in each iteration as before, may an example labeled stays labeled. The but the constraint that label change, but a labeled example cannot become unlabeled again. We eliminate only the threshold ζ or (equivalently) fix it at 1 / L. As a result, the which examples is that remain unlabeled after The problem the labeling with step are those for is π x the it uniform distribution. minimum prevents the algorithm from an arbitrary threshold of that problem H. converging that gradually decreases to 1 to a A threshold would / L would also address the but complicate the analysis. The To modified obtain algorithm, 1 proof, induced it will Y-1, is given in Table a necessary to make 4. be in an assumption π by the base learner step 2.1 of about the supervised classifier ( t + ) the algorithm. A natural assumption is that the base learner chooses ( t + 1 ) to minimize ∑ D ( φ ( t ) We x ∈ Λ ( t ) x ||π ( t + 1 ) weaker will however. π so as x ) . A assumption if That suffice, is, we assume that the base learner reduces divergence, possible. assume ∆ D Λ ≡ ∑ D ( φ x ( t ) ||π x ( t + 1 ) ) − ∑ D ( φ x ( t ) ||π x ( t ) ) ≤ 0 ( 7 ) x ∈ Λ ( t ) x ∈ Λ ( t ) with equality only if there is no classifier π ( t + 1 ) ∈ Π that makes ∆ D Λ < 0. Note that learning algorithm that minimizes ∑ D ( φ ( t ) ||π ( t + 1 ) any x ∈ Λ ( t ) x x ) satisfies the weaker assumption (7), inasmuch the option of setting ( t + 1 ) ( t ) as π x = π x is always available. Abney Understanding the Yarowsky Algorithm Table The 4 modified generic Yarowsky algorithm (Y-1). (1) Given: X,Y ( 0 ) (2) For t ∈{ 0,1, ...} (2.1) Train classifier on (Λ ( t ) ,Y ( t ) ) ; result is π ( t + 1 ) (2.2) For each example x ∈ X:

(2.2.1) Set argmaxj ( t + 1 ) ŷ = π x  Y ( 0 ) (2.2.2) Set Y ( 1 )  x t + x = ŷ (2.3) If Y 1 Y , ⊥ ( t + ) ( t ) = stop ( j ) if x ∈ Λ ( 0 ) if ∈ Λ ( t ) ∨π ( t + 1 ) otherwise x x ( ŷ ) > 1 / L We also consider over a somewhat stronger assumption, over namely, that the base learner reduces divergence all examples, not just labeled examples: ∆ D X ≡ ∑ x ∈ X D ( φ x ( t ) ||π x ( t + 1 ) ) − ∑ x ∈ X D ( φ x ( t ) ||π x ( t ) ) ≤ 0 ( 8 ) If a more base learning algorithm satisfies (8), the proof satisfy. of theorem 1 is shorter; but (7) is the We natural condition main for a base learner now the theorem of to can state this section. Theorem If 1 iteration the base learning it algorithm satisfies of (7) or H. (8), algorithm Y-1 decreases H at each until reaches a critical point We require the following lemma in order to prove the theorem: Lemma 1 For all distributions p with equality iff p is the uniform distribution. Proof By definition, for all k:

≤ ( k ) ≥ p Since this is true for all k, it is true if we take the expectation with respect to p: log p ( k ) 1 max p ( j ) j k H ( p ) ≥ ≥ Computational Linguistics Volume 30, Number 3 We have equality only if p ( k )= max j p ( j ) for all k, that is, only if p is the uniform distribution. We now prove the theorem. Proof The of Theorem 1 algorithm 1 produces a sequence of labelings π ) π ( 1 φ ( 0 ) , , φ ( 1 ) , The ) is trained on , classifier t + φ ( t ) , ... and a sequence of clas- sifiers π ( ) ( 2 and the labeling φ ( t + 1 ) .... 1 . is created using π ( t + ) Recall that H = ∑[ x ∈ X H ( φ x )+ D ( φ x ||π x ) ] In the training (2.2), step (2.1) we of the algorithm, we hold . We φ will fixed and change π , and in the labeling minimizes step hold in as a function of π fixed π , and change φ and the labeling step minimizes show that the training step H in which it is of H. Hence, H is asa function of in φ except each iteration examples of at a the is critical point H algorithm and strictly decreasing unless ( φ t ) , nonincreasing ( π ( t ) of H. ) is a critical point Let us consider we the labeling step first. In this step, π is held constant, but φ (pos- sibly) changes, and have ∆ H = ∑ x ∈ X ∆ H ( x ) where ∆ H ( x ) ≡ H ( φ ( x t + 1 ) ||π x ( t + 1 ) ) − H ( φ x ( t ) ||π x ( t + 1 ) ) We x. can show that ∆ H is nonpositive if we can show that ∆ H ( x ) is nonpositive for all We guarantee that ∆ H ( ) ≤ 0 if φ ( t + 1 ) minimizes H ( ||π ( t + 1 ) can of p. By x p x ) viewed as a function definition: We wish to find the distribution that minimizes H ( ||π ( t + 1 ) p p x ) . Clearly, we accomplish that by placing all the mass of in , where j minimizes log ( t + 1 ) p p j ∗ ∗ − π x ( j ) . If there is more than one minimizer, H ( ||π ( t + 1 ) p x ) is minimized by any distribution p that dis- tributes all its mass the minimizers of log ( t among + 1 ) − π x ( j ) . Observe further that argmax ( t + 1 ) = π x ( j ) j = ŷ That is, we minimize H ( ||π ( t + 1 ) p x ) by setting p j = [[j = ŷ]], which is to say, by labeling predicted by ( t + 1 ) . That is how algorithm Y-1 defines φ ( t + 1 ) x as ∈ ( t + 1 ) whose π labels modifiable (that is, excluding Λ ( 0 x ) ). for all examples x Λ are x ∈ Note that φ ( t + 1 ) does not minimize H ( p ||π ( t + 1 ) ) for examples x ∈ V ( t + 1 ) x is remain unlabeled at t + 1. However, x in algorithm Y-1, , that is, for examples x that 1 is t, so any example that unlabeled at t + necessarily also unlabeled at for any such example, Abney Understanding the Yarowsky Algorithm ∆ H ( x )= 0. Hence, if any label changes in in the labeling step, H decreases, increase. and if no label changes, We H remains unchanged; either case, H does can show further even for x ∈ V ( t + 1 not that examples ) , the labeling distribution φ ( t + 1 ) assigned by Y-1 represents a critical point of H. For any example x ∈ V ( t + 1 ) x , the prediction distribution ( t + 1 ) x). Hence π x is the uniform distribution 1 (otherwise divergence t + ) ( t + 1 ) is zero, Y-1 would have labeled the between φ ( and π and thus at minimum. It would be possible to decrease H ( φ ( t + 1 ) ||π ( t + 1 ) ) by decreasing H ( φ ( t + 1 ) a x x x ) at the cost of increase in D ( φ ( t + 1 ) an x ||π ( t + 1 ) increased x ) , probability but mass) all directions of motion (all That is ways say, of selecting labels of to receive is zero; we are equally good. to the gradient Essentially, H we are at a critical point. have reached with a saddle nonzero point. We have minimized H with respect to φ x ( j we ) along those dimensions maximum, a without gradient. Along the remaining dimen- sions, of are actually at a local but a gradient to choose a direction descent. Now let us consider constant, . so the change in the H of is algorithm’s training step equal to the change in (2.1). In this step, φ is held By D—recall The that H ( φ||π )= H ( φ D )+ ( φ||π (7) ) or the (8). hypothesis If it (8), the theorem, there are minimizes two cases: base learner either immediately minimizes the base learner of D . as a function of satisfies it satisfies it π , hence follows Suppose instead that H asa function (7). We π that the base learner satisfies can express H as H = ∑ x ∈ X H ( φ x )+ x ∈ ∑ Λ ( t ) D ( φ x ||π x )+ x ∈ ∑ V ( t ) D ( φ x ||π x ) In the training But step, the first term may increase. remains constant. However, The we second term decreases, increase by hypothesis. in the is third more term offset in can show that any the Consider third term than an x in the arbitrary V labeling ( t ) . Since step. example it is unlabeled at time t, we know that φ ( t ) x is the uniform distribution u:

( j )= 1 u L [BAR] Moreover, ( t ) π x must of in also be the previous iteration. uniform Therefore distribution; otherwise have been labeled iteration a is : of example x would the value H ( x )= H ( φ x ||π x ) at the beginning t H 0 1 If unchanged in the training step, then the distribution ( t + 1 ) old π x one, remains is new in particular, and is the example remains unlabeled. On Hence π x , like the is the in uniform no H, distribution, change H nonincreasing, desired. the other there and as hand, if does change, then the distribution ( t + 1 ) π x new π x is nonuniform, and the example is Computational Linguistics Volume 30, Number 3 labeled in the labeling is step. : Hence the value of the labeling step, H 2 By Lemma we 1, H observed 2 H ( u ) ; hence H 2 , H 0 . As increase in above, H1 H is 0 but if more we consider offset in the change overall, we find that the the training step than the labeling step: ∆ H ( x )= H 2 − H 1 + H 1 − H 0 0

H ( x ) at the end of the iteration, after

log 1 1 3.1 The When Original one Decision of List Induction Yarowsky Algorithm speaks Y-0 (or Y-1), the one whose often DL-0 algorithm, has in mind includes not just the generic algorithm of but an algorithm made Yarowsky. specification Specifically, Yarowsky’s the particular choice base learning algorithm learner constructs list, that is, by list of rules of form f j, where base a decision a → f. is with fj . A rule f → The j matches example x if f is a feature and j a label, score is of x possesses the feature matches label predicted x. for a given example x the label the highest scoring Yarowsky rule that uses smoothed probability f ) is precision for rule is a version of scoring. precision smoothed (raw) As the name smoothed q̃ precision q f ( j ) , which suggests, ( j it matches is the that rule f → j correct given that { ( j ) | 1L Λ fj | | Λ f | if otherwise | Λ f | 0 q f ≡ ( 9 ) where Λ f is the set with of labeled examples that j. possess feature f, and Λ fj is the set of labeled Smoothed examples feature f j f; ) is and label precision q̃ ( | defined as follows: q̃ ( j | f; ) ≡ [BAR] | | Λ Λ f fj | | + + L ( 10 ) We also Yarowsky write q̃ f ( j ) when is clear from a score to context. defines rule’s be its smoothed precision: fj = q̃ f ( j ) Anticipating later needs, we will also consider raw precision as an alternative: fj = q f ( j ) . Both raw Generally, and smoothed f. we precision have the properties of a conditional view fj as a distribution over probability distribution. conditional labels j for a fixed feature Yarowsky defines matches the confidence instance of the decision list This to be is the score of the highest- scoring rule that the being classified. equivalent to defining x ( j ) ∝ max f ∈ F x fj ( 11 ) (Recall that in F x is the (1), set of features of x.) maximizes Since the classifier’s , prediction (11) for implies x is defined, equation to be the label that x ( j ) definition Abney Understanding the Yarowsky Algorithm Table The 5 decision list induction in Z algorithm is . DL-0. The value accumulated in N [ f,j ] is | Λ fj | , and the value accumulated [ f ] | Λ f | (0) Given: Initialize a fixed value N f,j for [ 0, 0 (1) For arrays ]= Z [ f ]= 0 for all f, j (1.1) each Let example x ∈ (2) For Increment be the label of Λ (1.2) j N f,j ,Z x [ ] [ f ] , for each feature f ofx each feature f and label j (2.1) Set N [ f,j (*) fj = [BAR] Z ]+ Define max [ f ]+ L x ( j ) ∝ f ∈ F x fj that the classifier’s prediction is the label of the highest-scoring rule matching desired. We have written will ∝ in (11) rather probability than = in because maximizing over θ fj across f ∈ F x for each label will j not general yield a distribution be bounded, and hence normalizable. Considering labels—though only the scores positive and inasmuch given example x, the normalization will the final predicted label ŷ for a will in way. have no effect, as all scores θ As Yarowsky, fj being compared be scaled by a list only the same characterized decision contains those rules f → measure. score f j ) exceeds the labeling threshold ζ . This j whose q̃ ( Including whose can be seen purely as falls the labeling threshold will an efficiency on rules score below the predictions, threshold will applied when have no effect is classifier’s to For as the this we be do the list. That is, we the classifier applied examples. decision list as set of reason, parameters {θ } , one not prune represent a a fj for every possible rule f → product of of of j in the cross The the set induction features and the set labels. we decision it list DL-0. Note algorithm used by Yarowsky (*) is is summarized in Table as labeled not actually a step of 5; induction refer to that the step rather how decision list is the algorithm but specifies the x. used to compute a prediction Unfortunately, distribution we π x for we prove a given example cannot DL-0 anything about DL-0 as it stands. In particular, are unable (7). to show In that reduces DL-EM, next section, we divergence between prediction and labeling distributions the describe (7); an we alternative decision Y-1/DL-EM satisfy it hence H. However, can apply Theorem list 1 induc- tion algorithm, that does to to that reduces disadvantage of the combination EM is it show 3.4 not resemble of DL-0 the algorithm DL-1 DL-0 used by Yarowsky. a We it return in DL- that does section to a close H, it variant called and K. show that though does not directly reduce does reduce the upper bound x, as 3.2 The The Decision DL-EM List Induction is Algorithm of DL-EM of algorithm DL-EM- a special DL-EM-X. case the They EM algorithm. in We DL-EM- consider two is versions the algorithm: only, whereas Λ and DL-EM-X is differ on that Λ trained on labeled examples However, outline of trained DL-EM basic the algorithm is both labeled and unlabeled examples. (11). First, the do not assume Yarowsky’s the same for both. the algorithms over the parameters θ fj can be thought of definition of π , given in As discussed above, f. Hence as (11) defining a prediction distribution θ f ( j ) labels j for each feature of equation to specifies how the prediction distributions θ f for the features example x are be combined to yield a Computational Linguistics Volume 30, Number 3 prediction across distribution in π x for x. (11), Instead DL-EM of combining distributions mixture of by : maximizing θ F fj f ∈ x as equation takes a the θ f Here m = | F x | is the number of features that x possesses; of for the sake assume have number Since of simplicity, examples the features. θ is we that all f, same of f a probability distribution for it each and since any convex in (12) combination is probability distributions is also a distribution, The follows that π x as two only in for x ( j ) , defined (11) and (12), will a is π often distribution. definitions have of the same mode ŷ, but that guaranteed Under (11), the rather severely is restricted case two features whereas (12) the prediction determined entirely by the strongest and two labels. θ Yarowsky f , definition definition wished permits a bloc of weaker of θ f to outvote interactions. the strongest one. (12), explicitly DL-EM, to avoid the out possibility make such of other Nonetheless, definition manageable, used we by DL-1 will turns YS assume it to analysis in henceforth, not only for DL-EM, base learners more and but also for the algorithms DL-EM and from discussed DL-0 improve in subsequent on DL-EM sections. also differs that does classifier. not construct In a classifier Yarowsky to a previous the context of “from scratch” but rather seeks outer previous classifier is the one from the previous iteration the algorithm, the We write the old fj for parameters and π old of loop. θ the x for the prediction distributions of the Conceptually, previous classifier. DL-EM considers the label j assigned to an example x to be gen- erated by choosing a feature f ∈ F x . and The then assigning of the label is j according to which ( j ) choice feature f a hidden variable. The the feature’s prediction distribution θ f degree to an example labeled j is imputed to feature f is determined by the old distribution: [BAR] π old x ( j ) One can think of π old ( j, f | x,j or ) either as the of posterior probability that x,j feature We write as the portion the labeled example ( ) that is f imputed was re- sponsible for to feature f. the label also π old xj ( f ) as a synonym for π old ( f | x,j ) . The new estimate θ fj is obtained DL-EM- by , summing imputed occurrences of ( f,j ) and normalizing across labels. For Λ this takes the form [[f ∈ F ]] θ old old x,j x ( f | )= fj π [BAR] [[g F old = ∑ g ∈ x ]] θ gj ∑ ∈ Λ π old x ( f | x,j ) θ j fj = [BAR] ∑∑ x Λ π old k ∈ ( f | x,k ) k The The algorithm is summarized second version of in Table 6. DL-EM- , it the algorithm, DL-EM-X, is summarized in Table 7. It is like Λ except that uses the update rule [BAR] ( 13 ) Update tually, it rule (13) includes unlabeled examples as well as labeled examples. Concep- divides each unlabeled example equally among the labels, then divides the resulting fractional labeled example among the example’s features. Abney Understanding the Yarowsky Algorithm Table 6 DL-EM- Λ decision list induction algorithm. Table DL-EM-X 7 decision list induction algorithm. of We EM-like note that both variants of the iteration DL-EM algorithm prove constitute a single iteration an multiple algorithm. iterations A would single suffices to the following theorem, though also be effective: Theorem 2 The classifier produced produced DL-EM-X by the DL-EM- Λ algorithm satisfies (8). equation (7), and the clas- sifier by the algorithm satisfies equation Combining Theorems 1 and 2 yields the following corollary: Corollary The Yarowsky algorithm Y-1, iteration using DL-EM- it Λ or DL-EM-X as its of base H. learning algo- rithm, decreases H at each until reaches a critical point Proof Let θ old of Theorem 2 represent the of parameter values at we the will beginning optimize, of the call to represent free variables that and let old DL-EM, let θ a family The π is and π set of distributions. labeling distribution φ fixed. For be the corresponding prediction examples obviously , let ∆ D α be the change in ∑ x ∈α D ( φ x ||π x ) resulting from any in . We α are interested in two cases: that in which the is change of θ (for particularly DL-EM-X) in which is of α the set all examples X and that α the set labeled examples Computational Linguistics Volume 30, Number 3 Λ (for DL-EM- of Λ ). In either D. case, we will show that ∆ D α ≤ 0, with equality only if no choice We θ decreases first derive an expression for − ∆ D α that we will put to use shortly: − ∆ D α = = = The EM if algorithm is based the fact that identical: divergence non-negative, and strictly positive the distributions compared are not ∑ [ D ( φ x ||π x old ) − D ( φ x ||π x ) x ∈α ∑ [ H ( φ x ||π x old ) − H ( φ x ) − H ( φ x ||π x )+ H ( φ x ) ] x ∈α ∑∑ φ ( j ) π ( ) log old ] x log j − x π x ( j ) x ∈α j ] [ (14) on is j x ∈α = ∑∑ old π old ∑ xj ( f ) φ ( j ) π xj ( f ) log [BAR] x j x ∈α f ∈ F x ∑∑ φ ( j ) ∑ π old x xj ( f ) log j x ∈α f ∈ F x which yields the inequality = [ ∑∑ φ x ( j ) π x ( j ) − π old ] log log x ( j ) ≥ ∑∑ φ x ( j ) π xj ( f ) log θ− fj log θ fj j x ∈α j x ∈α f ∈ F x By (14), this can be written as [ − ∆ D ≥ ∑ ∑ φ x ( j ) ∑ π old α xj ( f ) log θ− fj log θ old ] fj ( 15 ) j x ∈α f ∈ F x Since θ old fj is constant, by maximizing ∑∑ φ x ( j ) ∑ π old xj ( f ) log θ fj ( 16 ) j x ∈α f ∈ F x we zero: maximize we a lower bound on − old . Since ∆ D . It is easy to is zero see that − α only if ∆ D α is bounded above by simply set θ fj = θ fj divergence the two distributions are identical, we have strict inequality in (15) unless the best choice for θ is θ old which no of θ makes ∆ D < 0. , in We wish It case choice maximize to show (16) that DL-EM α remains computes the parameter set θ that maximizes (16). across to of j, under so we the constraints Lagrange’s that method. the values We {θ fj } for fixed f sum to unity in choices apply express the constraints the form C f = 0 where C f ≡ ∑ j θ− fj 1 π xj ( f ) ( θ old fj π x ( j )

[BAR] π old ( j ) · [BAR] x θ fj ) old [ ∑ old ] Abney Understanding the Yarowsky Algorithm We of seek (16) a solution to the family of of equations that results of from expressing the gradient as a linear combination the gradients the constraints: We derive an expression for the derivative on the left-hand side: Similarly for the right-hand side: ∂θ fj Substituting these into equation (17): x ∈ ∑ X f ∩α φ x ( j ) π xj old ( f ) [BAR] θ 1 fj = λ f Using the constraint C f = 0 and solving for λ f : ∑∑ j x ∈ X f ∩α φ x ( j ) π xj old ( f ) [BAR] λ 1 f − 1 = 0 λ f = ∑ j x ∈ ∑ X f ∩α φ x ( j ) π xj old ( f ) Substituting back into (18): θ fj = [BAR] ∑∑ ∑ k x ∈ x X ∈ f X ∩α f ∩α φ x φ ( x j ) ( π k ) xj old π xk old ( f ) ( f ) we obtain If we consider the case where α is the set of all examples and expand φ x where Z normalizes intermediate θ f . It is not hard to see that this is the update rule that DL-EM-X computes, using the values: N [ f,j ] ∑ π old xj ( f )

x ∈ Λ fj U [ f,j ] ∑ π old xj ( f ) x ∈ V f = = ( 17 ) ( 18 ) ( 19 ) in (19), Computational Linguistics Volume 30, Number 3 If we we consider obtain the case where α is the set of labeled examples and expand φ x in (19), θ 1 fj Z ∑ π old = [BAR] xj ( f )

x ∈ Λ fj This , is the update DL-EM- rule that DL-EM- . Λ computes. Thus we see that DL-EM-X reduces D X and Λ reduces D Λ We inasmuch note in as it is closing that DL-EM-X = f,j 1 can be simplified L for all ( f,j ) , when that fj where used with algorithm Y-1, known θ / f ∈ F x for some x ∈ V. Then the expression for U [ ] simplifies as follows: ∑ π old ( f ) x ∈ V f = = The DL-EM-X, dependence on j disappears, so can increment delete step 1 m.” 2.1, and replace step 2.2 with the statement “For each f ∈ F x , U [ f ] by /

Y-1/DL-EM is the only variation H. on The the Yarowsky algorithm we that in we can show to reduce negative Y-1/DL-1 log-likelihood, YS, variants that objective discuss the remainder K, which of the article, and reduce an alternative function, we now we The define. value K Jensen’s (or, more inequality, precisely, the value K / m) is an upper bound on H, which derive using as follows: H = ≤ = We define By minimizing K, we minimize if to reduce to zero. Since an upper possible K H ( φ x ||θ g )= H ( φ x )+ its D ( φ x ||θ g ) ,K is reduced all labeled, each feature prediction distribution in to zero examples are single In of concentrates example minimizer agrees of with the is prediction minimizer of a it label, and the label every every of feature possesses. this limiting case, any K also a H. xj x ∑ ∈ V f [ [BAR] ∑ g ∈ 1 / F x L 1 / L ] | V f | [BAR] m we replace U [ f,j ] with U [ f ] in algorithm − ∑ x ∈ X ∑ j φ xj log g ∑ ∈ F x [BAR] m 1 θ gj − ∑ x ∈ X ∑ j φ xj g ∑ ∈ F x [BAR] m 1 log θ gj K ≡ ∑∑ H ( φ x ||θ g ) x ∈ Xg ∈ F x ( 20 ) bound on H. Moreover, it is in principle Abney Understanding the Yarowsky Algorithm Table The 8 decision list induction algorithm DL-1-R. Table The 9 decision list induction algorithm DL-1-VS. (0) (1) Initialize For N [ f,j ]= 0, Z [ f ]= 0, x,j U [ f ]= 0 for all f, j (2) For (1.1) each For example-label pair each f ∈ F x , ( feature increment ) N [ f,j ] ,Z [ f ] (2.1) each (3) For For unlabeled example each feature f ∈ F x , x increment U [ f ] (3.1) each Set feature U f L and label j = [ f ] The We hasten provides to add a proviso: It is not possible to reduce K to zero for all data sets. Consider following a necessary and sufficient is undirected bipartite graph G whose condition for being able to do so. There an nodes are Define between example and f just in examples x feature case f is and features. an edge G. x x 2 to be neighbors if a feature of x. examples of 1 and , is reducible to zero if they and only both if belong to the same connected component 0 K of in x 1 and 0 . x 2 have the same label according to Y ( ) for all pairs neighbors x 1 and x Λ ( ) 2 3.4 We Algorithm DL-1 in consider ways. two First, variants DL-1 of DL-0, called DL-1-R and DL-1-VS. They differ (12) algorithms assume the “mean” definition of from DL-0 two the in induction than the “max” definition itself, of in equation way (11). This is x given in equation rather not is actually a difference the algorithm . but the the decision list used to construct Second, a prediction of DL-0. DL-1 distribution x the DL-1-R algorithms (Table 8) use update rules that instead differ from of the smoothed pre- cision DL-1-VS (Table 9) uses raw precision DL-0, DL-1-VS smoothed precision. uses smoothed precision, but from unlike a smoothing constant rather varies feature to does feature. Specifically, not use fixed , DL-1-VS ; of uses | V f | L as its value for . in computing The the score fj Let value used by DL-1-VS can be expressed in another way that will prove useful. us define p (Λ | f ) ≡ [BAR] | | X Λ f f | | p ( V | f ) ≡ [BAR] | | X V f f | | Computational Linguistics Volume 30, Number 3 Lemma The 2 parameter values { fj } computed by DL-1-VS can be expressed as

fj = p (Λ | f ) q f ( j )+ p ( V | f ) u ( j ) where u ( j ) is the uniform distribution over labels. ( 21 ) Proof If | Λ f | = 0, then p (Λ , | f )= 0 and is fj = proved. u ( j ) . Further, Hence N we [ f,j ]= Z [ only f ]= 0, so DL-1-VS computes 0. fj = u ( j ) and the lemma need consider the case | Λ f | of First we show (9) that smoothed precision can be Define expressed as a convex raw and the uniform distribution. Λ . Then: combination precision = q̃ f ( j ) = = =

= Now we show that of the mixing coefficient when ( + ) the same as the coefficient p (Λ | f ) the lemma, = = = L = 1 The Y-1/DL-1 main theorem of this section in (Theorem iteration 5) is that it the specific Yarowsky corollary each until reaches a critical point. It al- gorithm proved decreases imizes of K two theorems. The first (Theorem 3) shows that DL-1 min- is as a Y-1 K as a function of , holding of constant, , and the second More (Theorem precisely, 4) minimizes asa function holding constant. DL-1-R shows that decreases X. Either is over K K labeled Y-1 examples to Λ , and DL-1-VS minimizes K over all examples sufficient for be effective. Theorem 3 DL-1 over minimizes K as a function , of DL-1-VS , holding minimizes constant. Specifically, over DL-1-R X. minimizes K labeled examples Λ and K all examples Proof We wish to minimize Kasa function of under the constraints C f ≡ ∑ j fj − 1 = 0 | f | | Λ fj | + [BAR] | Λ f | + L | Λ fj | | Λ f | + [BAR] 1 1 L of (22) is mixing 1 | V f | L as in step 3.1 of DL-1-VS: | V f | [BAR] L p (Λ | f ) p (Λ | f ) Abney Understanding the Yarowsky Algorithm for each f. of As before, to minimize K under the constraints C f = a combination of the gradients of 0, we express the gradient K as of linear the constraints and solve the resulting system equations: ∂θ fj ∂θ fj ( 23 ) First we derive of expressions over for which the we derivatives of minimizing C f and K: K. The variable α represents the set examples are ∂θ fj = − x ∈ ∑ X f ∩α φ xj [BAR] θ 1 fj We substitute these expressions into (23) and solve for θ fj : θ fj = − x ∈ ∑ X f ∩α φ xj /λ f Substituting the latter expression into the equation for C f = 0 and solving for f yields   ∑ j  − x ∈ ∑ X f ∩α φ xj /λ f  = 1 −| X f ∩α| = λ f Substituting this back into the expression for θ fj gives us θ fj = [BAR] | X 1 ∑ φ xj ( 24 ) If α =Λ , we have θ fj =

= This is the update minimize computed by over DL-1-R, showing that . computes the parameter values {θ fj } that K the labeled examples Λ f ∩α| x ∈ X f ∩α q f ( j ) DL-1-R Computational Linguistics Volume 30, Number 3 If α = X, we have θ fj = = = By Lemma over 2, this is the of update computed X. by hence the complete set examples [BAR] | | X Λ f f | | · [BAR] | | Λ Λ fj f | | + [BAR] | | X V f f | | · [BAR] L 1 p (Λ | f ) · q f ( j )+ p ( V | f ) · u ( j ) DL-1-VS, DL-1-VS minimizes K Theorem 4 If the base learner decreases K over X or over Λ , where the prediction distribution is computed as π x ( j )= [BAR] m 1 f ∑ ∈ F x θ fj then algorithm Y-1 decreases of K with at each iteration until it reaches a critical point, con- sidering K as a function φ θ held constant. Proof The proof We minimize has the same structure as of the proof of Theorem 1, so we give only a sketch here. K as a function K ( x ) = = To minimize K ( x ) , we choose φ xj so as to concentrate This increase If is the labeling rule in on minimizes used by Y-1. the base learner over is Λ only, rather than Theorem unlabeled examples for in X, it can be shown that compensated the labeling step, as in any proof of K 1. the Theorem iteration The 5 specific Yarowsky algorithms Y-1/DL-1-R and Y-1/DL-1-VS decrease K at each until they reach a critical point. Proof Immediate from Theorems 3 and 4.

The in Yarowsky algorithm variants we have considered up to now do “parallel” the the parameters {θ } iteration. updates sense that fj are completely recomputed at each In φ by minimizing it for each example separately: g ∑ ∈ F x H ( φ x ||θ g ) θ gj all mass in Abney Understanding the Yarowsky Algorithm this section, in we consider which a family YS is of “sequential” variants of the Yarowsky a feature selected for update at each iteration. The al- gorithm, single “Yarowsky-Cautious” of Collins & Singer YS algorithms resemble the from in algorithm in (1999), though they differ that algorithm of that they ation, than small features, in update a single Yarowsky-Cautious. feature The YS each iter- rather intended a set be The YS the Y-1/DL-1 as are close to algorithm from is algorithms to as one another, as consonant from with single- feature updates. In of algorithms differ particular, An interesting range of and update rules work in Y-1/DL-1, in the choice update rule. with , in the sequential setting. Y-0/DL-0, works in smoothed precision fixed with proviso as the original will algorithm out later. the sequential setting, though a that be spelled Instead of an initial labeled set, there is an initial classifier consisting of a set of selected features S 0 and initial parameter set θ ( 0 ) with θ ( 0 ) fj = 1 / L for all f S 0 . At each iteration, one in feature is selected It to is be added to the selected set. to A feature, once selected, once; remains the selected set. permissible for a feature be selected more than In this permits is us to continue of reducing K even after ,f , all , features have been selected. short, there a sequence selected features f 0 1 ... and S t + 1 = S t f t The parameters , with for the , may selected modified, feature are also other updated. At iteration t, the pa- rameters θ gj g = That is: f t be but all parameters remain constant. θ ( t + 1 ) θ ( t ) gj = gj It follows that, for all t: for g = f t However, role of parameters manually for features in the In each iteration, labeled one data. selects a feature f t and tion θ for the selected feature f . Then computes (or recomputes) the predic- Recall distribution (equation argmax f t x ( j ) , where The we t labels are recomputed as follows. that ŷ j π (12)). of continue to assume is π x ( j ) if to have the “mix- ture” definition (though S t + 1 . In particular, label example x set to ŷ any feature of x belongs to may all previously labeled examples continue to be labeled their labels change), and any unlabeled examples possessing feature become labeled. The algorithm is summarized in Table 10. It We is actually an algorithm schema; the definition one for “update” needs to be supplied. its consider three different one update functions: that uses one raw precision in as opposite prediction distribution, what we that might uses smoothed precision, and we that goes the direction, using call mixture “peaked of precision.” As have seen, smoothed (i.e., maximum-entropy) precision can be expressed raw and the uniform distribution (22). as a Peaked precision precision q̂ ( f ) mixes its mass in a on certain amount of maximizes the point (i.e., minimum-entropy) distribution that has all the label that raw precision: S t S 0 may not be modified, inasmuch as they play f t q̂ f ( j ) p (Λ ( t ) f ) q f ( j )+ p ( V ( t ) f ) [[j = j ]] ( 25 ) where j argmax q f ( j ) j ( 26 ) Computational Linguistics Volume 30, Number 3 Table The 10 sequential algorithm YS. (0) Given: S ( 0 ) , θ ( 0 ) , with θ ( 0 ) 1 / L for g S ( 0 ) gj = (1) Initialization (1.1) Set S S ( 0 ) , θ θ ( 0 ) = = (1.2) For each example x X If x possesses a feature in S ( 0 ) , set Y x = ŷ, else set Y x = (2) Loop:

(2.1) Choose a feature f S ( 0 ) such that Λ= f and θ f = If is q f (2.2) there (2.3) For f to S none, stop Add each label j, set θ fj = (2.4) For update ( f,j ) each example x possessing a feature in S, set Y x = ŷ Note that peaked on precision involves proportions a variable of amount of “peaking”; the mixing rameters the relative labeled and unlabeled examples. Note pa- depend also The that j † is instantiations a function of f, of though we YS do not we explicitly represent that dependence. three algorithm that consider are YS-P YS-R (“peaked”) YS-FS (“raw”) (“fixed smoothing”) θ fj = q̂ f ( j ) θ fj = q f ( j ) θ fj = q̃ f ( j ) We will show that the first YS-FS, two algorithms reduce in iterations K in in each which iteration. is We will show that the third algorithm, Unfortunately, reduces we K YS-FS f t a new feature, not when previously is selected. This are unable to show that which t a previously selected is feature. suggests employing a mixed reduces K in f or algorithm smoothed precision used for new features but raw peaked precision is used for previously in A issue with selected features. final The the algorithm schema YS concerns step schema stated does specify which the selection feature is to of features In 2.1. manner in as which not essence, selected does not matter, be selected. the room improvement, rules are in as long as one selects rules that have f from for raw precision q . (The the sense that the current prediction distribution in Theorem 9.) θ The differs in f justification for this choice is iteration, sections show that K decreases in given theorems the following each One so long as any such rule can be found. (27)), could choose in greedily by choosing we the feature that maximizes gain G (equa- tion though (Theorems the next 6 section 7). give lower bounds for G that are rather more easily computed and 4.2 Gain From this t. point We write on, we old consider old a single iteration of the YS algorithm and discard the variable iteration, we θ write and φ for the parameter set and labeling at the beginning of ing. the The and (respectively, simply V) θ and φ for the new parameter set and Λ examples that labeled (respectively, new label- set of represents iteration. the The are unlabeled) in We wish at the beginning the selected feature is f. iteration. to choose The a prediction in distribution iteration for is f so as to guarantee that K decreases each gain the current G = ∑∑ x ∈ Xg ∈ F x [ H ( φ old x ||θ g old ) − H ( φ x ||θ g ) ] ( 27 ) Abney Understanding the Yarowsky Algorithm Gain is In the negative change in K; in it is positive from when considering the reduction φ old , old K decreases. intermediate K ( θ ) to ( φ , θ ) , it will be convenient to consider the following values: K 0 K 1 K 2 K 3 = = = = { = j ∗ ]] x ψ old otherwise ∈ xj = φ xj j ∗≡ argmax θ fj j ( 28 ) old ∑ ∑ H ( φ old x ||θ old g ) x ∈ Xg ∈ F x ∑ x ∈ Xg ∑ ∈ F x H ( ψ x ||θ g old ) ∑∑ H ( ψ x ||θ g ) x ∈ Xg ∈ F x ∑ x ∈ Xg ∑ ∈ F x H ( φ x ||θ g ) [[j if Vf V + θ + Λ = = = K 0 − K 1 K 1 − K 2 K 2 − K 3

is where and One should note that • θ f is the new prediction distribution for the candidate f; θ gj = θ gj for g = f. • φ is the new label distribution, after relabeling. It is defined as for x ∈ V f , It the only selected feature at t + 1 is f, hence j ∗ = on in V . ŷ They for such examples. on follows that ψ and φ agree examples are at t + 1, f also agree examples that If unlabeled differ, it assigning is only on them old the uniform label ( ) distribution. ψ and φ need given the addition of labeled examples Λ that to be relabeled, f. φ xj = ( 29 ) • ing The gain intermediate G can be represented as the sum of three intermediate gains, correspond- to the values just defined: G = G G G ( 30 ) where G V G θ G Λ The gain G V intuitively in represents with the gain that attributable of .Thegain to labeling previously unlabeled is examples accordance the predictions , where θ is G θ represents attributable the f the feature. The the gain that to changing values θ fj selected Computational Linguistics Volume 30, Number 3 gain G Λ represents the gain make that is attributable with to changing the labeled the predictions of labels of previously examples to labels agree of YS, in which the is new model θ is . The gain G θ corresponds to step 2.3 algorithm θ changed and is combined G V and G Λ gains correspond to step 2.4 of but φ held constant; in which the while algorithm YS, In φ changed we of holding θ the remainder this section, we constant. derive YS-P, two YS-R, lower bounds YS-FS for G. In following sections, show that the updates and is guarantee that the lower bounds given below are non-negative, and hence that G non-negative. Lemma 3 G V = 0 Proof We of show that we K remains is unchanged it if we with substitute old on ψ for φ old in K 0 . The only property ψ Since that need ψ x φ old that agrees x ∈ Λ , we φ = x need only previously labeled examples. for of iteration, consider examples are at the beginning the none of in V. Since these examples unlabeled old 1 g. Hence their features have been selected, hence θ gj = / L for all their features K 1 = = = = (Note that ψ xj is not in general equal to φ xj but ∑ j ψ xj and ∑ j φ xj both equal shows that K 0 = K 1 , and hence that G V = 0. Lemma 4 G Λ ≥ 0 . We must show that relabeling old labeled examples—that is, setting φ x ( j )= [[j = x Λ —does not increase K. The proof has the same structure the proof of ŷ]] for Theorem ∈ 1 is omitted. as and Lemma 5 G θ is equal to Proof By definition, G θ = K 1 − K 2 ,and K 1 and K 2 are identical everywhere except on examples − ∑ ψ xj log θ old gj x ∈ V ∑∑ g ∈ F x j   − ∑∑∑ φ old log θ old xj gj = K 0 x ∈ V g ∈ F x j old , old 1.) This Abney Understanding the Yarowsky Algorithm in X f . Hence G θ = x ∑∑ ∈ X f g ∈ F x [ H ( ψ x ||θ g old ) − H ( ψ x ||θ g ) ] We divide this sum into three partial sums: G θ A B C We consider each partial sum = = = = A + B + C ∑ [ H ( ψ x ||θ f old ) − H ( ψ x ||θ f ) ] x ∈ Λ f (32) x ∑ ∈ Vf [ H ( ψ x ||θ f old ) − H ( ψ x ||θ f ) ] x ∑∑ ∈ X f g = f ∈ F x [ H ( ψ x ||θ g old ) − H ( ψ x ||θ g ) ] separately: A B The justification old for the fk 1 last step / all k, a and the substitution is a new then L for valid. On the other not selected, θ = hand, if f is a previously selected feature, then | V f | = 0, and even though the substitution of 389 = = = = = = = = = = = ∑ [ H ( ψ x ||θ f old ) − H ( ψ x ||θ f ) x ∈ Λ f − ∑∑ ψ xk [ log θ fk old − log θ fk ] x ∈ Λ f k [[x [ ∑∑ ∈ Λ ]] log θ old k fk − log θ fk x ∈ Λ f k − ∑ | Λ fk | [ log θ fk old − log θ fk ] k

k | Λ f | [ H ( q f ||θ f old ) − H ( q f ||θ f ) ] ] ] − [ −| Λ | ∑ q ( k ) log θ old ] f f fk − log θ fk (33) x ∑ ∈ V f [ H ( ψ x ||θ f old ) − H ( ψ x ||θ f ) [ − ∑∑ ψ xk log θ old fk − log θ fk x ∈ V f k [ − ∑∑ [[k = j ∗ ]] log θ old fk − log θ fk x ∈ V f k ] ] ] [

θ fj ∗ is bit subtle. If (34) f is feature, previously Computational Linguistics Volume 30, Number 3 1 / L for θ old fj may not be valid, it is innocuous. C = [ H ( ψ x ||θ g old ) − H ( ψ x ||θ g ) ] ∑∑ x X f g = f F x = x ∑∑ X f g = f F x [ H ( ψ x ||θ g old ) − H ( ψ x ||θ g old ) ] = 0 (35) Combining (32), (33), (34), and (35) yields the lemma. Theorem 6 G is bounded below by (31). Proof Combining (30) with Lemmas 3, 4, and 5. Theorem is 7 G bounded below by | Λ f | [ H ( q f ||θ f old ) − H ( q f ||θ f ) ] Proof The theorem follows immediately from Theorem 6 if we can show that Observe By first Lemma that log 1, we L = H ( u ) . (Recall that u ( j )= 1 / L is the uniform distribution over labels.) know that ≥ 0 θ fj The latter follows because the uniform distribution maximizes entropy. Theorem is 8 G bounded below by | Λ f | [ D ( q f ||θ f old ) − D ( q f ||θ f ) ] Proof Immediate from Theorem 7 and the fact that H ( q f ||θ old f ) − H ( q f ||θ f ) = H ( q f )+ D ( q f ||θ old f ) − H ( q f ) − D ( q f ||θ f ) = D ( q f ||θ f old ) − D ( q f ||θ f ) Theorem 9 If θ old f = q f , then there is a choice of θ f that yields strictly positive gain. Abney Understanding the Yarowsky Algorithm Proof If θ old f = q f , then Setting θ f = q f has the result that D ( q ||θ old f f ) > 0 | Λ f | [ D ( q f ||θ f old ) − D ( q f ||θ f ) ] = | Λ f | D ( q f ||θ f old ) > 0 Hence G > 0 by Theorem 8. 4.3 We Algorithm YS-P now in use the results of it the previous in section iteration. to show that the algorithm YS-P is correct the sense that reduces K every Theorem 10 In each iteration of algorithm YS-P, K decreases. Proof We By Theorem wish to show 9, that G > 0. By Theorem 6, that make is true we exist choices for θ f that (31) if expression (31) positive, hence in is particular, positive. there guarantee G > 0 by maximizing (31). We maximize (31) by minimizing Since H ( q f ||θ f )= H ( q f )+ D ( q f ||θ f ) we minimize (36) by minimizing Both terms are nonnegative. The first term its is mass zero in if θ f = q f . The second it is term is zero for any distribution that concentrates all a single label j ∗ in (37) of decreases monotonically mode of as θ fj ∗ approaches one. ; all and Hence, symmetric choices j ∗ minimum of will j the , though it may be more the , at of have ∗ f cost an increase equal in to q f the first term, but offset by a decrease in peaked than q the the second Recall term. that j † = argmax minimum j q f ( j ) of . By (37). the Hence reasoning we of minimize the previous (37) paragraph, minimizing we know that j † = j ∗ at the can by | Λ f | D ( q f ||θ f ) −| V f | ∑ [[k = j † ]]log θ fk k We compute the gradient: [ ] [ ] ( 38 ) Computational Linguistics Volume 30, Number 3 As before, the derivative of the constraint C f = 0 is one, and we minimize (38) under the constraint by solving −| Λ f | q f ( j ) [BAR] θ 1 fj −| V f | [[j = j † ]] [BAR] θ 1 fj = λ θ fj = ( −| Λ f | q f ( j ) −| V f | [[j = j † ]] ) /λ ( 40 ) Substituting into the constraint gives us ∑( −| Λ f | q f ( j ) −| V f | [[j = j † ]] ) /λ = 1 −| Λ f |−| V f | = λ

−| X f | = λ j Substituting this back into (40) yields: θ fj = p (Λ | f ) q f ( j )+ p ( V | f ) [[j = j † ]] That is, the maximizing solution is peaked precision, which is the update rule for YS-P. 4.4 We Algorithm YS-R now show proven. that YS-R also decreases K in each iteration. In fact, it has essentially already been ( 41 ) Theorem 11 Algorithm YS-R decreases K in each iteration. Proof In the proof of Theorem 9, we showed that the choice yields strictly positive gain. This is the update rule used by YS-R. 4.5 The Algorithm original Yarowsky YS-FS We algorithm YS-0/DL-0 used smoothed precision update have unsuccessful justifying this choice of with fixed as in rule. However, we been at are at to show it update least that does decrease K when rule general. is able the selected feature a new feature, not previously selected. 392 θ f = q f Abney Understanding the Yarowsky Algorithm Theorem 12 Algorithm YS-FS has positive gain in each iteration in which the selected feature has not been previously selected. Proof By Theorem 7, gain is positive if H ( q f || old f ) H ( q f || f ) ( 42 ) By the assumption that the selected feature f has not been previously selected, old is the uniform distribution u, and the left-hand side of (42) is equal to H ( q f || u ) . It verify . p || u )= H ( u ) for any distribution p; left-hand side of is to ( hence the (42) easy that H Further, YS-FS , so is equal to H ( u ) uses smoothed precision as update rule, f = q̃ f (42) can be rewritten as

H ( u ) H ( q f || q̃ f ) This condition But does we not hold trivially, it inasmuch in as cross entropy, like divergence, is unbounded. We can show that holds : this particular case. derive an upper bound for H ( q q̃ ) Observe that iff iff We know that old u. Because inequality ( u ) ≥ of is ( q f know that the strict by on the following in reasoning. f restriction step 2.1 algorithm YS, Since old f is a = q f , new feature, = the f hence q f = u, and Hence H ( u ) is (44) strictly is greater than H ( q f ) proving and combining (44) . true, with (43), we have shown (42) to be true, the theorem. H ( q f || q̃ f ) = = ≤ = = H H f || f − ∑ q f ( j ) log q̃ f ( j ) j [ 1 [BAR] log q f ( j )+ L 1 L [BAR] 1 L log u ( j ) + + ] − ∑ q f ( j ) j H ( u ) H ( q f ) ) because the uniform distribution (43) ( 44 ) maximizes entropy. We Computational Linguistics Volume 30, Number 3

At the beginning of the Yarowsky article, the co-training There is in algorithm was mentioned as an alterna- tive to the Yarowsky algorithm. In original fact a connection between (Blum co-training Mitchell and the was algorithm. the co-training paper and maximize 1998), it suggested that the algorithm be understood on as seeking to Littman, McAllester two different “views” proven of agreement on unlabeled data work between (Dasgupta, classifiers trained the data. Subse- quent error and cross-view 2001) has on a direct connection between In classifier and such context, is agreement unlabeled data. the current However, Yarowsky there also justification for pursuing agreement on independent algorithm does not assume the existence of unla- beled data. the of Rather, is motivation two conditionally on views the data. there Recall original data objective between arbitrary pairs entropy function, H, of a for seeking agreement unlabeled our features. that can be expressed as the sum of an term and a divergence term: H = ∑[ x ∈ X H ( φ x )+ D ( φ x ||π x ) ] As D ( φ x ||π x ) we becomes small and H ( π x ) becomes small, H ( φ x ) necessarily hence can H by limiting H ( ) and D ( φ ||π ) . Intuitively, also we becomes small; limit of model’s π x while x improving x wish to reduce model’s the uncertainty the predictions, also the fit between the Let predictions on and the known labels. us focus now the uncertainty of the model’s predictions: H ( π x ) = = ≤ = = = = = In other words, by decreasing the uncertainty increasing the prediction distributions vidual features and simultaneously we the agreement among features their divergence), bound on (that is, decreasing pairwise decrease an upper H ( π x ) . This − ∑ π x ( j ) log π x ( j ) j       ] (45) of of indiAbney Understanding the Yarowsky Algorithm motivates interfeature agreement without recourse to an assumption of independent views.

In this article, we we have presented optimize a number of variants of objective the Yarowsky We algorithm, and have shown modified that they objective Yarowsky natural (which is algorithm Y-1 functions. and showed maximizing that it considered first the generic minimizes function equivalent likelihood), provided the H H. to that its base We learner reduces Y-1/DL-EM then considered (Y-1/DL-EM- three families of specific Yarowsky-like DL-EM Λ Y-1/DL-EM-X) minimize algorithms. The algorithms and much base learner Yarowsky’s has no similarity original to Yarowsky’s H learner. the approximation original but have the disadvantage that is provided base DL-1, A better Y-1/DL-1 to (Y-1/DL-1-R Y-1/DL-1-VS) base learner by minimize and the objective algorithms K, and H. Finally, were the bound for YS shown to (YS-P, YS-R, YS-FS) function an upper of Yarowsky-Cautious the algorithms of and Collins are Singer; sequential we variants, reminiscent To showed that they minimize the algorithm and that algorithms the of K. the extent provide these capture of Yarowsky’s essence approach. the original Even Yarowsky algorithm, they a formal much understanding from original on its workings, if they are deemed to diverge too least of the to cast light bootstrapping algorithms with solid mathematical they at represent a new family foundations. References Abney, Proceedings Steven. 2002. 40th Bootstrapping. Meeting In of Computational Annual Linguistics of the Association (ACL), Philadelphia, for Blum, Tom Mitchell. pages 360–367. Combining Avrim and 1998. with labeled In and Proceedings unlabeled data co-training. Conference Computational of the 11th Annual Learning Theory (COLT) on Morgan Kaufmann, San , Francisco. pages 92–100. Collins, Unsupervised Michael and models Yoram Singer. 1999. In Proceedings for named Empirical entity classification. Methods Natural Language Processing of in (EMNLP) 100–110. , College Park, MD, Dasgupta, pages David McAllester. Sanjoy, Michael PAC Littman, and 2001. generalization Proceedings bounds for Neural co-training. In Information Processing of Advances Systems in Vancouver, British Columbia, 14 Canada. (NIPS) , Yarowsky, David. 1995. Unsupervised word

sense methods. disambiguation In Proceedings rivaling supervised Meeting of the Linguistics , Cambridge, Association MA, for Computational 33rd Annual of the 189–196. pages