<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="496"/><title>Multi-Class Confidence Weighted Algorithms</title><author surname="Crammer" givenname="Koby"><org  name="University of Pennsylvania" country="USA" city="Philadelphia"/></author><author surname="Dredze" givenname="Mark"><org  name="University of Pennsylvania" country="USA" city="Philadelphia"/></author><author surname="Kulesza" givenname="Alex"><org  name="University of Pennsylvania" country="USA" city="Philadelphia"/></author></firstpageheader><frontmatter><p><b>Multi-Class Confidence Weighted Algorithms</b></p><p><b>Koby Crammer* Mark Dredze^ Alex Kulesza*</b></p><p>*Department of Computer ^Human Language Technology</p><p>and Information Science Center of Excellence</p><p>University of Pennsylvania Johns Hopkins University</p><p>Philadelphia, PA 19104 Baltimore, MD 21211</p><p>{crammer,kulesza}@cis.upenn.edu mdredze@cs.jhu.edu</p></frontmatter><abstract>The recently introduced online confidence-weighted (CW) learning algorithm for binary classification per­forms well on many binary NLP tasks. However, for multi-class problems CW learning updates and inference cannot be computed analytically or solved as convex optimization problems as they are in the binary case. We derive learning algorithms for the multi-class CW setting and provide extensive evaluation using nine NLP datasets, including three derived from the recently released New York Limes corpus. Our best algorithm out­performs state-of-the-art online and batch methods on eight of the nine tasks. We also show that the confidence information maintained during learning yields useful probabilistic information at test time. </abstract></header><body><section number="1" title="Introduction"><p>Online learning algorithms such as the Perceptron process one example at a time, yielding simple and fast updates. Lhey generally make few statisti­cal assumptions about the data and are often used for natural language problems, where high dimen­sional feature representations, e.g., bags-of-words, demand efficiency. Most online algorithms, how­ever, do not take into account the unique properties of such data, where many features are extremely rare and a few are very frequent.</p><p>Dredze, Crammer and Pereira (Dredze et al., 2008; Crammer et al., 2008) recently introduced confidence weighted (CW) online learning for bi­nary prediction problems. CW learning explicitly models classifier weight uncertainty using a multi­variate Gaussian distribution over weight vectors. Lhe learner makes online updates based on its con­fidence in the current parameters, making larger changes in the weights of infrequently observed features. Empirical evaluation has demonstrated the advantages of this approach for a number of bi­nary natural language processing (NLP) problems.</p><p>In this work, we develop and test multi-class confidence weighted online learning algorithms. Lor binary problems, the update rule is a sim­ple convex optimization problem and inference is analytically computable. However, neither is true in the multi-class setting. We discuss sev­eral efficient online learning updates. Lhese up­date rules can involve one, some, or all of the competing (incorrect) labels. We then perform an extensive evaluation of our algorithms using nine multi-class NLP classification problems, includ­ing three derived from the recently released New York Limes corpus (Sandhaus, 2008). Lo the best of our knowledge, this is the first learning evalua­tion on these data. Our best algorithm outperforms state-of-the-art online algorithms and batch algo­rithms on eight of the nine datasets.</p><p>Surprisingly, we find that a simple algorithm in which updates consider only a single competing label often performs as well as or better than multi-constraint variants if it makes multiple passes over the data. Lhis is especially promising for large datasets, where the efficiency of the update can be important. In the true online setting, where only one iteration is possible, multi-constraint al­gorithms yield better performance.</p><p>Pinally, we demonstrate that the label distribu­tions induced by the Gaussian parameter distribu­tions resulting from our methods have interesting properties, such as higher entropy, compared to those from maximum entropy models. Improved label distributions may be useful in a variety of learning settings.</p></section><section number="2" title="Problem Setting"><p>In the multi-class setting, instances from an input space <i>X </i>take labels from a finite set <i>y, \y\ = K.</i></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">496</doubt><p><i>Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, </i>pages 496-504, Singapore, 6-7 August 2009. ©2009 ACL and AFNLP<page local="2" global="497"/></p><p>We use a standard approach (Collins, 2002) for generalizing binary classification and assume a feature function <i>f</i><i>(x,y) </i>G <i>M.d </i>mapping instances <i>x </i>G <i>X </i>and labels <i>y </i>G <i>y </i>into a common space.</p><p>We work in the online framework, where learn­ing is performed in rounds. On each round the learner receives an input <i>Xi, </i>makes a prediction <i>fji </i>according to its current rule, and then learns the true label <i>yi. </i>The learner uses the new example <i>(xi, yi) </i>to modify its prediction rule. Its goal is to minimize the total number of rounds with incor­rect predictions, <i>\{i : yi ^ fji}\.</i></p><p>In this work we focus on linear models parame­terized by weights <i>w </i>and utilizing prediction func­tions of the form <i>hw(x) = </i>arg max2 <i>w ■ </i><i>f</i><i>(x, z). </i>Note that since we can choose <i>f</i><i>(x,y) </i>to be the vectorized Cartesian product of an input feature function <i>g</i><i>(x) </i>and <i>y, </i>this setup generalizes the use of unique weight vectors for each element of <i>y.</i></p></section><section number="3" title="Confidence Weighted Learning"><p>Dredze, Crammer, and Pereira (2008) introduced online confidence weighted (CW) learning for bi­nary classification, where <i>X = Rd </i>and <i>y = </i>{±1}. Rather than using a single parameter vec­tor <i>w, </i>CW maintains a distribution over param­eters TV (/it, E), where TV (/it, E) the multivariate normal distribution with mean <b><i>/j, </i></b>G <i>M.d </i>and co-variance matrix E G Rdx&lt;i. Given an input in­stance <i>x, </i>a Gibbs classifier draws a weight vector <i>w </i>from the distribution and then makes a predic­tion according to the sign of <i>w ■ x.</i></p><p>This prediction rule is robust if the example is classified correctly with high-probability, that is, for some confidence parameter .5 &lt; <i>rj &lt; </i>1, Pr^, <i>[y (w ■ x) &gt; </i>0] &gt; <i>rj. </i>To learn a binary CW classifier in the online framework, the robustness property is enforced at each iteration while mak­ing a minimal update to the parameter distribution in the KL sense:</p><doubt alpha="45.5" length="11" tooSmall="False" monospace="0.0">(Mi+lÄ+l) =</doubt><doubt alpha="42.1" length="38" tooSmall="False" monospace="0.0">arg min DKL(A/" (/it, E) || TV(^, E;))</doubt><doubt alpha="40.7" length="27" tooSmall="False" monospace="0.0">s.t. Prw[yi (w ■ x^ &gt;0] &gt;rj</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(1)</doubt><p>Dredze et al. (2008) showed that this optimization can be solved in closed form, yielding the updates <i>ßi+l</i><i> = Vi + </i><i>OLiYliXi</i> for appropriate    and <i>ßi.</i><i></i></p><doubt alpha="56.2" length="16" tooSmall="False" monospace="0.0">Ei+i= (E"1+ßiXiX</doubt><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">T\-l</doubt><doubt alpha="0.0" length="7" tooSmall="False" monospace="0.0">(2) (3)</doubt><p>For prediction, they use the Bayesian rule which for binary labels is equivalent to using the mean parameters directly, <i>y = </i>sign <b><i>(/j</i></b><i>, ■ x).</i><i></i></p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">y</doubt><doubt alpha="45.2" length="31" tooSmall="False" monospace="0.0">arg maxPrw^M{E)[z (x ■ w) &gt;0] ,</doubt></section><section number="4" title="Multi-Class Confidence Weighted Learning"><p>As in the binary case, we maintain a distribution over weight vectors <i>w </i>~ TV (/it, E). Given an in­put instance <i>x, </i>a Gibbs classifier draws a weight vector <i>w </i>~ TV (/it, E) and then predicts the label with the maximal score, argmax^ <i>(w ■ </i><i>f</i><i>(x,z)). </i>As in the binary case, we use this prediction rule to define a robustness condition and corresponding learning updates.</p><p>We generalize the robustness condition used in Crammer et al. (2008). Following the update on round <i>i, </i>we require that the <i>ith </i>instance is correctly labeled with probability at least <i>rj &lt; </i>1. Among the distributions that satisfy this condition, we choose the one that has the minimal KL distance from the current distribution. Lhis yields the update</p><doubt alpha="44.4" length="9" tooSmall="False" monospace="0.0">(A^+iÄ+i)</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(4)</doubt><doubt alpha="48.5" length="66" tooSmall="False" monospace="0.0">arg min DKL(A/" (/it, E) || TV (/Lti;E^)) s.t.Vr[yi\xi,n,YJ] &gt; rj,</doubt><doubt alpha="100.0" length="5" tooSmall="False" monospace="0.0">where</doubt><doubt alpha="46.2" length="13" tooSmall="False" monospace="0.0">Pr[y|x,/x,E]=</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">Pr</doubt><doubt alpha="25.0" length="12" tooSmall="False" monospace="0.0">tt)~A/"(&gt;,£)</doubt><doubt alpha="50.0" length="22" tooSmall="False" monospace="0.0">y =arg max(w ■f(x, z))</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">zey</doubt><p>Due to the max operator in the constraint, this op­timization is not convex when <i>K &gt; </i>2, and it does not permit a closed form solution. We therefore develop approximations that can be solved effi­ciently. We define the following set of events for a general input <i>x:</i></p><doubt alpha="37.9" length="87" tooSmall="False" monospace="0.0">Ar,s(x)d={w : w ■f(x, r) &gt; w ■f(x, s)} Br(x)d={w : w ■f(x, r) &gt; w ■f(x, s)Vs}=p|Ar&gt;8(x)</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">s^r</doubt><p>We assume the probability that <i>w ■ </i><i>f</i><i>(x,r) = w ■ </i><i>f</i><i>(x,s) </i>for some <i>s  /  r </i>is zero, which holds for non-trivial distribution parameters and feature vectors.<page local="3" global="498"/> We rewrite the prediction <i>y = </i>argmaxr Pr <i>[Br(xj\, </i>and the constraint from Eq. (4) becomes</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">497</doubt><doubt alpha="17.6" length="17" tooSmall="False" monospace="0.0">Pr[2?w(*)]&gt;»?.(5)</doubt><p>We focus now on approximating the event <i>Byi(x) </i>in terms of events <i>Ayur.</i><i> </i>We rely on the fact that the level sets of Pr <i>[Ayur] </i>are convex in <b><i>/j, </i></b>and E. This leads to convex constraints of the form <b>Outer Bound:</b><b> </b>Since <i>Br(x) </i>ç <i>Ar&gt;s(x), </i>it holds trivially that Pr <i>[Bm (x)} &gt; rj </i><i>=&gt;</i><i> Pr[Ayur] &gt; </i>?y,Vr / <i>yi. </i>Thus we can replace the constraint Pr <b>[^(a;)] &gt; </b><i>r] </i>with Pr <i>[Ayur] &gt; </i><i>r\</i><i> </i>to achieve an outer bound. We can simultaneously apply <i>all </i>of the pairwise constraints to achieve a tighter bound:</p><doubt alpha="37.5" length="8" tooSmall="False" monospace="0.0">^[AmA&gt;7-</doubt><doubt alpha="61.1" length="18" tooSmall="False" monospace="0.0">Pr[Ayur] &gt;r] Vr^yt</doubt><p>This yields a convex approximation to Eq. (4) that may improve the objective value at the cost of violating the constraint. In the context of learn­ing, this means that the new parameter distribu­tion will be close to the previous one, but may not achieve the desired confidence on the current ex­ample. This makes the updates more conservative.</p><p><b>Inner Bound: </b>We can also consider an inner bound. Note that <i>Byi(x)c = (nrAyitr(x))c = UrAyur(x)c, </i>thus the constraint Pr <i>[Byi{x)\ &gt; </i><i>r\</i><i> </i>is equivalent to and by the union bound, this follows whenever ^Pr^^H^l-r?.</p><doubt alpha="41.7" length="24" tooSmall="False" monospace="0.0">Vr[\JrAyur{x)c] &lt;1 — ?7,</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">r</doubt><p>We can achieve this by choosing non-negative Cr &gt; 0, <i>J2r </i>Cr = 1. and constraining</p><p>Pr <i>[Ayur(x)} &gt; </i>1 - (1 - <i>rj) (r </i>for   <i>r + </i><i>y%</i><i> </i><i>.</i></p><p>This formulation yields an inner bound on the original constraint, guaranteeing its satisfaction while possibly increasing the objective. In the context of learning, this is a more aggressive up­date, ensuring that the current example is robustly classified even if doing so requires a larger change to the parameter distribution.</p><p><b>Algorithm 1 </b>Multi-Class CW Online Algorithm <b>Input: </b>Confidence parameter r?</p><p>Feature function <i>f</i><i>(x, y) </i>G <i>M.d</i><i></i><b>Initialize: </b>/Lt1 = <b>0 </b>, Ei = / <b>for </b><i>i = </i>1,2 <b>... do </b>Receive <i>Xi </i>G <i>X</i></p><p>Predict ranking of labels <i>yi, y</i><b><i>2,</i></b><b><i> </i></b><b><i>■</i></b><b><i> </i></b><b><i>■</i></b><b><i> </i></b><b><i>■</i></b><b><i> </i></b>Receive <i>yi</i><i> </i>G <i>y</i></p><p>Set £j+i by approximately solving</p><p>Eq. (4) using one of the following:</p><p>Single-constraint update (Sec. 5.1)</p><p>Exact many-constraint update   (Sec. 5.2) Seq. many-constraint approx.    (Sec. 5.2) Parallel many-constraint approx. (Sec. 5.2) <b>end for</b> <b>Output:</b><b> </b>Final <i>n </i>and E <b>Discussion:</b><b> </b>The two approximations are quite similar in form. Both replace the constraint Pr <i>[Byi(x)\ &gt; T] </i>with one or more constraints of the form</p><doubt alpha="47.4" length="19" tooSmall="False" monospace="0.0">Pr[Ayur(x)}&gt;r]r.(6)</doubt><p>To achieve an outer bound we choose <i>r]r</i><i> </i><i>=</i><i> </i><i>r]</i><i> </i>for any set of <i>r</i><i> </i><i>/</i><i> </i><i>yi.</i><i> </i>To achieve an inner bound we use all <i>K</i><i> </i><i>—</i><i> </i>1 possible constraints, setting <i>r]r</i><i> </i><i>=</i><i> </i>1 — (1 — <i>rj)</i><i> </i><i>Ç</i><i> </i>for suitable Cr- A simple choice is</p><p>Cr <i>=</i><i> </i><i>l/(K-l).</i></p><p>In practice, <i>r]</i><i> </i>is a learning parameter whose value will be optimized for each task. In this case, the outer bound (when all constraints are included) and inner bound (when <i>(r = 1/(K — </i>1)) can be seen as equivalent, since for any fixed value of ^(in) for the inner bound we can choose for the outer bound and the resulting <i>r\r</i><i> </i>will be equal. By optimizing <i>r\</i><i> </i>we automatically tune the approximation to achieve the best compromise be­tween the inner and outer bounds. In the follow­ing, we will therefore assume <i>r]r</i><i> </i><i>=</i><i> </i><i>r\.</i></p></section><section number="5" title="Online Updates"><p>Our algorithms are online and process examples one at a time. Pseudo-code for our approach is given in algorithm 1. We approximate the pre­diction step by ranking each label <i>y </i>according to the score given by the mean weight vector, /it • <i>f</i><i>(xi, y). </i>Although this approach is Bayes op­timal for binary problems (Dredze et al., 2008), it is an approximation in general. We note that more accurate inference can be performed in the multi-class case by sampling weight vectors from the distribution A/"(/x, E) or selecting labels sen­sitive to the variance of prediction; however, in our experiments this did not improve performance and required significantly more computation.<page local="4" global="499"/> We therefore proceed with this simple and effective approximation.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">498</doubt><p>The update rule is given by an approximation of the type described in Sec. 4. All that remains is to choose the constraint set and solve the opti­mization efficiently. We discuss several schemes for minimizing KL divergence subject to one or more constraints of the form Pr [Ayi r(a;)] &gt; <i>rj. </i>We start with a single constraint.</p><subsection number="5.1" title="Single-Constraint Updates"><p>The simplest approach is to select the single con­straint Pr [Aw r(a;)] &gt; <i>rj</i><i> </i>corresponding to the highest-ranking label <i>r ^ </i><i>yi. </i>This ensures that, following the update, the true label is more likely to be predicted than the label that was its closest competitor. We refer to this as the <i>k = </i>1 update.</p><p>Whenever we have only a single constraint, we can reduce the optimization to one of the closed-form CW updates used for binary classification. Several have been proposed, based on linear ap­proximations (Dredze et al., 2008) and exact for­mulations (Crammer et al., 2008). For simplicity, we use the Variance method from Dredze et al. (2008), which did well in our initial evaluations. This method leads to the following update rules. Note that in practice E is projected to a diagonal matrix as part of the update; this is necessary due to the large number of features that we use.</p><doubt alpha="48.0" length="25" tooSmall="False" monospace="0.0">Mi+i = Mi +ai^i9i,yi,r(7)</doubt><doubt alpha="60.9" length="23" tooSmall="False" monospace="0.0">=(X"1+ZOifaiMsQlyis)(8)</doubt><doubt alpha="46.7" length="30" tooSmall="False" monospace="0.0">9i,yi,r= f(xi,Vi) -f(xi,r)(f)=</doubt><p>The scale is given by max(7i, 0), where 7^ is equal to</p><doubt alpha="33.3" length="51" tooSmall="False" monospace="0.0">-(1 +2&lt;fmn)+ ./(l +2&lt;j)mj)2- S&lt;j){mj- &lt;j)Vj) 4(f)Vi</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">and</doubt><doubt alpha="48.8" length="41" tooSmall="False" monospace="0.0">mi = H-i ' 9i,yitrVi=9i,yi,r'^'i9i,yi,r •</doubt><p>These rules derive directly from Dredze et al. (2008) or Figure 1 in Crammer et al. (2008); we simply substitute <i>y% = </i><i>1 </i>and <i>Xi</i><i> </i><i>=</i><i> </i><i>9iyir-</i></p></subsection><subsection number="5.2" title="Many-Constraints Updates"><p>A more accurate approximation can be obtained by selecting multiple constraints. Analogously, we choose the <i>k </i><i>&lt; K — l </i>constraints corresponding to the labels n,... <i>,rk </i>/ <i>yi </i>that achieve the highest predicted ranks. The resulting optimization is con­vex and can be solved by a standard Hildreth-like algorithm (Censor &amp; Zenios, 1997). We refer to this update as Exact. However, Exact is expen­sive to compute, and tends to over-fit in practice (Sec. 6.2). We propose several approximate alter­natives.</p><p><b>Sequential Update: </b>The Hildreth algorithm it­erates over the constraints, updating with respect to each until convergence is reached. We approxi­mate this solution by making only a single pass:</p><p>• Set <b><i>fj,i&gt;0 </i></b><i>= Hi </i>and Ei;0 = E;.</p><doubt alpha="66.2" length="77" tooSmall="False" monospace="0.0">• Forj=1,...,k,set (/^j, Ejj) to the solu­tion of the following optimization:</doubt><doubt alpha="44.4" length="27" tooSmall="False" monospace="0.0">mm DKL(A/-(/x,E)\\M {m,j-i,</doubt><doubt alpha="56.2" length="16" tooSmall="False" monospace="0.0">s.tPr[Ay^ix)]&gt;r,</doubt><p>• Set       <i>=</i><i> </i><i>ni&gt;k</i><i> </i>and Em = Ei;fc.</p><p><b>Parallel Update: </b>As an alternative to the Hil­dreth algorithm, we consider the simultaneous al­gorithm of Iusem and Pierro (1987), which finds an exact solution by iterating over the constraints in parallel. As above, we approximate the exact solution by performing only one iteration. The process is as follows.</p><doubt alpha="48.9" length="47" tooSmall="False" monospace="0.0">minDKL(AA(/lx,E) ||AA(^,Et))s.t.Pr[^.)r.(x)]&gt;rj</doubt><doubt alpha="47.2" length="36" tooSmall="False" monospace="0.0">• Let A be a vector,Xj&gt;0,J2j-\? = 1-</doubt><doubt alpha="48.1" length="27" tooSmall="False" monospace="0.0">• Setni+1=^2jXj^ij,Ei+\=52j</doubt><p>In practice we set <i>Xj</i><i> </i><i>=</i><i> </i><i>1</i><i>/k</i><i> </i>for all <i>j.</i></p></subsection></section><section number="6" title="Experiments 6.1 Datasets"><p>Following the approach of Dredze et al. (2008), we evaluate using five natural language classifica­tion tasks over nine datasets that vary in difficulty, size, and label/feature counts. See Lable 1 for an overview. Brief descriptions follow.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">499</doubt><page local="5" global="500"/><p>Table 1: A summary of the nine datasets, includ­ing the number of instances, features, and labels, and whether the numbers of examples in each class are balanced.</p><p><b>Amazon </b>Amazon product reviews. Using the data of Dredze et al. (2008), we created two do­main classification datasets from seven product types (apparel, books, dvds, electronics, kitchen, music, video). <i>Amazon</i><i> </i><i>7</i><i> </i>includes all seven prod­uct types and <i>Amazon</i><i> </i><i>3</i><i> </i>includes books, dvds, and music. Feature extraction follows Blitzer et al. (2007) (bigram features and counts).</p><p><b>20 Newsgroups </b>Approximately 20,000 news­group messages, partitioned across 20 different newsgroups.<footnote anchor="1"/> This dataset is a popular choice for binary and multi-class text classification as well as unsupervised clustering. We represent each mes­sage as a binary bag-of-words.</p><p><b>Enron </b>Automatic sorting of emails into fold­ers.<footnote anchor="2"/> We selected two users with many email folders and messages: f armer-d <i>(Enron</i><i> </i><i>A)</i><i> </i>and kaminski-v <i>(Enron</i><i> B). </i>We used the ten largest folders for each user, excluding non-archival email folders such as "inbox," "deleted items," and "dis­cussion threads." Emails were represented as bi­nary bags-of-words with stop-words removed.</p><p><b>NY Times </b>To the best of our knowledge we are the first to evaluate machine learning methods on the New York Times corpus. The corpus con­tains 1.8 million articles that appeared from 1987 to 2007 (Sandhaus, 2008). In addition to being one of the largest collections of raw news text, it is possibly the largest collection of publicly re­leased annotated news text, and therefore an ideal corpus for large scale NLP tasks. Among other annotations, each article is labeled with the desk that produced the story (Financial, Sports, etc.) <i>(NYTD), </i>the online section to which the article was</p><footnote>http://people.csail.mit.edu/jrennie/20Newsgroups/</footnote><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">2</doubt><doubt alpha="55.3" length="38" tooSmall="False" monospace="0.0">http : //www . cs . emu . edu/"enroll/</doubt><p>Table 2: A comparison of <i>k = </i>oo updates. While the two approximations (sequential and parallel) are roughly the same, the exact solution over-fits.</p><p>posted <i>(NYTO), </i>and the section in which the arti­cle was printed <i>(NYTS). </i>Articles were represented as bags-of-words with feature counts (stop-words removed).</p><p><b>Reuters </b>Over 800,000 manually categorized newswire stories (RCVl-v2/ LYRL2004). Each article contains one or more labels describing its general topic, industry, and region. We performed topic classification with the four general topics: corporate, economic, government, and markets. Details on document preparation and feature ex­traction are given by Lewis et al. (2004).</p><subsection number="6.2" title="Evaluations"><p>We first set out to compare the three update ap­proaches proposed in Sec. 5.2: an exact solution and two approximations (sequential and parallel). Results (Lable 2) show that the two approxima­tions perform similarly. For every experiment the CW parameter r? and the number of iterations (up to 10) were optimized using a single randomized iteration. However, sequential converges faster, needing an average of 4.33 iterations compared to 7.56 for parallel across all datasets. Lherefore, we select sequential for our subsequent experiments.</p><p>Lhe exact method performs poorly, displaying the lowest performance on almost every dataset. Lhis is unsurprising given similar results for bi­nary CW learning Dredze et al. (2008), where ex­act updates were shown to over-fit but converged after a single iteration of training. Similarly, our exact implementation converges after an average of 1.25 iterations, much faster than either of the approximations. However, this rapid convergence appears to come at the expense of accuracy. Fig. 1 shows the accuracy on Amazon 7 test data after each training iteration. While both sequential and parallel improve with several iterations, exact de-<page local="6" global="501"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">500</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Task</i></p></td><td class="cell"><p><i>Instances</i></p></td><td class="cell"><p><i>Features</i></p></td><td class="cell"><p><i>Labels</i></p></td><td class="cell"><p><i>Bal.</i></p></td><td class="cell"><p></p></td><td class="cell"><p><i>Task</i></p></td><td class="cell"><p><i>Sequential</i></p></td><td class="cell"><p><i>Parallel</i></p></td><td class="cell"><p><i>Exact</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>20 News</p></td><td class="cell"><p>18,828</p></td><td class="cell"><p>252,115</p></td><td class="cell"><p>20</p></td><td class="cell"><p><i>Y</i></p></td><td class="cell"><p></p></td><td class="cell"><p>20 News</p></td><td class="cell"><p>92.16</p></td><td class="cell"><p>91.41</p></td><td class="cell"><p>88.08</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Amazon 7</p></td><td class="cell"><p>13,580</p></td><td class="cell"><p>686,724</p></td><td class="cell"><p>7</p></td><td class="cell"><p><i>Y</i></p></td><td class="cell"><p></p></td><td class="cell"><p>Amazon 7</p></td><td class="cell"><p>77.98</p></td><td class="cell"><p>78.35</p></td><td class="cell"><p>77.92</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Amazon 3</p></td><td class="cell"><p>7,000</p></td><td class="cell"><p>494,481</p></td><td class="cell"><p>3</p></td><td class="cell"><p><i>Y</i></p></td><td class="cell"><p></p></td><td class="cell"><p>Amazon 3</p></td><td class="cell"><p>93.54</p></td><td class="cell"><p>93.81</p></td><td class="cell"><p>93.00</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Enron A</p></td><td class="cell"><p>3,000</p></td><td class="cell"><p>13,559</p></td><td class="cell"><p>10</p></td><td class="cell"><p>N</p></td><td class="cell"><p></p></td><td class="cell"><p>Enron A</p></td><td class="cell"><p>82.40</p></td><td class="cell"><p>81.30</p></td><td class="cell"><p>77.07</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Enron B</p></td><td class="cell"><p>3,000</p></td><td class="cell"><p>18,065</p></td><td class="cell"><p>10</p></td><td class="cell"><p>N</p></td><td class="cell"><p></p></td><td class="cell"><p>Enron B</p></td><td class="cell"><p>71.80</p></td><td class="cell"><p>72.13</p></td><td class="cell"><p>68.00</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NYTD</p></td><td class="cell"><p>10,000</p></td><td class="cell"><p>108,671</p></td><td class="cell"><p>26</p></td><td class="cell"><p>N</p></td><td class="cell"><p></p></td><td class="cell"><p>NYTD</p></td><td class="cell"><p>83.43</p></td><td class="cell"><p>81.43</p></td><td class="cell"><p>80.92</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NYTO</p></td><td class="cell"><p>10,000</p></td><td class="cell"><p>108,671</p></td><td class="cell"><p>34</p></td><td class="cell"><p>N</p></td><td class="cell"><p></p></td><td class="cell"><p>NYTO</p></td><td class="cell"><p>82.02</p></td><td class="cell"><p>78.67</p></td><td class="cell"><p>80.60</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NYTS</p></td><td class="cell"><p>10,000</p></td><td class="cell"><p>114,316</p></td><td class="cell"><p>20</p></td><td class="cell"><p>N</p></td><td class="cell"><p></p></td><td class="cell"><p>NYTS</p></td><td class="cell"><p>52.96</p></td><td class="cell"><p>54.78</p></td><td class="cell"><p>51.62</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Reuters</p></td><td class="cell"><p>4,000</p></td><td class="cell"><p>23,699</p></td><td class="cell"><p>4</p></td><td class="cell"><p>N</p></td><td class="cell"><p></p></td><td class="cell"><p>Reuters</p></td><td class="cell"><p>93.60</p></td><td class="cell"><p>93.97</p></td><td class="cell"><p>93.47</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><doubt alpha="0.0" length="4" tooSmall="True" monospace="0.0">78.5</doubt><doubt alpha="0.0" length="9" tooSmall="True" monospace="0.0">1 2 3 4 5</doubt><p><b>Training Iterations</b></p><p>Figure 1 : Accuracy on test data after each iteration on the Amazon 7 dataset.</p><p>grades after the first iteration, suggesting that it may over-fit to the training data. The approxima­tions appear to smooth learning and produce better performance in the long run.</p></subsection><subsection number="6.3" title="Relaxing Many-Constraints"><p>While enforcing many constraints may seem op­timal, there are advantages to pruning the con­straints as well. It may be time consuming to en­force dozens or hundreds of constraints for tasks with many labels. Structured prediction tasks of­ten involve exponentially many constraints, mak­ing pruning mandatory. Furthermore, many real world datasets, especially in NLP, are noisy, and enforcing too many constraints can lead to over-fitting. Lherefore, we consider the impact of re­ducing the constraint set in terms of both reducing run-time and improving accuracy.</p><p>We compared using all constraints <i>(k</i><i> </i><i>=</i><i> </i>oo) with using 5 constraints <i>(k</i><i> </i><i>=</i><i> </i>5) for the sequential update method (Lable 3). First, we observe that <i>k = </i>5 performs better than <i>k = </i>oo on nearly every dataset: fewer constraints help avoid over-fitting and once again, simpler is better. Additionally, <i>k = </i>5 converges faster than <i>k = </i>oo in an average of 2.22 iterations compared with 4.33 iterations. Lherefore, reducing the number of constraints im­proves both speed and accuracy. In comparing <i>k = </i>5 with the further reduced <i>k = </i>1 results, we observe the latter improves on seven of the nine methods. Lhis surprising result suggests that CW learning can perform well even without consid­ering more than a single constraint per example. However, <i>k = </i>1 exceeds the performance of multiple constraints only through repeated training it­erations, <i>k = </i>5 CW learning converges faster — 2.22 iterations compared with 6.67 for <i>k = </i>1 — a desirable property in many resource restricted set­tings. (In the true online setting, only a single it­eration may be possible.) Fig. 1 plots the perfor­mance of <i>k = </i>1 and <i>k = </i>5 CW on test data after each training iteration. While <i>k = </i>1 does better in the long run, it lags behind <i>k = </i>5 for several iterations. In fact, after a single training iteration, <i>k = </i>5 outperforms <i>k = </i>1 on eight out of nine datasets. Thus, there is again a tradeoff between faster convergence <i>(k</i><i> </i><i>=</i><i> </i>5) and increased accuracy <i>(k</i><i> </i><i>=</i><i> </i>1). While the <i>k</i><i> </i><i>=</i><i> </i>5 update takes longer per iteration, the time required for the approximate so­lutions grows only linearly in the number of con­straints. The evaluation in Fig. 1 required 3 sec­onds for the first iteration of <i>k = </i>1, 10 seconds for <i>k = </i>5 and 11 seconds for one iteration of all 7 constraints. These differences are insignificant compared to the cost of performing multiple itera­tions over a large dataset. We note that, while both approximate methods took about the same amount of time, the exact solution took over 4 minutes for its first iteration.</p><p>Finally, we compare CW methods with sev­eral baselines in Table 3. Online baselines in­clude Top-1 Perceptron (Collins, 2002), Top-1 Passive-Aggressive (PA), and fc-best PA (Cram­mer &amp; Singer, 2003; McDonald et al., 2004). Batch algorithms include Maximum Entropy (de­fault configuration in McCallum (2002)) and sup­port vector machines (LibSVM (Chang &amp; Lin, 2001) for one-against-one classification and multi-class (MC) (Crammer &amp; Singer, 2001)). Classifier parameters (C for PA/SVM and maxent's Gaus­sian prior) and number of iterations (up to 10) for the online methods were optimized using a sin­gle randomized iteration. On eight of the nine datasets, CW improves over all baselines. In gen­eral, CW provides faster and more accurate multi-class predictions.</p></subsection></section><section number="7" title="Error and Probabilistic Output"><p>Our focus so far has been on accuracy and speed. However, there are other important considerations for selecting learning algorithms. Maximum en­tropy and other probabilistic classification algo­rithms are sometimes favored for their probabil­ity scores, which can be useful for integration with other learning systems. However, practition-<page local="7" global="502"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">501</doubt><p>Table 3: A comparison of CW learning <i>(k</i><i> </i>= 1, 5, oo with sequential updates) with several baseline algorithms. CW learning achieves the best performance eight out of nine times. Statistical significance (McNemar) is measured against <i>all </i>baselines (* indicates 0.05 and ** 0.001) or against online baselines (t indicates 0.05 and ft 0.001).</p><p>Figure 2: <b>First panel: </b>Error versus prediction entropy on Enron B. As CW converges (right to left) error and entropy are reduced. <b>Second panel: </b>Number of test examples per prediction probability bin. The red bars correspond to maxent and the blue bars to CW, with increasing numbers of epochs from left to right. <b>Third panel: </b>The contribution of each bin to the total test error. <b>Fourth panel: </b>Test error conditioned on prediction probability.</p><p>ers have observed that maxent probabilities can have low entropy and be unreliable for estimating prediction confidence (Malkin &amp; Bilmes, 2008). Since CW also produces label probabilities — and does so in a conceptually distinct way — we in­vestigate in this section some empirical properties of the label distributions induced by CW's param­eter distributions and compare them with those of maxent.</p><p>We trained maxent and CW <i>k = </i>1 classi­fiers on the Enron B dataset, optimizing parame­ters as before (maxent's Gaussian prior and CW's <i>rj). </i>We estimated the label distributions from our CW classifiers after each iteration and on every test example <i>x </i>by Gibbs sampling weight vec­tors <i>w </i>~ A/"(/x, £), and for each label <i>y </i>count­ing the fraction of weight vectors for which <i>y = </i>argmax^ <i>w</i><i> </i><i>■</i><i> </i><i>f(x,</i><i> </i><i>z).</i><i> </i>Normalizing these counts yields the label distributions Pr <i>[y\x].</i><i> </i>We denote by <i>y</i><i> </i>the predicted label for a given <i>x,</i><i> </i>and refer to Pr <i>[y\x]</i><i> </i>as the prediction probability.</p><p>The leftmost panel of Fig. 2 plots each method's  prediction  error  against  the normalized entropy of the label distribution <b>" Ei E, </b>Pr <i>[z</i><i>\xi]</i><i> </i>log (Pr <i>[z</i><i>\xi]))</i><i> </i><i>/</i><i> </i><i>\og(K).</i><i> </i>Each CW iteration (moving from right to left in the plot) reduces both error and entropy. From our maxent results we make the common observation that maxent distributions have (ironically) low entropy. In contrast, while CW accuracy exceeds maxent after its second iteration, normalized entropy remains high. Higher entropy suggests a distribution over labels that is less peaked and potentially more informative than those from maxent. We found that the average probability assigned to a correct prediction was 0.75 for CW versus 0.83 for maxent and for an incorrect prediction was 0.44 for CW versus 0.56 for maxent.</p><p>Next, we investigate how these probabilities relate to label accuracy. In the remaining pan­els, we binned examples according to their pre­diction probabilities Pr <i>[y\</i><i> </i><i>x]</i><i> </i><i>=</i><i> </i><i>maxyPv[y\x].</i><i> </i>The second panel of Fig. 2 shows the numbers of test examples with Pr <i>[y\x]</i><i> </i>e <i>[0,9</i><i> </i><i>+</i><i> </i>0.1) for <i>9 = </i>0.1, 0.2 ... 0.9. (Note that since there are 10 classes in this problem, we must have Pr <i>[y\x] &gt; </i>0.<page local="8" global="503"/>1.) The red (leftmost) bar corresponds to the maximum entropy classifier, and the blue bars cor­respond, from left to right, to CW after each suc­cessive training epoch.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">502</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p><i>PA</i></p></td><td class="cell"><p></p></td><td class="cell"><p><i>CW</i></p></td><td class="cell"><p></p></td><td class="cell"><p><i>SVM</i></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Task</i></p></td><td class="cell"><p><i>Perceptron</i></p></td><td class="cell"><p><i>K=l</i></p></td><td class="cell"><p><i>K=5</i></p></td><td class="cell"><p><i>K=l</i></p></td><td class="cell"><p><i>K=5</i></p></td><td class="cell"><p><i>K=&lt;x</i></p></td><td class="cell"><p><i>1 vs. 1</i></p></td><td class="cell"><p><i>MC</i></p></td><td class="cell"><p><i>Maxent</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>20 News</p></td><td class="cell"><p>81.07</p></td><td class="cell"><p>88.59</p></td><td class="cell"><p>88.60</p></td><td class="cell"><p>**92.90</p></td><td class="cell"><p>**92.78</p></td><td class="cell"><p>**92.16</p></td><td class="cell"><p>85.18</p></td><td class="cell"><p>90.33</p></td><td class="cell"><p>88.94</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Amazon 7</p></td><td class="cell"><p>74.93</p></td><td class="cell"><p>76.55</p></td><td class="cell"><p>76.72</p></td><td class="cell"><p>**78.70</p></td><td class="cell"><p>**78.04</p></td><td class="cell"><p>**77.98</p></td><td class="cell"><p>75.11</p></td><td class="cell"><p>76.60</p></td><td class="cell"><p>76.40</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Amazon 3</p></td><td class="cell"><p>92.26</p></td><td class="cell"><p>92.47</p></td><td class="cell"><p>93.29</p></td><td class="cell"><p>f94.01</p></td><td class="cell"><p>**94.29</p></td><td class="cell"><p>93.54</p></td><td class="cell"><p>92.83</p></td><td class="cell"><p>93.60</p></td><td class="cell"><p>93.60</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Enron A</p></td><td class="cell"><p>74.23</p></td><td class="cell"><p>79.27</p></td><td class="cell"><p>80.77</p></td><td class="cell"><p>ff83.83</p></td><td class="cell"><p>f82.23</p></td><td class="cell"><p>f82.40</p></td><td class="cell"><p>80.23</p></td><td class="cell"><p>82.60</p></td><td class="cell"><p>82.80</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Enron B</p></td><td class="cell"><p>66.30</p></td><td class="cell"><p>69.93</p></td><td class="cell"><p>68.90</p></td><td class="cell"><p>**73.57</p></td><td class="cell"><p>**72.27</p></td><td class="cell"><p>**71.80</p></td><td class="cell"><p>65.97</p></td><td class="cell"><p>71.87</p></td><td class="cell"><p>69.47</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NYTD</p></td><td class="cell"><p>80.67</p></td><td class="cell"><p>83.12</p></td><td class="cell"><p>81.31</p></td><td class="cell"><p>**84.57</p></td><td class="cell"><p>*83.94</p></td><td class="cell"><p>83.43</p></td><td class="cell"><p>82.95</p></td><td class="cell"><p>82.00</p></td><td class="cell"><p>83.54</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NYTO</p></td><td class="cell"><p>78.47</p></td><td class="cell"><p>81.93</p></td><td class="cell"><p>81.22</p></td><td class="cell"><p>f82.72</p></td><td class="cell"><p>f82.55</p></td><td class="cell"><p>82.02</p></td><td class="cell"><p>82.13</p></td><td class="cell"><p>81.01</p></td><td class="cell"><p>82.53</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>NYTS</p></td><td class="cell"><p>50.80</p></td><td class="cell"><p>56.19</p></td><td class="cell"><p>55.04</p></td><td class="cell"><p>54.67</p></td><td class="cell"><p>54.26</p></td><td class="cell"><p>52.96</p></td><td class="cell"><p>55.81</p></td><td class="cell"><p>56.74</p></td><td class="cell"><p>53.82</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Reuters</p></td><td class="cell"><p>92.10</p></td><td class="cell"><p>93.12</p></td><td class="cell"><p>93.30</p></td><td class="cell"><p>93.60</p></td><td class="cell"><p>93.67</p></td><td class="cell"><p>93.60</p></td><td class="cell"><p>92.97</p></td><td class="cell"><p>93.32</p></td><td class="cell"><p>93.40</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>From the plot we observe that the maxent classi­fier assigns prediction probability greater than 0.9 to more than 1,200 test examples out of 3,000. Only 50 examples predicted by maxent fall in the lowest bin, and the rest of examples are distributed nearly uniformly across the remaining bins. The large number of examples with very high predic­tion probability explains the low entropy observed for the maximum entropy classifier.</p><p>In contrast, the CW classifier shows the oppo­site behavior after one epoch of training (the left­most blue bar), assigning low prediction probabil­ity (less than 0.3) to more than 1,200 examples and prediction probability of at least 0.9 to only 100 examples. As CW makes additional passes over the training data, its prediction confidence increases and shifts toward more peaked distribu­tions. After seven epochs fewer than 100 examples have low prediction probability and almost 1,000 have high prediction probability. Nonetheless, we note that this distribution is still less skewed than that of the maximum entropy classifier.</p><p>Given the frequency of high probability maxent predictions, it seems likely that many of the high probability maxent labels will be wrong. This is demonstrated in the third panel, which shows the contribution of each bin to the total test error. Each bar reflects the number of mistakes per bin divided by the size of the complete test set (3,000). Thus, the sum of the heights of the corresponding bars in each bin is proportional to test error. Much of the error of the maxent classifier comes not only from the low-probability bins, due to their inac­curacy, but also from the highest bin, due to its very high population. In contrast, the CW clas­sifiers see very little error contribution from the high-probability bins. As training progresses, we see again that the CW classifiers move in the direc­tion of the maxent classifier but remain essentially unimodal.</p><p>Finally, the rightmost panel shows the condi­tional test error given bin identity, or the fraction of test examples from each bin where the predic­tion was incorrect. This is the pointwise ratio be­tween corresponding values of the previous two histograms. For both methods, there is a monotonically decreasing trend in error as prediction prob­ability increases; that is, the higher the value of the prediction probability, the more likely that the prediction it provides is correct. As CW is trained, we see an increase in the conditional test error, yet the overall error decreases (not shown). This sug­gests that as CW is trained and its overall accuracy improves, there are more examples with high pre­diction probability, and the cost for this is a rela­tive increase in the conditional test error per bin. The maxent classifier produces an extremely large number of test examples with very high prediction probabilities, which yields relatively high condi­tional test error. In nearly all cases, the conditional error values for the CW classifiers are smaller than the corresponding values for maximum entropy. These observations suggest that CW assigns prob­abilities more conservatively than maxent does, and that the (fewer) high confidence predictions it makes are of a higher quality. This is a potentially valuable property, e.g., for system combination.</p></section><section number="8" title="Conclusion"><p>We have proposed a series of approximations for multi-class confidence weighted learning, where the simple analytical solutions of binary CW learning do not apply. Our best CW method out­performs online and batch baselines on eight of nine NLP tasks, and is highly scalable due to the use of a single optimization constraint. Alterna­tively, our multi-constraint algorithms provide im­proved performance for systems that can afford only a single pass through the training data, as in the true online setting. This result stands in con­trast to previously observed behaviors in non-CW settings (McDonald et al., 2004). Additionally, we found improvements in both label entropy and ac­curacy as compared to a maximum entropy clas­sifier. We plan to extend these ideas to structured problems with exponentially many labels and de­velop methods that efficiently model label correla­tions. An implementation of CW multi-class algo­rithms is available upon request from the authors.</p></section><references><p>Blitzer, I., Dredze, M., &amp; Perdra, F (2007). Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. <i>Association for Computational Linguistics (ACL).</i></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">503</doubt><page local="9" global="504"/><p>Censor, Y., &amp; Zenios, S. (1997). <i>Parallel opti­mization: Theory, algorithms, and applications. </i>Oxford University Press, New York, NY, USA.</p><p>Chang, C.-C, &amp; Lin, C.-J. (2001). <i>LIBSVM: a library for support vector machines. </i>Software available at http : / /www. csie . ntu. edu. tw/~cjlin/libsvm.</p><p>Collins, M. (2002). Discriminative training meth­ods for hidden markov models: Theory and ex­periments with perceptron algorithms. <i>Empir­ical Methods in Natural Language Processing (EMNLP).</i></p><p>Crammer, K., Dredze, M., &amp; Pereira, F. (2008). Exact confidence-weighted learning. <i>Advances in Neural Information Processing Systems 22.</i></p><p>Crammer, K., &amp; Singer, Y (2001). On the al­gorithmic implementation of multiclass kernel-based vector machines. <i>Jornal of Machine Learning Research, 2, </i>265-292.</p><p>Crammer, K., &amp; Singer, Y (2003). Ultraconserva-tive online algorithms for multiclass problems. <i>Jornal of Machine Learning Research (JMLR), 3, </i>951-991.</p><p>Dredze, M., Crammer, K., &amp; Pereira, F (2008). Confidence-weighted linear classification. <i>In­ternational Conference on Machine Learning (ICML).</i></p><p>Iusem, A., &amp; Pierro, A. D. (1987). A simultaneous iterative method for computing projections on polyhedra. <i>SIAM J. Control and Optimization, 25.</i></p><p>Lewis, D. D., Yang, Y, Rose, T. G., &amp; Li, F (2004). Rcvl: A new benchmark collection for text categorization research. <i>Journal of Machine Learning Research (JMLR), 5, </i>361-397.</p><p>Malkin, J., &amp; Bilmes, J. (2008). Ratio semi-definite classifiers. <i>IEEE Int. Conf. on Acous­tics, Speech, and Signal Processing.</i></p><p>McCallum, A. (2002). MALLET: A machine learning for language toolkit. http:// mallet.cs.umass.edu.</p><p>McDonald, R., Crammer, K., &amp; Pereira, F (2004). Large margin online learning algorithms for scalable structured classification. <i>NIPS Work­shop on Structured Outputs.</i></p><p>Sandhaus, E. (2008). The new york times an­notated corpus. Linguistic Data Consortium, Philadelphia.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">504</doubt></references></body></article>