<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="65"/><title>A Fast Accurate Two-stage Training Algorithm for L1-regularized CRFs with Heuristic Line Search Strategy</title><pubinfo>Proceedings of the 5th International Joint Conference on Natural Language Processing,pages 65-74, Chiang Mai, Thailand, November 8 - 13, 2011. ©2011 AFNLP</pubinfo><author surname="Zhou" givenname="Jinlong"><org  name="Fudan University" country="China" city="Shanghai"/></author><author surname="Qiu" givenname="Xipeng"><org  name="Fudan University" country="China" city="Shanghai"/></author><author surname="Huang" givenname="Xuanjing"><org  name="Fudan University" country="China" city="Shanghai"/></author></firstpageheader><frontmatter><p><b>A Fast Accurate Two-stage Training Algorithm for Ll-regularized CRFs with Heuristic Line Search Strategy</b></p><p><b>Jinlong Zhou</b></p><p>Fudan University Shanghai, China</p><p>abc97 0 3@gmail.com</p><p><b>Xipeng Qiu</b></p><p>xpqiu@fudan.edu.cn</p><p><b>Xuanjing Huang</b></p><p>xjhuang@fudan.edu.cn</p></frontmatter><abstract>Sparse learning framework, which is very popular in the field of nature language processing recently due to the advantages of efficiency and generalizability, can be applied to Conditional Random Fields (CRFs) with L1 regularization method. Stochastic gradient descent (SGD) method has been used in training L1-regularized CRFs, because it often requires much less training time than the batch training algo­rithm like quasi-Newton method in prac­tice. Nevertheless, SGD method some­times fails to converge to the optimum, and it can be very sensitive to the learn­ing rate parameter settings. We present a two-stage training algorithm which guar­antees the convergence, and use heuris­tic line search strategy to make the first stage of SGD training process more robust and stable. Experimental evaluations on Chinese word segmentation and name en­tity recognition tasks demonstrate that our method can produce more accurate and compact model with less training time for L1 regularization. </abstract></header><body><section title="l Introduction"><p>Conditional Random Fields (CRFs) (Lafferty et al., 2001; Sutton and McCallum, 2006) are one of the most widely-used machine learning approach in the field of nature language processing, for their ability to handle large feature sets and structural dependency between output labels. The applica­tions of CRFs cover a wide range of tasks such as part-of-speech (POS) tagging (Lafferty et al., 2001), semantic role labeling (Toutannova et al., 2005) and syntactic parsing (Finkel et al., 2008). CRFs outperform other models like Maximum En­tropy Markov models (McCallum et al., 2000), be­cause they overcome the problem of "label bias".</p><p>Moreover, CRFs can output the probabilistic of la­beling result for further use as pipeline or reranking.</p><p>For all types of CRFs, the maximum-likelihood method can be applied for parameter estimation, which means training the model is done by max­imizing the log-likelihood on the training data. To avoid overfitting the likelihood is often penal­ized with the regularization term. There were two common regularization methods named L1 and L2 regularization. L1 regularization, also called Laplace prior, penalizes the weight vector with its Ll-norm. L2 regularization, also called Gaussian prior, uses L2-form. Based on the work of Gao et al. (2007), there is no significant difference be­tween these two regularization methods in terms of accuracy. But Ll regularization has a major ad­vantage that L1-regularized training can produce models, of which the feature weights can be very sparse, then the size of the model will be much smaller than that produced by L2 regularization. Compact models are more interpretable, general-izable and manageable, require less resources like memory and storage. It is very meaningful espe­cially for the rapid development of mobile appli­cation nowadays, which suffer the scarcity of re­sources. In many NLP tasks, the feature sets can reach the magnitude of several million.</p><p>Besides, L1 regularization method can implic­itly perform the feature selection, and provide the result for further process such as iterative approach (Vail et al., 2007; Peng and McCallum, 2004). This task requires that we need to train the model as accurate as possible, as to converge to the op­timum. The feature selection can be regarded as reliable and unbiased after such a process.</p><p>Quasi-Newtion method was successfully and efficiently used in Ll-regularized model by An­drew and Gao (2007). They presented an algorithm called Orthant-Wise Limited-memory Quasi-Newton (OWL-QN), which is based on L-<page local="2" global="66"/></p><p>BFGS algorithm (Liu and Nocedal, 1989) and achieve better convergence than the method intro­duced by Kazama and Tsujii (2003).</p><p>Stochastic gradient descent (SGD) methods are another kind of L1-regularized training methods. It is a very attractive framework for it often re­quires much less training time than the batch train­ing algorithm in practice. Tsuruoka et al. (2009) presented a variant of SGD that can efficiently pro­duce compact models with L1 regularization. The main idea is to keep track of the total penalty and the penalty each weight has applied, so that the penalization smooth away the noisy gradient.</p><p>Although SGD method with cumulative penalty is very efficient, it sometimes fails to converge to the optimum, because the training process is usu­ally terminated at a certain number of iterations without explicit stop criteria as in quasi-Newton method. Another problem is that the training re­sult of SGD method is very sensitive to the param­eter settings of learning rate, therefore we have to tune the values of parameters for different tasks, which is not efficient in practice.</p><p>In this paper, we present a two-stage L1-regularized training algorithm to solve these two problems. In the first stage, we use the SGD method to get a relative good solution quickly. In the second stage, we use the OWL-QN method to improve the model which has been dealt with the SGD method. By this means we can fast get the accurate model. The learning rate scheduling in the first stage is done by heuristic line search, which makes the process more robust and stable.</p><p>Our experiments are conducted on two tasks, Chinese word segmentation and name entity recognition. We show that our method can pro­duce more accurate and compact model with less training time for L1 regularization. We also ver­ify that the result of SGD training method will be more robust when using the heuristic line search strategy.</p><p>The rest of the paper is organized as follows. Section 2 introduces the basics of CRFs. Sec­tion 3 describe the two-stage algorithm for L1-regularized models. Experimental results are shown in Section 4. We conclude the work in Sec­tion 5.</p></section><section number="2" title="Conditional Random Fields"><p>In this section, we briefly describe the basics of conditional random fields (CRFs) (Lafferty et al., 2001; Sutton and McCallum, 2006) and introduce the definition of some concepts and parameters.</p><subsection number="2.1" title="Linear-chain CRFs"><p>CRFs defines the conditional probabilistic distri­bution over possible output sequences <b>y </b>for obser­vation <b>x </b>as following:</p><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">1</doubt><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">Z(x)</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">exp</doubt><doubt alpha="60.0" length="15" tooSmall="False" monospace="0.0">j]TAfcFk(x, y)|</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(1)</doubt><p>where Fk (x, y)is equal to £ £=i <i>f</i><i>k </i>(x, <i>y</i><i>t-i</i><i>,y</i><i>t, </i><i>t). </i><i>{f</i><i>k} </i>is a set of feature function and Ak is the weight of the feature, and Z(x) is the normaliza­tion factor defined by</p><doubt alpha="60.0" length="25" tooSmall="False" monospace="0.0">X)exp&lt;YXkFk(x, y) \ yU=iJ</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(2)</doubt><p>The feature function can be divided into uni­gram features and bigram features, here we simply rewrite fk<i>(x,y</i><i>t</i><i>,t) </i>as fk<i>(x,y</i><i>t-i</i><i>,y</i><i>t</i><i>,t) </i>for conve­nient.</p></subsection><subsection number="2.2" title="Training"><p>The maximum-likelihood method is a commonly used way applied for parameter estimation, which means we train the model by minimize the negated conditional log-likelihood L(A) on the training</p><doubt alpha="80.0" length="5" tooSmall="False" monospace="0.0">data:</doubt><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">L(A)</doubt><p><i>Ylog</i><i> p(x </i><b>y</b><i>)</i></p><doubt alpha="40.0" length="5" tooSmall="False" monospace="0.0">(x,y)</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(3)</doubt><doubt alpha="52.0" length="25" tooSmall="False" monospace="0.0">Y(log Z(x)- ]TAkFk(x, y)|</doubt><p>To avoid overfitting, the likelihood is often penal­ized with the regularization term, which we will talk about in the later sections.</p><p>The partial derivative of L(A) by the feature weights A<i>k </i>are given by</p><doubt alpha="66.7" length="6" tooSmall="False" monospace="0.0">_d_dAk</doubt><doubt alpha="54.3" length="35" tooSmall="False" monospace="0.0">L=YJ2Ep(y\x)fk(x,yt-i,yt,t)(x,y)t=1</doubt><doubt alpha="48.3" length="29" tooSmall="False" monospace="0.0">Y12fk(x,yt-i,yt,t)(4)(x,y)t=i</doubt><doubt alpha="100.0" length="5" tooSmall="False" monospace="0.0">where</doubt><doubt alpha="56.5" length="23" tooSmall="False" monospace="0.0">Ep(y\x)fk(x,yt-i,yt,t)=</doubt><doubt alpha="57.9" length="19" tooSmall="False" monospace="0.0">Yfk(x,y/,y,t)P(yt-i</doubt><doubt alpha="33.3" length="6" tooSmall="False" monospace="0.0">(y',y)</doubt><doubt alpha="35.7" length="14" tooSmall="False" monospace="0.0">y/,yt= y|x)(5)</doubt><p>E<b>p(y\x) </b>denotes the conditional expectation under the model distribution:</p><page local="3" global="67"/><p>Computing the conditional expectation directly is impractical for the large number of possible tag sequences, which is exponential in the length of the observation. Thus, a dynamic program­ming approach known as the Forward-Backward algorithm originally described for Hidden Markov models (Rabiner, 1989), is applied in a slightly modified form. For the forward recursions, we have</p><doubt alpha="33.3" length="9" tooSmall="False" monospace="0.0">ao(-L)= 1</doubt><p>at<b>+i</b>(y) = Ey' at(yOexp{£ f<b>=i </b>Ak fk <b>(x</b>,y/,y,t)} and for the backward recursion, we have for 0 &lt; t &lt; T and y G Y, where L_ and T are de­fined as special states for the begin and end of the sequence. Then the normalization factor is com­puted by and the conditional probabilities P(yt-i = y/,yt = y<b>|x) </b>are given by</p><doubt alpha="40.0" length="10" tooSmall="False" monospace="0.0">ßT+i(T)= 1</doubt><doubt alpha="59.1" length="22" tooSmall="False" monospace="0.0">ßt(y/) =Eyßt+i(y)exp{E</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">K</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">k=i</doubt><doubt alpha="57.1" length="14" tooSmall="False" monospace="0.0">Akfk(x,y/,y,t)</doubt><doubt alpha="31.2" length="16" tooSmall="False" monospace="0.0">Z(x)=ßo(-L), (6)</doubt><doubt alpha="55.8" length="43" tooSmall="False" monospace="0.0">at(y/)expjYAkfk(x,y/,y,t) ) ßt+i(y)/Z(x)k=i</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">3</doubt><p>The logarithmic loss function L(A) defined by (3) is usually penalized with an additional regulariza-tion term, which prevents the model from overfit-ting the training data. There are two common reg-ularization methods named L1 and L2 regulariza-tion, in the case of L1 regularization, the term is defined as:</p><doubt alpha="46.2" length="13" tooSmall="False" monospace="0.0">R(A)=C]T|Ak|,</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(7)</doubt><p>where C is the regularization parameter that con­trols the trade-off between fitting exactly the ob­servations and the L1-norm of the weight vector. This value is usually tuned by cross-validation or using the heldout data.</p><p>Now we can redefine the objective loss function</p><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">as</doubt></subsection><subsection number="3.2" title="Orthant-Wise Limited-memory Quasi-Newton"><p>It is not easy to use some common numerical opti­mization strategies such as limited memory BFGS (Liu and Nocedal, 1989) directly with L1 regular-ization, because the regularization term is not dif-ferentiable when the weight is zero.</p><p>A very efficient strategy called Orthant-Wise Limited-memory Quasi-Newton (OWL-QN) method is introduced in (Andrew and Gao, 2007). This algorithm is motivated by the observation that the L1 regularization term is differentiable when restricted to a set of points in which each coor­dinate never changes sign (called its "orthant"). Furthermore it is a linear function of its argu­ment, which means the second-order behavior of the regularized objective function on a given or-thant is determined by the log-likelihood compo­nent alone. Only a few steps of the standard L-BFGS algorithm have been changed in the OWL-QN method, and these differences are listed be­low:</p><p>1. The "pseudo-gradient" is used in place of the gradient.</p><p>2. The resulting search direction is constrained to match the sign pattern of the negated pseudo-gradient.</p><p>3. Each parameter is projected back onto the ini­tial orthant of the previous value during the line search.</p><p>Andrew and Gao (2007) proved that OWL-QN method is guaranteed to converge to a globally op­timal result.</p></subsection><subsection number="3.3" title="Stochastic Gradient Descent"><p>Stochastic gradient approaches use a small batch of the observations to get a crude approximation of the gradient of the objective function given by (3). The small batch size makes us possible to update the parameters more frequently than the origin gradient descent and speed up the conver­gence. Only considering the log-likelihood term, the updates have the following form</p><doubt alpha="60.0" length="5" tooSmall="False" monospace="0.0">dL(A)</doubt><doubt alpha="75.0" length="4" tooSmall="False" monospace="0.0">A=Aj</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(9)</doubt><p>L(A) + R(A).</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(8)</doubt><p>where <i>j </i>is the iteration counter and <i>r</i><i>\j </i>is the learn­ing rate. It should be noted that the partial deriva­tive we presented here is not the true gradient <b>L1 Regularization in CRFs</b></p></subsection><subsection number="3.1" title="Regularization"><page local="4" global="68"/><p>but the crude approximation from small randomly-selected subset of the training samples. In (Tsu-ruoka et al., 2009), a variant of SGD that can effi­ciently train L1-regularized CRFs was presented. The main idea can be concluded as follows:</p><p>1. Only update the weights of the features that are used in the current observation, called "lazy update".</p><p>2. "Clip" the parameter value when it crosses zero.</p><p>3. Keep track of the cumulated penalty that the weights of features should been received if the fluctuationless gradient were used, and use this value to the update.</p><p>Let <i>Z</i><i>j</i><i> </i>be the total Ll-penalty that each weight should been received, it is simply accumulated as:</p><doubt alpha="50.0" length="12" tooSmall="False" monospace="0.0">zj=CEvt-(10)</doubt><p>Then the process of regularization can be formal­ized as follows where <i>q<footnote anchor="3"/>k</i><i> </i>is the total Ll-penalty that Ak has actu­ally received:</p><doubt alpha="50.9" length="55" tooSmall="False" monospace="0.0">Aj+i=fmax(0,\k-(zj+qk-i)Ak&gt;0k\min(0,Ak+(zj - qk-i)Ak &lt;0</doubt><doubt alpha="52.6" length="19" tooSmall="False" monospace="0.0">qk=E(Ak+i- Ak)-(ii)</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">t=i</doubt><p>Tsuruoka et al. (2009) demonstrated that this al­gorithm can be much more quickly than the OWL-QN method and yield a comparable performance, while the value of objective function and the num­ber of active features are not as good as OWL-QN. The reason is that we usually terminated the train­ing process at a certain number of iterations, be­cause there are no explicit stop criteria for SGD.</p><p>Another issue is that the scheduling of learning rates can be very tricky. Tsuruoka et al. (2009) suggest that exponential decay is a good choice in practice compared with the method used in (Collins et al., 2008). This kind of scheduling of learning rates have the following form:</p><doubt alpha="50.0" length="14" tooSmall="False" monospace="0.0">Vj=noaj/N,(12)</doubt><p>where n<b>o </b>and a are both constant. We name n<b>o </b>the initiation learning rate parameter and a the de­scent learning rate parameter. These learning rate parameters have a great influence on the result of SGD training, and they need to be tuned for differ­ent tasks, which is not very efficient in practice.</p></subsection><subsection number="3.4" title="Two-stage L1-regularized Training"><p>Based on what we have mentioned above, we know that the SGD method sometimes fails to converge to the global optimal solution, for it does not have the explicit stop criteria as in the quasi-Newton method. Although the scheduling of learning rate found in (Collins et al., 2008):</p><doubt alpha="42.9" length="14" tooSmall="False" monospace="0.0">Vj= ÎT7/NV(13)</doubt><p>guarantees ultimate convergence theoretically. Its actual convergence speed is poor in practice (Darken and Moody,1990). We have to take quite a number of iterations if we need the result close enough to the best solution. This contradicts to the main motivation we use SGD method for pa­rameter estimation that can speed up the training process.</p><p>On the other hand, based on the work ofAndrew guarantees the convergence. And we can test the relative change in the objective function value av­eraged over the several previous iterations for stop criteria.</p><doubt alpha="64.3" length="42" tooSmall="False" monospace="0.0">and Gao (2007), we know that OWL-QN method</doubt><p>Tsuruoka et al. (2009) demonstrated that SGD method converges much faster than OWL-QN method especially in the first few iterations. This fact motivates us to use a two-stage training strat­egy. In the first stage, we use the SGD method to quickly get a relative good solution. In the second stage, we use the OWL-QN method to improve the model which has been dealt with the SGD method.</p><p>This method can be also driven from an alter­native view. In the theory of convex optimization (Boyd and Vandenberghe, 2004), the asymptotic convergence rate of Newton's method is quadratic if we start at a point close enough to the global op-timum.<footnote anchor="1"/> In fact, the iterations in Newton's method can fall into two stages. The second stage, which occurs once the searching point is quite close to the optimum solution, is called "quadratically con­vergent stage". The first stage is usually referred as the "damped Newton phase", because the algo­rithm may choose a step size that is different from the exact Newton step to satisfy the backtracking condition.<page local="5" global="69"/><footnote anchor="2"/> The quadratically convergent stage is also called the "pure Newton phase" since the full Newton step is always chosen in these iterations . This fact demonstrates that if we can find a solu­tion close to the global optimum, it will not only increase the average convergence rates, but also reduce the time consuming needed for backtrack­ing line search. This is what we achieved by the first stage of SGD training method.</p><footnote label="1">Quasi-Newton method shares many properties with Newton's method, though its convergence rate is generally superlinear but not quadratic.</footnote></subsection><subsection number="3.5" title="Heuristic Line Search"><p>Another problem of SGD method is the trouble­some learning rate parameters tuning, and these parameters have a significant influence on the re­sult of SGD training. No matter which way to set the learning rate, if it is fixed without taking the actual effect of the current training sample update into consideration, it may be too large or too small for some situation. In order to get a more robust and stable method for learning rate scheduling, we present a heuristic line search strategy inspired by the implementation of CRFsgd (Bottou, 2007) for learning rate calibration.</p><p>For the purpose of convenience, we define the objective function for a single sample as</p><doubt alpha="37.5" length="32" tooSmall="False" monospace="0.0">1(A,x)= - logp(x, y)+CE|Ak|-(14)</doubt><doubt alpha="100.0" length="7" tooSmall="False" monospace="0.0">JVAfcex</doubt><p>Notice here we only use these active features in the current sample as the L1 regularization term, for we only update these associate parameters in a lazy fashion.</p><p>Now we try to find the learning rate that de­crease the value of this objective function as much as possible without consuming too much search time. We simply use a heuristic line search strat­egy as follows: (1) We use the learning rate cal­culated by Eq.12 as initiation and get the initial value of Eq.14. (2) Then we go on to increase the learning rate until the maximum number of tri­als for search is reached or the value of Eq.14 is worse than the initial value, we just decrease the learning rate from the initiation if the latter situ­ation happens. (3) At last we just use the learn­ing rate that yields the best result of Eq.14 dur­ing the search. For the calculation of Eq.14 only needs the weights of the features that are used in the current sample, so it will still be very efficient. The whole algorithm in pseudo-code was showed in Algorithm 1.</p><footnote label="2">To ensure the objective function decrease a certain value and guarantee the convergence.</footnote><p><b>Algorithm 1 </b>SGD heuristic line search</p></subsection></section><section number="1." title="for k = 0 to MaxIterations"></section><section number="2." title="Select sample j"></section><section number="3." title="bestr &lt; — LearningRate (k)"></section><section number="4." title="UpdateWeights (j ,bestr)"></section><section number="5." title="procedure LearningRate (k)"><doubt alpha="51.9" length="27" tooSmall="False" monospace="0.0">6.r(0)&lt;— initialed by Eq.12</doubt><doubt alpha="64.5" length="31" tooSmall="False" monospace="0.0">7.init.obj&lt;— initialed by Eq.14</doubt></section><section number="8." title="for i = 0 to MaxTrialTime"></section><section number="9." title="UpdateWeights (j ,r(i))"><doubt alpha="35.3" length="17" tooSmall="False" monospace="0.0">10.obj(i)&lt;- Eq.14</doubt></section><section number="11." title="Recover weights before update"></section><section number="12." title="if obj(i) is worse than init.obj then"></section><section number="13." title="label flag"></section><section number="14." title="if flag status is changed then"><doubt alpha="36.4" length="22" tooSmall="False" monospace="0.0">15.r(i+ 1)&lt;—r(0)*decay</doubt><doubt alpha="57.1" length="7" tooSmall="False" monospace="0.0">16.else</doubt></section><section number="17." title="if flag is not set then"><doubt alpha="40.9" length="22" tooSmall="False" monospace="0.0">18.r(i+ 1)&lt;—r(i)/decay</doubt><doubt alpha="57.1" length="7" tooSmall="False" monospace="0.0">19.else</doubt><doubt alpha="40.9" length="22" tooSmall="False" monospace="0.0">20.r(i+ 1)&lt;—r(i)*decay</doubt><doubt alpha="65.4" length="26" tooSmall="False" monospace="0.0">21.bestr=argminr(i)obj (i)</doubt><p>22. <b>return </b>bestr.</p><p>It should be noted that we need not set a large number for maximum trial time, because it will generally take a lot of search time and may not yield a good result but arrives at the local opti­mum, for we only optimize the objective value of a single training sample. Here we set the value to 3 empirically. The changing rate for line search can be any positive number that smaller than 1, it is set to 0.5 as mostly accepted.</p></section><section number="4" title="Experiments"><p>We evaluate the effectiveness and performance of our training algorithm using two NLP tasks that includes Chinese words segmentation and name entity recognition, which are very typical prob­lems in the field of NLP.</p><p>To show the improvement of our algorithm, we compare it with the OWL-QN algorithm and SGD algorithm on the same data sets. For the pur­pose of run-times comparison, we implemented all the algorithm in a quite similar way, especially in feature extraction and gradient computation. For example, we compute the forward/backward scores in logarithm domain instead of scaling method, though the latter method was claimed faster (Lavergne et al., 2010).<page local="6" global="70"/> All experiments were performed on a server with Xeon 2.66GHz.</p><table caption="Table 1: Feature templates for Chinese word seg­mentation task."></table><doubt alpha="57.1" length="21" tooSmall="False" monospace="0.0">(1)ci-iy»,ciyi,ci+iy»</doubt><doubt alpha="50.0" length="32" tooSmall="False" monospace="0.0">(2)c»-iCiy»,CiCi+iy», c»-ic»+iy»</doubt><doubt alpha="33.3" length="9" tooSmall="False" monospace="0.0">(3)y»-iy_</doubt><subsection number="4.1" title="Chinese Word Segmentation"><p>The first set of experiments used the Chinese word segmentation corpus from the Second In­ternational SIGHAN Bakeoff data sets (Emerson, 2005), provided by Peking University. The train­ing data consists of 19,054 sentences, 1,109,947 Chinese words, 1,826,448 Chinese characters and the testing data consists of 1,944 sentences, 104,372 Chinese words, 172,733 Chinese charac­ters. We separated 1,000 sentences from the train­ing data and use them as the heldout data. The test data was only used for the final accuracy report.</p><p>The feature templates we used in this experi­ment were listed in Table 1, where ci denotes the <i>ith </i>Chinese character in an instance, yi denotes the ith label in the instance. Based on the work of Huang and Zhao (2007), it was shown that 6 label representation is a better choice in practice. Compare with the origin 2 label representation or 4 label representation, it can represent richer label information. We did not use any extra knowledge such as Chinese and Arabic numbers.</p><p>For OWL-QN method and SGD method, we followed the experiment settings in (Tsuruoka et method were the same with the default settings of the optimizer developed by Andrew and Gao (2007), the convergence tolerance was 1e-4; the L-BFGS memory parameter was 10. The regular-ization parameter C was tuned in the way that it maximized the log-likelihood of the heldout data when using the OWL-QN algorithm. We also used this value as the regularization parameter in the SGD method. The learning rate parameters for SGD were tuned in the way that they maximized the value of the objective function in 30 passes. We first set the initiation learning rate parameter (no) by testing 1.0, 0.5, 0.2, and 0.1, then we set the descent learning rate parameter (a) by testing 0.9, 0.85, and 0.8 with the fixed initiation learning rate parameter.</p><doubt alpha="64.3" length="42" tooSmall="False" monospace="0.0">al., 2009). The meta-parameters for OWL-QN</doubt><doubt alpha="14.3" length="7" tooSmall="False" monospace="0.0">50000J—</doubt><doubt alpha="0.0" length="15" tooSmall="False" monospace="0.0">1 6 11 16 21 26</doubt><doubt alpha="46.7" length="15" tooSmall="False" monospace="0.0">Passes (a=0.85)</doubt><figure caption="Figure 2: Bakeoff 2005 Chinese word segmenta­tion task: Objective function with fixedn0."></figure><doubt alpha="0.0" length="22" tooSmall="False" monospace="0.0">1 6 11 16        21 26</doubt><doubt alpha="57.1" length="14" tooSmall="False" monospace="0.0">Passes(n0=l-0)</doubt><p>For our method, we first measured the progress of the SGD algorithm with heuristic line search we presented against the origin SGD method. We use the same parameters settings with the for­mer method including both regularization param­eter and learning rate parameters. The number of passes performed over the training data was also set to 30. Then we compare the results of both methods during the training process of the model with the same parameters, and they were shown in Figure 1 and Figure 2.</p><p>Figure 1 shows how the value of the objective function changed as the training proceeded with the same descent learning rate parameter (a = 0.85), the figure contains six curves represent­ing the results of SGD method with heuristic line search and the origin SGD method with differ­ent initiation learning rate parameter settings (n0). "HLS" stands for the heuristic line search strategy. The results shows SGD method with heuristic line search shows better convergence and more robust result than the origin SGD method when using the same learning rate parameter settings.<page local="7" global="71"/> Figure 2 shows the results with different settings of learn­ing rate parameters (fixed n0 = 1), and it demon­strates the same trend as Figure 1.</p><table caption="Table 2: Bakeoff2005 Chinese word segmentation task. Accuracy of the model on the testdata."></table><p>Then we trained the models with the training data and evaluated the accuracy of the Chinese word segmenter on the test data. The number of passes performed over the training data in SGD was also set to 30. In our method, we set the SGD iteration times to 5. It is worth noting that we didn't spend much time in tuning the value of this parameter. Based on a cursory view of the train­ing process, we found that it converge to a relative "good" result after the first 5 iteration. We used this value throughout all the experiments. Because we would take the OWL-QN method to guarantee the final convergence, and the SGD method with heuristic line search strategy is insensitive to the learning rate parameters, this value would not have a significant influence on the performance.</p><p>The results are shown in Table 2 and Table 3. In Table 2, the second column shows the final value of the objective function. The third column shows the number of active features in the final result­ing model. The fourth column shows the F score of the Chinese word segment results, which is the harmonic mean of precision P (percentage of out­put Chinese words that exactly match the golden standard Chinese words) and recall R (percentage of golden standard Chinese words that returned by our system). In Table 3, the second column shows the number of passes performed in the training, in our method, this value includes the the number of passes both in the first stage of SGD and the sec­ond stage of OWL-QN process. The third column shows the training time.</p><table caption="Table 4: Feature templates for name entity recog­nition task."></table><doubt alpha="62.9" length="35" tooSmall="False" monospace="0.0">(1)Ci-2yi,Ci-iyi,Ciyi,Ci+iyi,Ci+2yi</doubt><doubt alpha="65.0" length="20" tooSmall="False" monospace="0.0">(2)Ci-1Ciyi,CiCi+iyi</doubt><doubt alpha="50.0" length="10" tooSmall="False" monospace="0.0">(3)yi-iyi_</doubt><p>In the terms of accuracy, there was no signif­icant difference between all the models, the ori­gin SGD method yield the slightly better result, probably due to the model has larger features sets. This doesn't contradict to our original purpose, for we have got a substantial improved result in both the final value of the objective function and the number of active features compared with the ori­gin SGD method, and to the same level as OWL-QN method. Notice the origin feature sets are over 6 millon, L1 regularization methods produced the models which are compact indeed. The official best result in the closed test achieved an F score of 95.00, and our result is quite close to that, ranked 4th of 23 official runs.</p><p>On the other hand, our method took about 30% less than the OWL-QN method in the training time. Our method only needs 88 passes over the whole training data in the second stage for convergence compared with 141 in the OWL-QN method, which shows a significant improvement in training time consuming, for we have used the first stage of SGD method to get a nearly optimal and stable result beforehand.</p></subsection><subsection number="4.2" title="Name Entity Recognition"><p>The second set of experiments used the name en­tity recognition corpus from the Fourth Interna­tional SIGHAN Bakeoff data sets (Jin and Chen, 2008), provided by Microsoft Research Asia. The training data consists of 23,182 sentences, 1,089,050 Chinese characters and the testing data consists of 4,636 sentences, 219,197 Chinese char­acters. We separated 1,000 sentences from the training data and use them as the heldout data. The training data is annotated with the "IOB" tags rep­resenting name entities including person, location and organization.</p><p>The feature templates we used in this experi­ment were listed in Table 4. Notice we did not change the label representation made by the origin<page local="8" global="72"/></p><table caption="Table 3: Bakeoff2005 Chinese word segmentation task. Training time of the model on the testdata." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>L + R</p></td><td class="cell"><p># Features</p></td><td class="cell"><p>F score</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>OWL-QN</p><p>SGD</p><p>Ours</p></td><td class="cell"><p>56451.8 61398.6 56481.9</p></td><td class="cell"><p>114,942 232,585 117,374</p></td><td class="cell"><p>94.81 94.92 94.78</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Passes</p></td><td class="cell"><p>Time</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>OWL-QN</p><p>SGD</p><p>Ours</p></td><td class="cell"><p>141</p><p>30 5 + 88</p></td><td class="cell"><p>2h59min</p><p>58min 2h06min</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><doubt alpha="0.0" length="15" tooSmall="False" monospace="0.0">1 6 11 16 21 26</doubt><doubt alpha="46.7" length="15" tooSmall="False" monospace="0.0">Passes (a=0.85)</doubt><doubt alpha="57.1" length="14" tooSmall="False" monospace="0.0">Passes(n0=l-0)</doubt><p>Table 5: Fourth SIGHAN Bakeoff name entity recognition task. Training time of the model on the testdata.</p><p>recognition task. Accuracy of the model on the testdata.</p><p>training data for convenient. Again a richer label representation may yield a better performance.</p><p>The other experiment settings are the same with the experiment on Chinese word segmentation. The comparison results are shown in Table 5, Fig­ure 3 and Figure 4. The trend in the results is the same as that of the Chinese word segmenta­tion task. SGD method with heuristic line search strategy produced more stable and robust result than the origin SGD method. Although there will have fluctuations sometimes (in Figure 4), the line search strategy shows the ability to find an appre­ciate step size in that case. Again our method con­verged to a much better solution against SGD in both the final value of the objective function and number of active features, and took about 40% less training time than OWL-QN.</p><p>The accuracy of the results is shown in Table 6, there was no significant difference between all the models as well. The F score of organization name entity recognition was worse than the results in person and location name entity, for organiza­tion name entities in Chinese often have a relative long distance dependency, which is not easy to be captured by our local feature templates in the Chi­nese character level.</p></subsection></section><section number="5" title="Conclusion"><p>We have presented a two-stage algorithm that can efficiently train L1-regularized CRFs. Experi­ments on two NLP tasks demonstrated that our method is effective and efficient by utilizing both the advantages of SGD and OWL-QN.</p><figure caption="Figure 3: Fourth SIGHAN Bakeoff name entity recognition task: Objective function with fixeda."></figure><doubt alpha="14.3" length="7" tooSmall="False" monospace="0.0">20000J—</doubt><doubt alpha="0.0" length="29" tooSmall="False" monospace="0.0">1 6 11        16        21 26</doubt><figure caption="Figure 4: Fourth SIGHAN Bakeoff name entity recognition task: Objective function with fixedn0."></figure><p>In the future, we intend to study how to use the results of the first stage of SGD learning to estimate the Hessian information, which can be provided for the second stage of quasi-Newton method to enhance the effectiveness of training. Borders et al. (2009) looked into this problem in a similar way. It is also worthwhile to investi­gate whether other adaptive learning rate schedul­ing algorithms can result in fast training with our method, as in (Vishwanathan et al., 2006; Huang</p><doubt alpha="65.5" length="29" tooSmall="False" monospace="0.0">et al., 2007).Acknowledgments</doubt><p>The author wishes to thank the anonymous re­viewer for their helpful suggestions. This work<page local="9" global="73"/></p><doubt alpha="59.1" length="44" tooSmall="False" monospace="0.0">was (partially) funded by NSFC (No. 61003091</doubt><doubt alpha="58.3" length="84" tooSmall="False" monospace="0.0">and No. 61073069) and Shanghai Committee of Science and Technology (No. 10511500703)</doubt><table caption="Table 6: Fourth SIGHAN Bakeoff name entity" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>L + R</p></td><td class="cell"><p>Passes</p></td><td class="cell"><p>Time</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>OWL-QN</p><p>SGD</p><p>Ours</p></td><td class="cell"><p>11247.1 13993.3 11245.5</p></td><td class="cell"><p>219 30</p><p>5 + 122</p></td><td class="cell"><p>5h26min 1h08min 3h10min</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p># Feat.</p></td><td class="cell"><p>LOC</p></td><td class="cell"><p>ORG</p></td><td class="cell"><p>PER</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>OWL-QN</p></td><td class="cell"><p>34,579</p></td><td class="cell"><p>89.94</p></td><td class="cell"><p>82.61</p></td><td class="cell"><p>90.65</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SGD</p></td><td class="cell"><p>113,005</p></td><td class="cell"><p>89.39</p></td><td class="cell"><p>82.75</p></td><td class="cell"><p>90.78</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Ours</p></td><td class="cell"><p>36,709</p></td><td class="cell"><p>90.05</p></td><td class="cell"><p>82.25</p></td><td class="cell"><p>90.49</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><references><p>Galen Andrew and Jianfeng Gao. 2007. Scalable train­ing of l1-regularized log-linear models. In <i>Proceed­ings of the International Conference on Machine Learning, </i>pages 33-40. Corvalis, Oregon, USA.</p><p>Antoine Bordes, Leon Bottou, and Patrick Gallinari.</p><p>2009. SGD-QN: Careful quasi-Newton stochastic gradient descent. In <i>The Journal of Machine Learn­ing Research, </i>10: 1737-1754.</p><p>Leon Bottou. 2007. Stochastic gradient descent (sgd) implementation. http://leon.bottou.org/projects/sgd.</p><p>Stephen Boyed and Lieven Vandenberghe. 2004. <i>Con­vex Optimiaztion. </i>Cambridge University Press.</p><p>Michael Collins, Amir Globerson, Terry Koo, Xavier Carreras, and Peter L. Bartlett. 2008. Exponentiated gradient algorithms for conditional random fields and max-margin markov networks. In <i>The Journal of Machine Learning Research, </i>9: 1775-1822.</p><p>Christian Darken and John Moody. 1990. Note on learning rate schedules for stochastic optimization.</p><p>In <i>Proceedings ofAdvances in Neural Information Processing Systems 3, </i>pages 832-838. Colorado,</p><p>USA.</p><p>Tom Emerson. 2005. The second international Chi­nese word segmentation bakeoff. In <i>Proceedings of Fourth SIGHAN Workshop on Chinese Language Processing. </i>Korea.</p><p>Jenny Rose Finkel, Alex Kleeman, and Christopher D.Manning. 2008. Efficient, feature-based, con­ditional random field parsing. In <i>Proceedings of the Annual Meeting ofthe Association for Computa­tional Linguistics, </i>pages 959-967. Columbus, Ohio,</p><p>Jianfeng Gao, Galen Andrew, Mark Johnson, and Kristina Toutanova. 2007. A comparative study of parameter estimation methods for statistical natural language processing. In <i>Proceedings ofthe Annual Meeting ofthe Association for Computational Lin­guistics, </i>pages 824-831. Prague, Czech republic.</p><p>Changning Huang and Hai Zhao. 2007. Chinese word segmentation: A decade review. In <i>Journal ofChi-nese Information Processing, </i>21(3): 8-19.</p><p>Han-Shen Huang, Yu-Ming Chang, and Chun-Nan Hsu. 2007. Training conditional random fields by periodic step size adaptation for large-scale text min­ing. In <i>Proceedings ofthe IEEE International Con­ference on Data Mining, </i>pages 511-516. Omaha,</p><p>Nebraska, USA.</p><p>Guangjin Jin and Xiao Chen. 2008. The fourth inter­national Chinese language processing bakeoff: Chi­nese word segmentation, named entity recognition and Chinese pos tagging. In <i>Proceedings ofSixth SIGHAN Workshop on Chinese Language Process­ing. </i>India.</p><p>Junichi Kazama and Junichi Tsujii. 2003. Evalua­tion and extension of maximum entropy models with inequality constraints. In <i>Proceedings ofthe Con­ference on Empirical Methods in Natural Language Processing, </i>pages 137-144.</p><p>John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: prob­abilistic models for segmenting and labeling se­quence data. In <i>Proceedings ofthe International Conference on Machine Learning, </i>pages 282-289. Prague, Czech republic.</p><p>Thomas Lavergne, Olivier,Cappe, and Francois Yvon. 2010. Practical very large scale CRFs. In <i>Pro­ceedings the Annual Meeting ofthe Association for Computational Linguistics, </i>pages 504-513. Upp­sala, Sweden.</p><p>Dong C. Liu and Jorge Nocedal. 1989. On the limited memory BFGS method for large scale optimization.</p><p><i>Mathematical Programming, </i>45: 503-528.</p><p>Andrew Mccallum, Dayne Freitag, and Fernando Pereira. 2000. Maximum Entropy Markov Mod­els for Information Extraction and Segmentation. In <i>Proceedings ofthe International Conference on Ma­chine Learning, </i>pages 591-598. California, USA.</p><p>Naoaki Okazaki. 2007. CRFsuite: A fast im­plementation of conditional random fields (CRFs). http://www.chokkan.org/software/crfsuite/.</p><p>Fuchun Peng, and Andrew McCallum. 2004. Accurate Information Extraction from Research Papers us­ing Conditional Random Fields. In <i>Proceedings of the North American Chapter ofthe Association for Computational Linguistics, </i>pages 329-336. Mas­sachusetts, USA.</p><p>Lawrence Rabiner. 1989. A Tutorial on Hid­den Markov Models and Selected Applications in Speech Recognition. In <i>Proceedings ofthe IEEE,</i> 77(2): 257-286.</p><p>Charles Sutton and Andrew McCallum. 2006. An introduction to conditional random fields for rela­tional learning. <i>Introduction to Statistical Rela­tional Learning. </i>The MIT Press.</p><p>Kristina Toutanova, Aria Haghighi, and Christopher Manning. 2005. Joint learning improves semantic role labeling. In <i>Proceedings ofthe Annual Meet­ing </i><i>ofthe</i><i> Association for Computational Linguis­tics, </i>pages 589-596. Michigan, USA.</p><p>Yoshimasa Tsuruoka, Junichi Tsujii, and Sophia Ana-niadou. 2009. Stochastic gradient descent train­ing for l1-regularized log-linear models with cumu­lative penalty. In <i>Proceedings the Annual Meeting of the Association for Computational Linguistics, </i>pages 477-485. Suntec, Singapore.</p><p>Douglas Vail, John Lafferty, and Manuela Veloso. 2007. Feature Selection in Conditional Random Fields for Activity Recognition. In <i>Proceedings of</i> <i>the IEEE/RSJ International Conference on Intelli­gent Robots and Systems, </i>pages 3379-3384.<page local="10" global="74"/> Cali­fornia, USA.</p><p>S. V. N. Vishwanathan, Nicol N. Schraudolph, MarkW. Schmidt, and Kevin P. Murphy. 2006. Accelerated training of conditional random fields with stochastic gradient methods. In <i>Proceedings ofthe Interna­tional Conference on Machine learning, </i>pages 969­976. Pittsburgh, Pennsylvania, USA.</p></references></body></article>