<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="835"/><title>Learning Decision Lists with Known Rules for Text Mining</title><author surname="Chakravarthy" givenname="Venkatesan"><org  name="IBM India Research Lab" country="India" city="Bangalore"/></author><author surname="Joshi" givenname="Sachindra"><org  name="IBM India Research Lab" country="India" city="Bangalore"/></author><author surname="Ramakrishnan" givenname="Ganesh"><org  name="IBM India Research Lab" country="India" city="Bangalore"/></author><author surname="Godbole" givenname="Shantanu"><org  name="IBM India Research Lab" country="India" city="Bangalore"/></author><author surname="Balakrishnan" givenname="Sreeram"><org  name="IBM India Research Lab" country="India" city="Bangalore"/></author></firstpageheader><frontmatter><p><b>Learning Decision Lists with Known Rules for Text Mining</b></p><p><b>Venkatesan Chakravarthy Sachindra Joshi Ganesh Ramakrishnan</b></p><p>IBM India Research Lab        IBM India Research Lab        IBM India Research Lab</p><p>vechakra@in.ibm.com jsachind@in.ibm.com ganramkr@in.ibm.com</p><p><b>Shantanu Godbole Sreeram Balakrishnan</b></p><p>IBM India Research Lab IBM Silicon Valley Lab</p><p>shgodbol@in.ibm.com sreevb@us.ibm.com</p></frontmatter><abstract>Many real-world systems for handling unstructured text data are rule-based. Examples of such systems are named entity annotators, information extraction systems, and text classifiers. In each of these appli­cations, ordering rules into a decision list is an im­portant issue. In this paper, we assume that a set of rules is given and study the problem (MaxDL) of or­dering them into an optimal decision list with respect to a given training set. We formalize this problem and show that it is NP-Hard and cannot be approxi­mated within any reasonable factors. We then propose some heuristic algorithms and conduct exhaustive ex­periments to evaluate their performance. In our ex­periments we also observe performance improvement over an existing decision list learning algorithm, by merely re-ordering the rules output by it. </abstract></header><body><section number="1" title="Introduction"><p>Rule-based systems have been extensively used for several problems in text mining. Some problems in text mining where rule-based systems have been successfully used are part of speech tagging (Brill, 1992), named entity annotation (Grishman, 1997; Appelt et al., 1995), information extraction (May-nard et al., 2001), question answering (Riloff and Thelen, 2000) and classification (Han et al., 2003; Li and Yamanishi, 1999; Sasaki and Kita, 1998). Sev­eral studies have been conducted that compare the performance of rule-based systems and other ma­chine learning techniques with mixed results. While there is no clear winner between the two approaches in terms of performance, the rule-based approach is clearly preferred in operational settings (Borth-wick, 1999; Varadarajan et al., 2002). Rule-based systems are human comprehensible and can be im­proved over time. Therefore, it is imperative to de­velop methods that assist in building rule-based sys­tems.</p><p>A rule-based system consists of a set of rules. These rules can either be manually designed or could be learnt from a training set using rule-induction techniques (J. and G, 1994; Cohen, 1995). Each rule consists of an <i>antecedent </i>or <i>pattern </i>and a <i>consequent </i>or <i>predicted annotation. </i>In this paper, we will restrict our attention to a broad class of rules in which the antecedent describes a series of condi­tions on the input item and the consequent specifies the label that applies to instances <i>covered </i>by the an­tecedent. The conditions could also be expressed as patterns in regular or more powerful grammars.</p><p>In general, rules could be ambiguous, <i>i.e., </i>multi­ple rules could <i>cover </i>an instance. A common ap­proach for resolving this ambiguity is to define an ordering on the rules (Maynard et al., 2001; Borth-wick, 1999). A decision list is one such mecha­nism (Rivest, 1987). A set of rules that are intended to be interpreted in a sequence is called a <i>decision list. </i>In other words, a decision list is an ordering of the given set of rules. Given an instance t, the rules are applied in the specified order until a pattern of a rule R covers t. The instance t is assigned the pre­dicted annotation associated with R.</p><p>In this paper, we study the problem of arranging a given set of rules into the "best" decision list. Learn­ing decision lists using training data has been stud­ied in the past (Rivest, 1987; J. and G, 1994; Cohen, 1995; Li and Yamanishi, 1999). These methods at­tempt to simultaneously learn rules and their order­ing. Typically they use <i>separate and conquer </i>(Wit­ten and Frank, 2005) strategy and order generated rules as they are discovered. The generation and or­dering of rules are not considered as two separate tasks.<page local="2" global="836"/> In contrast, we assume that the rules are given to us and study the problem of arranging them into an optimal decision list, where optimality is deter­mined over a training data set. Our approach is mo­tivated by the observation that in many operational settings, it is easier and preferred to get a set of rules designed by domain experts (Lewis et al., 2003). Al­ternatively, the set of rules can be determined using existing techniques for rule learning (J. and G, 1994; Cohen, 1995; Califf and Mooney, 1998). The sepa­ration of rule ordering from rule generation allows us to analyze the problem of ordering in detail and to develop <i>effective </i>methods for rule ordering. We demonstrate the usefulness of the proposed methods for ordering manually designed rules in the task of named entity annotation and machine learnt rules in the task of classification.</p><p>We determine the ordering of the given set of rules based on a training set. A training set consists of a set of pairs (tj,aj) where <i>tj</i><i> </i>is an instance and ajis its actual annotation. Given a set of rules and a training data set, we define the problem as follows: <i>Arrange the rules into a decision list such that max­imum number of instances are assigned the correct annotation. </i>We refer to this problem as the M axDL problem. We show that this problem is NP hard and cannot approximated within a factor of n1-e, for any e &gt; 0. We then propose some heuristics and present an experimental study of these heuris­tics. Our experimental results show performance im­provement over an existing decision list learning al­gorithm, by merely reordering the rules output by that algorithm. We also illustrate the performance improvements obtained by applying our algorithms for ordering named entity annotation rules and clas­sification rules.</p><p>In the rest of the paper we formalize the MaxDL problem (§2), show it is NP-hard and can't be approximated within reasonable factors (§3), and propose heuristics in a greedy framework (§4). We present experiments (§5) and conclude with Section§6.</p></section><section number="2" title="MaxDL Problem Definition and Notations"><p>The input consists of a set of <i>instances T </i>= (t1; t2,..., tm}, a set of annotations <b>A </b>and a set of rules <b>R </b>= (R1; R2,..., Rn}. Each rule Rj = (p, a) is a pair, where p is called the <i>pattern </i>and a <b>e A </b>is called the <i>predicted annotation. </i>The patten p will be given as a set p <b>C I</b>; we say that the instances in p are <i>covered </i>by R. The input also includes a map­ping A : <b>T — A, </b>that provides for each instance t an annotation A(t), called the <i>actual annotation </i>of t. The pair ( <b>T, </b>A) is the <i>training data.</i></p><p>Given the above input, a <i>decision list </i>L is an or­dering (i.e. permutation) of the input rules. The list L assigns an annotation to each instance t as defined below. We consider each rule according to the order­ing given by L until we find a rule Rj = (p, a) that covers t and assign the annotation a to t. We denote by L(t) the annotation assigned by L to t. Thus, L defines a function L : <b>I — A. </b>We say that the list L <i>correctly annotates </i>an instance t, if the annota­tion assigned by L matches the actual annotation of t, i.e., L(t) = A(t).</p><p>Given the above input, the MaxDL problem is to to construct a decision list L such that the number of instances correctly annotated by L, is maximized i.e., we want to maximize |{t|A(t) = L(t)}| . <b>Notations:</b></p><p>Let R = ( p, a) be a rule and t be an instance covered by R. We say that a rule R <i>correctly covers </i>t, if a = A(t). Similarly, R said to <i>incorrectly cover </i>t, if a = A(t).</p><p>Let L be a decision list. We say that an instance t is <i>happy </i>under L, if L correctly annotates t, i.e., L(t) = A(t). Let Happy(L) denote the set of in­stances that are happy under L. Notice that the M axDL problem asks for a decision list L such that |Happy(L)| is maximized.</p></section><section number="3" title="NP -Hardness and Inapproximability"><p>In this section, we prove that the MaxDL problem is NP-Hard and also show that the problem cannot even be approximated with any constant factor.</p><p><b>Theorem 1 </b><i>The </i>MaxDL <i>problem is </i><i>NP</i><i>-Hard.</i></p><p><i>Proof: </i>We give a reduction from the maximum inde­pendent set problem (MaxIS ), a well-known NP-Hard problem (Garey and Johnson, 1979). Recall that an independent set in a graph refers to any sub­set of vertices such that no two vertices from the set share an edge. The MaxIS problem is to find the largest independent set in a given undirected graph.</p><page local="3" global="837"/><p>Let G = (V, E) be the input graph having vertex set V = v2,..., vn}. We create an instance of the MaxDL problem as follows. For each vertex vj, we add an annotation aj to <b>A, </b>an instance tj to <b>T </b>and a rule Rj to <b>R. </b>We declare aj to be the actual annotation of tj. The predicted annotation of Rj is set to aj. We define Rj to cover only the instance tj and the instances corresponding to the neighbors of vj. Meaning, Rj covers the instances in the set {tj} <b>U </b>{tj Vj) <b>e </b>E<b>}</b>. This completes the reduc­tion. We claim that given a decision list L having k happy instances, we can construct an independent set of size k and vice versa. The NP-Hardness of MaxDL follows from the claim. We now proceed to prove the claim.</p><p>Consider a decision list L. Notice that for any instance tj, Rj is the only rule that correctly covers tj. Take any two different instances tj and tj that are happy under L. Without loss of generality, assume that Rj appears before Rj in L. Now, if Rj covers tj, tj would be unhappy under L. So, Rj does not cover tj, which implies that Vj is not a neighbor of vj (i.e., Vj) <b>e </b>E). Hence, the set / = {vj|tj <b>e </b>Happy(L)<b>} </b>is an independent set of G. We note that |1<b>1 </b>= |Happy(L)|.</p><p>Conversely, consider an independent set / of G. Let R(I) = {Rj|vj <b>e </b>/<b>}</b>. Form a decision list L by first arranging the rules from R(I) in any arbitrary order followed by arranging the rest of rules in any arbitray order. Notice that for any vertex vj <b>e </b>I, Rj correctly covers tj and no other rule appearing before Rj covers tj. Thus, tj is happy under L. It follows that |Happy(L)| <b>&gt; </b>|1<b>1. </b>We have proved that the MaxDL problem is NP-Hard. □</p><p>In our NP-Hardness reduction, we had shown that given a decision list L, we can construct an inde­pendent set / such that |Happy(L)| = |1<b>1, </b>and vice versa. This means that any approximation algo­rithm for the MaxDL problem can be translated (by combining it with our NP-Hardness reduction) into an equally good approximation algorithm for the MaxIS problem. Corollary 1 follows from (Zuck-erman, 2006).</p><p><b>Corollary 1 </b><i>If </i>NP = P <i>then for any </i>e &gt; 0, <i>the </i>MaxDL <i>problem cannot approximated within a factor of </i>n1-e. <i>In particular, the problem is not approximable within any constant factor.</i></p></section><section number="4" title="Heuristic Algorithms for the MaxDL Problem"><p>As the MaxDL problem is hard to approximate, we turn to heuristic approaches. All our heuristics fall into a natural greedy framework, described below.</p><subsection number="4.1" title="A Greedy Framework"><p>Our greedy framework for finding a decision list is as follows. In each iteration we greedily choose a rule and output it. For this purpose, we use some scoring function for assigning scores to the rules and choose the rule having the maximum score. Then the chosen rule is deleted. The process is contin­ued until all the rules are output. The above proce­dure gives us a decision list. We present this general framework in the Figure 1. The only unspecified part in the above framework is the scoring function. In­tuitively, the scoring function tries to measure the goodness of a rule.</p><doubt alpha="66.3" length="83" tooSmall="False" monospace="0.0">Given rule set R = {Ri,R2, ■ ■ ■ ,Rn}, instance set T and the actual annotations AO</doubt><doubt alpha="85.7" length="14" tooSmall="True" monospace="0.0">whileR =nulldo</doubt><doubt alpha="79.4" length="68" tooSmall="True" monospace="0.0">(re)compute scores for each rule in R, based on the scoring function</doubt><doubt alpha="81.8" length="44" tooSmall="True" monospace="0.0">select the rule R that has the maximum score</doubt><doubt alpha="78.3" length="23" tooSmall="True" monospace="0.0">remove R from the set R</doubt><doubt alpha="83.0" length="53" tooSmall="True" monospace="0.0">remove from T all the instances covered by Rend while</doubt><figure caption="Figure 1: A Greedy Framework for MaxDL prob­lem"></figure><p>For a rule R and an instance t, we define follow­ing notations for further use:</p><doubt alpha="75.6" length="45" tooSmall="True" monospace="0.0">InstR ={t\Rcoverst}Rulest ={R\tiscoveredby R}</doubt><doubt alpha="75.8" length="66" tooSmall="True" monospace="0.0">Inst+={t\Rcorrectly covers t}Rules+={R\tis correctly covered by R}</doubt><doubt alpha="78.3" length="69" tooSmall="True" monospace="0.0">Inst-={t\Rincorrectly coverst}Rules-={R\tis incorrectly covered by R}</doubt></subsection><subsection number="4.2" title="Simple Precision Scoring"><p>We now present our first candidate scoring function, which we call <i>simple precision scoring. </i>A natural score for a rule R is its <i>precision: </i>the fraction of in­stances covered correctly by R among the instances covered by it.</p><doubt alpha="80.0" length="15" tooSmall="True" monospace="0.0">IInst+I IInst+I</doubt><doubt alpha="50.0" length="20" tooSmall="True" monospace="0.0">Scoresp(R)=-R=--+R'_</doubt><doubt alpha="68.2" length="22" tooSmall="True" monospace="0.0">\InstR\\InstR\+\InstR\</doubt></subsection><subsection number="4.3" title="Weighted Precision Scoring"><p>Under Scoresp, the score of a rule R is determined only by the number of instances covered correctly</p><p><b>(|lnst+|) </b>and incorrectly <b>(|lnst-|). </b>The nature of</p><p>instances are not taken into account. The variants of Scoresp proposed here assigns weights to instances, based on which the scores are computed. We assign weights to the instances based on how easy it is to make them happy.<page local="4" global="838"/> For an instance t, define the <i>hap­piness quotient </i>h(t) to be the fraction of rules that correctly cover t among all the rules that cover t:</p><doubt alpha="33.3" length="6" tooSmall="False" monospace="0.0">h(t) :</doubt><doubt alpha="85.7" length="14" tooSmall="True" monospace="0.0">\Rulesf\Rulest</doubt><p>The value h( t) is a measure of how easy it is to make t happy; the larger the value of h( t) , it is easier to make t happy. For instance, if h(t) <b>w </b>1, then <b>|Rules+| </b><b>w </b><b>|Rulest|, </b>meaning that almost any rule that covers t will annotate it correctly. Thus, it is easy to make t happy. On the other extreme, if h(t) <b>w </b>0, then only a small fraction of the rules that cover t annotate it correctly. Thus it is harder to make t happy.</p><p>When we schedule a rule R, the instances in <b>Inst+ </b>become happy and those in <b>Inst- </b>become unhappy. Our new scoring functions give credit to R for each instance in <b>Inst+ </b>and award a penalty R for each instance in <b>Inst-. </b>The credit and the penalty depend on the happiness quotient of the in­stance. Informally, we want to give more credit R for making hard instances happy; similarly, we want to penalize R for making easy instances unhappy. A natural way of accomplishing the above is to award a credit of (1 <b>— </b>h(t)) for each instance t <b>e </b><b>Inst+ </b>and a penalty of h(t) for each instance t <b>e </b><b>Inst-. </b>Below, we formally define the above quantities as <i>gain </i>and <i>loss </i>associated with R. For each rule R, define</p><doubt alpha="71.4" length="7" tooSmall="True" monospace="0.0">Gain(R)</doubt><doubt alpha="30.0" length="10" tooSmall="False" monospace="0.0">E(!- h(t))</doubt><doubt alpha="57.1" length="14" tooSmall="False" monospace="0.0">Loss(R)=E h(t)</doubt><doubt alpha="62.5" length="8" tooSmall="True" monospace="0.0">t£ Inst—</doubt><p>Based on the above quantities, we define a natural scoring function, called <i>Weighted Precision:</i></p><doubt alpha="72.7" length="11" tooSmall="True" monospace="0.0">Scorewp(R):</doubt><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">n(R)</doubt><doubt alpha="58.8" length="17" tooSmall="True" monospace="0.0">Gain(R) + Loss(R)</doubt></subsection><subsection number="4.4" title="Refined Weighted Precision Scoring"><p>Our third scoring function is a refinement of the weighted precision scoring. In ScoreWP, we com­pute the happiness quotient of a token by taking in account the number of rules that cover the token and among those the ones that cover it correctly. The re­finement is obtained by also considering the nature of these rules. We define</p><doubt alpha="50.0" length="8" tooSmall="True" monospace="0.0">hRp(t) :</doubt><p>E „,„ , + precision(R)ER£Rulest precision(R)</p><p>Gain, loss and the scoring function are defined sim­ilar to that of ScoreWP:</p><doubt alpha="63.6" length="11" tooSmall="True" monospace="0.0">GainRp(R) =</doubt><doubt alpha="35.7" length="14" tooSmall="False" monospace="0.0">]T(1- hRp (t))</doubt><doubt alpha="57.1" length="14" tooSmall="False" monospace="0.0">L°SSRp(R)=E(t)</doubt><doubt alpha="66.7" length="9" tooSmall="True" monospace="0.0">t £ InstR</doubt><doubt alpha="72.7" length="11" tooSmall="True" monospace="0.0">ScoreRp(R)=</doubt><doubt alpha="67.7" length="31" tooSmall="True" monospace="0.0">GainRp(R) GainRp(R) + LossRp(R)</doubt></subsection></section><section number="5" title="Experiments"><p>In this section, we describe rule-ordering experi­ments on two real-world tasks. 1) named-entity (NE) annotation that relied on hand-crafted rules for MUC-7 dataset. 2) The second application we con­sider is rule-based multi-class text classification. We order rules learnt on benchmark text classification datasets and observe consistent improvements by merely re-ordering rules learnt by other rule learn­ers.</p><subsection number="5.1" title="Named Entity Annotation"><p>Rule-based named entity annotation is a natural in­stance of a decision list problem. Typically, rule-based NE annotation systems (Cunningham et al., 2002) require rules to be manually written as well as ordered manually. In this section, we show that our proposed rule-ordering algorithms perform bet­ter than the natural heuristic. Note that we do not intend to build a rule-based decision list which per­forms better than existing methods. <b>Setup: </b>In our problem formulation of M<b>AX</b>DL , the set of instances <b>T </b>and mapping A from in­stances to actual annotations, together form a train­ing set. We have access to a set of documents D = d2,..., that have all its named en­tities annotated. To generate pairs <b>(T, </b>A) using the set of documents D, let Tdi represent the set of token sequences that are annotated in a document dj <b>e </b>D. Let A(t) be the actual annotation for an instance t <b>e </b>Tdi. Given a set of rules <b>R </b>and a document collection D, each rule R <b>e R </b>is applied to each document dj <b>e </b>D. The set of token sequences (in­stances here) which R covers <b>(InstR), </b>is included in the set of instances <b>T</b>. For all instances t <b>e </b>Tdi, we add a mapping t <b>— </b>A(t) in A. For all other instances t <b>e </b><b>{InstR </b><b>— </b>Tdi<b>}</b>, we have a mapping t <b>— </b>null included in A. We perform these addi­tions for each document and rule pair. Finally, we add a rule R* = (<b>*</b>, null) to the rule set <b>R. </b>The pat­tern <b>* </b>matches every instance t <b>e      </b>[J <b>InstR</b> and associates a null annotation with the instance.<page local="5" global="839"/></p><p>We only consider "person name", "organization" and "place name" annotations. We use two different rule sets containing about 30 rules each.</p><p>Table 1 presents accuracy achieved by our pro­posed algorithms for the two chosen rule sets. In all the cases our proposed methods perform better than ScoresP. The result also shows that our proposed methods generalize better than simple ScoresP.</p></subsection><subsection number="5.2" title="Ordering classification rules"><p>In this section, we show another application of our algorithms in ordering classification rules. The antecedent of a classification rule is a series of tests on the input and the consequent gives the class label. Since different rules can assign conflicting classes, rule-ordering becomes important in choosing a correct class. These rules come from a variety of sources and could be hand-crafted or machine-learnt. Machine learnt rules could be generated using association mining (Agrawal and Srikant, 1994), inductive logic programming (Lavrac and Dzeroski, 1994), or Ripper (Cohen, 1995). Even classifiers can be seen as rules, <i>e.g., </i>linear discrim­inants are rules that assign one of two classes to exclusive partitions of input space. Due to domain specificity and unavailability of hand-tuned rules we illustrate rule-ordering on: (1) rules induced heterogeneous set of rules obtained from naive Bayes and decision trees <i>(BinRules).</i><i></i></p><doubt alpha="58.1" length="43" tooSmall="False" monospace="0.0">by Ripper (Cohen, 1995)(RipRules),and (2) a</doubt><p><b>Setup: </b>We used benchmark text classification datasets (Forman, 2003) available from the Weka site<footnote anchor="1"/>. These multi-class datasets represent 229 binary text classification problems, with positive class size avg. 149, and class skews avg. 1 : 31. These are subsets of various benchmark tasks like Reuters, TREC, and Ohsumed (oh). We present only a subset of the results (with only ScoreWP and ScoreWP) here for lack of space. We report experiments over 10 random 50 : 50 train-test splits. The training split is used to learn rules and their ordering. The orderings are evaluated on the test split and average train and test accuracies reported.</p><p><b>Results:</b></p><p><b>The </b><i>RipRules </i><b>setting: </b>We induce rules (from the train split) using the JRip implementation in Weka<footnote anchor="2"/> (Witten and Frank, 2005). We apply our vari­ous algorithms to merely re-order the rules output by JRip. In Table 2 we present results comparing JRip output with their re-ordered versions obtained from ScoreSP, ScoreWP and ScoreWP. Along with the name of each data set, the average number of rules induced from the training splits are also mentioned in parentheses. The best accuracies are marked in bold. We observe that the re-ordered rule-sets us­ing ScoreWP and ScoreWP perform better than both baselines ScoreSP and JRip with lower deviations. <b>The </b><i>BinRules </i><b>setting: </b>For an n-class problem we obtain classification rules by training a heteroge­neous collection of one-vs-rest binary classifiers. Each classifier is either a naive Bayes or a decision tree classifier trained to discriminate one class from the rest (2n classifiers). We treat each binary clas­sifier as a classification rule that <i>covers </i>an instance if the binary classifier assigns its associated class to that instance. In addition, corresponding to every class, we introduce a default classification rule that assigns the associated class to any instance it encounters.<page local="6" global="840"/> This gives us 3n rules. We used the naive Bayes and J48 implementations in Weka to obtain binary rules, ordered using ScoreWP and ScoreWP, and compared with ScoreSP baseline in Table 3. We also show individual classifier accuracy, and the best are marked bold. It is encouraging to note that all our rule-ordering techniques always outperform their multi-class counterparts on the test data set. We outperform the baseline ScoreSP method on all data sets with lower deviations.</p><footnote label="1">http://www.cs.waikato.ac.nz/ml/weka/ index_datasets.html</footnote><footnote>http://www.cs.waikato.ac.nz/ml/weka/</footnote><table caption="Table 1: Accuracies (in %) for different algorithms" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Rule-sets</p></td><td class="cell"><p>Accuracy</p></td><td class="cell"><p>Scoresp</p></td><td class="cell"><p>Scorewp</p></td><td class="cell"><p>ScoreWP</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Rule-set 1</p></td><td class="cell"><p>Trng</p></td><td class="cell"><p>76.4</p></td><td class="cell"><p>76.7</p></td><td class="cell"><p><b>78.9</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Test</p></td><td class="cell"><p>50.0</p></td><td class="cell"><p>52.7</p></td><td class="cell"><p><b>54.5</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Rule-set 2</p></td><td class="cell"><p>Training</p></td><td class="cell"><p>70.1</p></td><td class="cell"><p>71.6</p></td><td class="cell"><p><b>73.3</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Test</p></td><td class="cell"><p>49.1</p></td><td class="cell"><p>51.4</p></td><td class="cell"><p><b>52.0</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 2: Accuracies (in %) forRipRules" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Dataset (avg. # rules)</p></td><td class="cell"><p>Acc--uracy</p></td><td class="cell"><p>JRip</p></td><td class="cell"><p>Scoresp</p></td><td class="cell"><p>Scorewp</p></td><td class="cell"><p>ScoreWP</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>la2s (37)</p></td><td class="cell"><p>Trng Test</p></td><td class="cell"><p>86.16±0.39 76.93±0.43</p></td><td class="cell"><p>86.02±0.16 77.88±0.16</p></td><td class="cell"><p>86.68±0.16 78.05±0.17</p></td><td class="cell"><p><b>87.04±0.17 78.1±0.15</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>oh5 (28)</p></td><td class="cell"><p>Trng Test</p></td><td class="cell"><p>86.95±0.41 76.43±0.58</p></td><td class="cell"><p>88.26±0.21 79.08±0.37</p></td><td class="cell"><p>88.8±0.16 79.37±0.38</p></td><td class="cell"><p><b>89.06±0.17 79.24±0.35</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>tr45 (17)</p></td><td class="cell"><p>Trng Test</p></td><td class="cell"><p>91.88±0.38 78.9±0.47</p></td><td class="cell"><p>92.61±0.18 80.99±0.29</p></td><td class="cell"><p>92.84±0.23 81.19±0.28</p></td><td class="cell"><p><b>93.3±0.21 81.3±0.3</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 3: Accuracies (in %) forBinRules" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Data set</p></td><td class="cell"><p>Accu--racy</p></td><td class="cell"><p>Multi</p><p>J48</p></td><td class="cell"><p>-class NaiveBayes</p></td><td class="cell"><p>Scoresp</p></td><td class="cell"><p>Scorewp</p></td><td class="cell"><p>ScoreWp</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>la2s(18)</p></td><td class="cell"><p>Trng Test</p></td><td class="cell"><p>94.75±0.39 73.43±0.64</p></td><td class="cell"><p>85.78±0.29 73.68±0.37</p></td><td class="cell"><p>94.64±0.14 78.0±0.21</p></td><td class="cell"><p>95.9±0.03 78.46±0.23</p></td><td class="cell"><p><b>95.99±0.01 78.64±0.29</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>oh5 (30)</p></td><td class="cell"><p>Trng Test</p></td><td class="cell"><p>95.08±0.21 78.08±0.76</p></td><td class="cell"><p><b>99.56±0.09</b></p><p>74.16±0.77</p></td><td class="cell"><p>96.27±0.14 82.72±0.25</p></td><td class="cell"><p>98.43±0.09 83.16±0.24</p></td><td class="cell"><p>98.45±0.09 <b>83.98±0.26</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>tr45 (30)</p></td><td class="cell"><p>Trng Test</p></td><td class="cell"><p>97.91 ±0.11 85.25±1.02</p></td><td class="cell"><p>87.16±1.18 69.91±1.33</p></td><td class="cell"><p>97.71±0.14 84.06±0.44</p></td><td class="cell"><p>98.93±0.06 86.1±0.39</p></td><td class="cell"><p><b>98.98±0.05 86.42±0.41</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></subsection></section><section number="6" title="Conclusions"><p>In this paper, we formulated and studied the MaxDL problem. We proved the hardness of the problem. We then proposed some heuristic ap­proaches and established the usefulness of our meth­ods experimentally. We observed improved perfor­mance in classification task by merely reordering the rules obtained by an existing decision list learning algorithm. In future work, we would like to ex­plore how rule-ordering formulation can be applied to ordering heterogeneous classifiers in the ensem­ble learning setting.</p></section><references><p>Rakesh Agrawal and Ramakrishnan Srikant. 1994. Fast algorithms for mining association rules. In <i>VLDB, </i>pages 487-499.</p><p>D. Appelt, J. Hobbs, J. Bear, D. Israel, M. Kameyama, D. Martin, K. Myers, and M. Tyson. 1995. Sri inter­national fastus system: Muc-6 test results and analysis. In <i>MUC6 '95: Proc. of the 6th conf. on Message un­derstanding.</i></p><p>A. Borthwick. 1999. <i>A Maximum Entropy Approach to Named Entity Recognition. </i>Ph.D. thesis, New York University.</p><p>Eric Brill. 1992. A simple rule-based part-of-speech tag­ger. In <i>Proceedings ofANLP.</i></p><p>M. E. Califf and R. J. Mooney. 1998. Relational learning of pattern-match rules for information extraction. In <i>Working Notes of AAAI Spring Symposium on Apply­ing Machine Learning to Discourse Processing.</i></p><p>William W. Cohen. 1995. Fast effective rule induction. In <i>ICML, </i>pages 115-123.</p><p>H. Cunningham, D. Maynard, K. Bontcheva, and</p><p>V. Tablan. 2002. GATE: A framework and graphi­cal development environment for robust NLP tools and applications. In <i>Proceedings ofACL.</i></p><p>George Forman. 2003. An extensive empirical study of feature selection metrics for text classification. <i>JMLR Special Issue on Variable and Feature Selection, </i>3:1289-1305.</p><p>M. R. Garey and D. S. Johnson. 1979. <i>Computers and Intractability. </i>Freeman.</p><p>R. Grishman. 1997. Information extraction: Techniques and challenges. In <i>SCIE '97: Intnl. summer School on Information Extraction.</i></p><p>Hui Han, Eren Manavoglu, C. Lee Giles, and Hongyuan Zha. 2003. Rule-based word clustering for text classi­fication. In <i>SIGIR, </i>pages 445-446. ACM Press.</p><p>Furnkranz J. and Widmer G. 1994. Incremental re­duced error pruning. In <i>Machine Learning: Proc. of the Eleventh International Conference.</i></p><p>Nada Lavrac and Saso Dzeroski. 1994. <i>Inductive Logic Programming:Techniques and Applications. </i>El­lis Horwood, New York.</p><p>David D. Lewis, Rayid Ghani, Dunja Mladenic, Isabelle Moulinier, and Mark Wasson. 2003. Workshop on operational text classification. In <i>conjunction with SIGKDD.</i></p><p>Hang Li and Kenji Yamanishi. 1999. Text classification using ESC-based stochastic decision lists. In <i>CIKM.</i></p><p>D. Maynard, V. Tablan, C. Ursu, H. Cunningham, and Y. Wilks. 2001. Named entity recognition from di­verse text types. In <i>RANLP.</i></p><p>Ellen Riloff and Michael Thelen. 2000. A rule-based question answering system for reading comprehension tests. In <i>ANLP/NAACL 2000 Workshop on Reading comprehension tests as evaluation for computer-based language understanding sytems.</i></p><p>Ronald L. Rivest. 1987. Learning decision lists. <i>Ma­chine Learning, </i>2(3):229-246.</p><p>Minoru Sasaki and Kenji Kita. 1998. Rule-based text categorization using hierarchical categories. In <i>Pro­ceedings of SMC-98, IEEE International Conference on Systems, Man, and Cybernetics, </i>pages 2827-2830.</p><p>Sundar Varadarajan, Kas Kasravi, and Ronen Feldman. 2002. Text-mining: Application development chal­lenges. In <i>Proceedings of the Twenty-second SGAI In­ternational Conference on Knowledge Based Systems and Applied Artificial Intelligence.</i></p><p>Ian H. Witten and Eibe Frank. 2005. <i>Data Mining: Prac­tical Machine Learning Tools and Techniques. </i>Mor­gan Kaufmann.</p><p>D. Zuckerman. 2006. Linear degree extractors and the inapproximability of max-clique and chromatic num­ber. In <i>STOC.</i></p></references></body></article>