<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="134"/><title>Document classification in structured military messages</title><author surname="Carr" givenname="Oliver"><org  name="Defense Science and Technology Organisation" country="Australia" city="Edinburgh"/></author><author surname="Estival" givenname="Dominique"><org  name="Defense Science and Technology Organisation" country="Australia" city="Edinburgh"/></author></firstpageheader><frontmatter><p><b>Document Classification in Structured Military Messages</b></p><p><b>Oliver Carr and Dominique Estival</b></p><p>Human Systems Integration Group Command and Control Division Defence Science and Technology Organisation</p><p>(Oliver.Carr,Dominique.Estival}@dsto.defence.gov.au</p></frontmatter><abstract>We present new results for the DSTO project on document classification of military messages. We report more specifically on the improvements to the Part-Of-Speech (POS) tagging, a probabilistic process that assigns a tag to a token, and discuss the training for Date Time Groups POS tags. A new implementation of the rule-based classifier is described. The results obtained on two databases of real military messages are encouraging and the document classification module has now been integrated with a query user interface. log operational events and Lotus e-mail for actions and administrative functions. Around 200 messages per day are entered into these Lotus Notes log databases. Many DJFHQ staff members have expressed difficulty in finding particular information in their information reservoirs and our goal is to develop a more effective query interface between DJFHQ staff and their information reservoirs. This work already resulted in the development of the Query Building Interface (QBI), which was designed to create a better search interface to multiple log databases and to the users e-mail database. The rule-based Document Classifier we describe here has been trained and evaluated on Lotus operational log databases (OPS logs) from DJFHQ. It can now provide a categorisation for each document from the OPS logs and is integrated with QBI, as described in Section 7.2. </abstract></header><body><section number="1" title="Introduction"><p>In (Carr and Estival, 2002), we presented the first tentative results of the Document Classification project we have been conducting at DSTO and we discussed the shortcomings of the approach we were using. In this paper, we present the results we have obtained in the continuation of that project, after having implemented improvements in the POS tagging component and taken a different approach for the rule-based classifier component. These results show that rule-based classifiers can give reasonable results for structured textual information, when using appropriate language models for POS tagging.</p><subsection number="1.1" title="Goals of the project"><p>A large part of the Defence Information Environment (DIE) used at the Deployable Joint Force Headquarters (DJFHQ) is based on Lotus collaborative and messaging applications. The staff members of DJFHQ use Lotus databases to</p></subsection><subsection number="1.2" title="Proposed Architecture"><p>Fig.1 below shows how QBI and the Document Classifier could be integrated in the existing IT infrastructure. In this new Server Environment, both QBI and the Document Classifier interface with the Lotus Notes database.</p><doubt alpha="94.4" length="18" tooSmall="True" monospace="0.0">Server Environment</doubt><p><b>Fig.1 Proposed Architecture</b></p><p>In this architecture, users enter and access documents through Lotus Notes as they do now, and they receive notification of the document classification. One possible scenario is to use a Text-to-Speech module to warn of the arrival of some pre-specified document types, eg.<page local="2" global="135"/> NOTICAS (Notification of Casualty) or MEDSITREP (Medical Situation Report). For a NOTICAS, the injured person's details could be automatically retrieved and read out to the Commander or sent to a different display.</p><doubt alpha="83.3" length="12" tooSmall="True" monospace="0.0">Lotus E-mail</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Query Building Interface</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>J</b></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>r3</b></p></td><td class="cell"><p>Document Classifier</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></subsection><subsection number="1.3" title="Structure of the system"><p>The Document Classifier module in Fig.1 is named SOP-MRC (Standard Operating Procedures Rule Based Multiclass Classifier), and as shown in Fig.2, it consists of two main components: a Part-Of-Speech (POS) Tagger and a Classifier.</p><p><b>Fig. 2 The SOP-MRC module</b></p><p>The POS Tagger component, described in Section 3, is a probabilistic process that assigns a tag to a token. We also describe the training of this component in Section 3 and present our extension of the POS tagset for the military message data.</p><p>The Classifier component, described in Section 4, takes as input the list of pairs &lt;token&gt; &lt;tag&gt; produced by the POS Tagger for an incoming message and uses rules to determine the document type (including "free text") of the message.</p><p>We present in Section 5 the results we obtained on data from two military exercises. One database (VP-02) contains messages used to train the POS Tagger and to develop the classifier rules. The second database (TT-01) contains similar documents from another military exercise.</p></subsection></section><section number="2" title="Shortcomings of the previous approach"><p>There were two types of problems with the first approach we took to classify the SOP documents from the DJFHQ message database. The first one was that the data did not conform to expectations and the second one was that the classification rules were too brittle. Both issues have been addressed by the new approach to writing the classification rules described in Section 4.2.</p><p>DJFHQ operators use formatted text in the free text fields of their Lotus Notes operational log databases. This formatted text is defined by Standard Operating Procedure (SOP) documents, and there are 88 different SOP Document Types corresponding to different message types. Our first approach had been to define rules based on the definition of the SOPs, which are available to the operators writing those messages as MS Word documents. However, the actual messages often do not follow the format prescribed by the SOPs and, in addition, they often contain attachments and other material, which makes classification more difficult. The new rules now take into account variations in the way operators actually write their messages and allow more flexibility in the classification. This is described in Section 4.</p><p>Another problem was that the POS tagset used by our POS Tagger did not cover some token types that are very important in military messages. In particular, one lesson from our earlier work was that it is necessary to recognise Date Time Group (DTG) expressions and that we would have to develop our own tagset to fit the military domain. The additions we made to the POS tagset, are discussed in more detail in Sections 3.2 and 3.5.</p></section><section number="3" title="POS Tagging 3.1 QTAG"><p>The POS tagger we chose to use is Qtag, a portable trainable language-independent probabilistic tagger developed by the University of Birmingham (Mason, 2003; Tufis and Mason, 1998). There are several training corpora available on the Internet to train POS taggers.<footnote anchor="1"/> Qtag was originally trained with the Industrial Parsing of Software Manuals (IPSM) (Sutcliffe et al, 1996), which uses the Penn Treebank tagset, and it comes with the Birmingham - Lancaster Tagset and the associated resource file trained for English.</p><p>Qtag takes free text as input and outputs SGML, with each line containing the tag and the token it corresponds to. An example of input from our corpus and of the output produced by Qtag is given in Fig. 3.</p><footnote label="1">See e.g. the Automatic Mapping Among Lexico-Grammatical Annotation Models (AMALGAM) project: http://www.comp.leeds.ac.uk/amalgam/amalgam/multi- parsed.html</footnote><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>(</b></p></td><td class="cell"><p>"""Language^ v   Model ,,</p><p><b>POS TAGGER</b></p></td><td class="cell"><p><b>)</b></p><p><i>f </i>TaggecT<b>^N </b>* Vjyiessage^y</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>(</b></p></td><td class="cell"><p>■^"és<b>^gT</b><b>^X </b><b>_ CLASSIFIER</b></p><p>^lassificatiorv^ <b>CLASSIFIER</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="3" global="136"/><p><b><u>Input (VP-02)</u> </b><b>I </b><b><u>Qtag Output</u></b></p><p>From HQCLSC, HSS facilities allocated to CLSC as follows, A. 34 Fd Hosp (U.K)</p><p><b>Fig. 3 Output of Qtag using original tagset</b></p><p>To deal with the particular type of text contained in SOP documents, 59 new POS tags (mainly formatting tags) were added to the original tagset of 45 tags. Fig. 4 shows the same text as Fig. 3, tagged by Qtag using the language model containing these additional domain specific tags.<footnote anchor="23"/></p><p>From HQCLSC,</p><p>HSS facilities allocated to CLSC as</p><doubt alpha="57.1" length="28" tooSmall="False" monospace="0.0">follows, A. 34 Fd Hosp (U.K)</doubt><doubt alpha="59.1" length="22" tooSmall="False" monospace="0.0">&lt;w pos="From"&gt;From&lt;/w&gt;</doubt><doubt alpha="59.1" length="22" tooSmall="False" monospace="0.0">&lt;w pos="VB"&gt;HQCLSC&lt;/w&gt;</doubt><doubt alpha="31.2" length="16" tooSmall="False" monospace="0.0">&lt;w pos=","&gt;,&lt;/w&gt;</doubt><doubt alpha="52.6" length="19" tooSmall="False" monospace="0.0">&lt;w pos="NN"&gt;HSS&lt;/w&gt;</doubt><doubt alpha="64.8" length="54" tooSmall="False" monospace="0.0">&lt;w pos="NNS"&gt;facilities&lt;/w&gt; &lt;w pos="VBN"&gt;allocated&lt;/w&gt;</doubt><doubt alpha="50.0" length="18" tooSmall="False" monospace="0.0">&lt;w pos="TO"&gt;to&lt;/w&gt;</doubt><doubt alpha="55.0" length="20" tooSmall="False" monospace="0.0">&lt;w pos="NN"&gt;CLSC&lt;/w&gt;</doubt><doubt alpha="46.3" length="162" tooSmall="False" monospace="0.0">&lt;w pos="as"&gt;as&lt;/w&gt; &lt;w pos="NPS"&gt;follows&lt;/w&gt; &lt;w pos=","&gt;,&lt;/w&gt; &lt;w pos="FrmA"&gt;A.&lt;/w&gt; &lt;w pos="CD"&gt;34&lt;/w&gt; &lt;w pos="NN"&gt;Fd&lt;/w&gt; &lt;w pos="NN"&gt;Hosp&lt;/w&gt;&lt;w pos="JJ"&gt;(U.K)&lt;/w&gt;_</doubt><p><b>Fig. 4 Output of Qtag using new tagset</b></p><subsection number="3.2" title="Date Time Groups (DTGs)"><p>As was described in (Carr and Estival, 2002), the analysis of our previous results showed that they were unsatisfactory in part because the POS Tagger did not recognise Date Time Groups (DTGs), which are very common in our texts, and</p><p>HQCLSC stands for "Headquarters Combined Logistics Support Command". Note that the POS tags assigned to it (JJ in Fig.3 and VB in Fig.4) are incorrect, as are several of the other tags.</p><footnote label="3">The tag &lt;FrmA&gt;, meaning a "formatted A character", covers the strings "\nA.", "\n(A)" and "\nA)".</footnote><p>which play an important role in document type recognition for SOPs. For the purpose of document classification, a DTG is a single unit of information, but there are 3 types of DTGs that appear in the SOPs:</p><p>• DTG_S, with time and time zone information,</p><p>• DTG_M,  with  day,  time  and  time zone information,</p><p>• DTG_L, with day, time, time zone, month, and year information.</p><p>Examples of these are given in (1) with their corresponding POS tag.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(1)</doubt><doubt alpha="80.0" length="5" tooSmall="False" monospace="0.0">DTG_S</doubt><doubt alpha="80.0" length="5" tooSmall="False" monospace="0.0">DTG_M</doubt><doubt alpha="80.0" length="5" tooSmall="False" monospace="0.0">DTG_L</doubt><doubt alpha="20.0" length="5" tooSmall="False" monospace="0.0">1259Z</doubt><doubt alpha="14.3" length="7" tooSmall="False" monospace="0.0">310745Z</doubt><doubt alpha="33.3" length="12" tooSmall="False" monospace="0.0">200830ZAUG02</doubt><p>Although the 3 different types of DTGs are not often distinguished by the classifier rules, the POS Tagger needs to be trained on these 3 different DTG types, to avoid confusion with other alphanumeric strings. (2) is an example of the output from Qtag, where the DTG_S tag is correctly assigned to the text "1100K".</p><doubt alpha="31.0" length="29" tooSmall="False" monospace="0.0">(2)        &lt;w pos="at"&gt;AT&lt;/w&gt;</doubt><doubt alpha="41.7" length="24" tooSmall="False" monospace="0.0">&lt;w pos="DTG_S"&gt;1100K&lt;/w&gt;</doubt><doubt alpha="40.0" length="20" tooSmall="False" monospace="0.0">&lt;w pos="NN"&gt;C130&lt;/w&gt;</doubt><p>As shown in Table 1, the baseline performance of Qtag (trained on 80% of the VP-02 data and tested on the remaining 20%) for DTGs was fairly low. This is due to the inadequate training data for DTGs in this corpus, which comes from one military exercise covering a short period of time, and thus conatining few variations for dates.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">3.3</doubt><table caption="Table 1 Baseline Performance of Qtag Training with additional data."></table><p>To improve recognition of DTGs, we decided to create additional examples of DTGs to boost the training data for Qtag. For each of the DTG types, additional data was created in a systematic way to obtain instances of DTGs covering a wider range of dates and times. Table 2 shows the performance of Qtag with additional training data for DTGs.</p><doubt alpha="54.8" length="135" tooSmall="False" monospace="0.0">&lt;w pos="IN"&gt;From&lt;/w&gt; &lt;w pos="JJ"&gt;HQCLSC&lt;/w&gt; &lt;w pos=","&gt;,&lt;/w&gt; &lt;w pos="NN"&gt;HSS&lt;/w&gt; &lt;w pos="NNS"&gt;facilities&lt;/w&gt; &lt;w pos="VBN"&gt;allocated&lt;/w&gt;</doubt><doubt alpha="46.5" length="200" tooSmall="False" monospace="0.0">&lt;w pos="TO"&gt;to&lt;/w&gt; &lt;w pos="NN"&gt;CLSC&lt;/w&gt; &lt;w pos="CS"&gt;as&lt;/w&gt; &lt;w pos="VBZ"&gt;follows&lt;/w&gt; &lt;w pos=","&gt;,&lt;/w&gt; &lt;w pos="NN"&gt;A.&lt;/w&gt; &lt;w pos="CD"&gt;34&lt;/w&gt; &lt;w pos="NN"&gt;Fd&lt;/w&gt; &lt;w pos="NN"&gt;Hosp&lt;/w&gt;&lt;w pos="NN"&gt;(U.K)&lt;/w&gt;_</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>|POS Baseline |</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>"^^^ DTG_S DTG_M</p></td><td class="cell"><p>DTG_L</p></td><td class="cell"><p>All Tags</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Recall        9.68% 15.72%</p></td><td class="cell"><p>24.28%</p></td><td class="cell"><p>74.39%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Precision    6.90% 13.74%</p></td><td class="cell"><p>14.86%</p></td><td class="cell"><p>75.23%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="4" global="137"/><p>Since the worst performing category had been DTG_M, we first added 482 additional instances of DTG_M to the training file for Qtag. The same process was repeated for DTG_S and DTG_L, with 158 and 8,063 additional instances respectively.</p></subsection><subsection number="3.4" title="Overtraining"><p>Table 2 shows that the performance of Qtag improved when the DTG_M data was first added but decreased significantly after DTG_L data was added (additional DTG_S training data did not make any significant difference). The decrease in performance after the DTG_L data was added is due to overtraining of Qtag. Using the same recursive algorithm, adding year information leads to the creation of many more instances of DTG_L than DTG_M and DTG_S and skews the training data, resulting in many false positives for that category. Since the training text with the added DTG_M and DTG_S gave the best performance, this is what we used to create the Qtag language model.</p></subsection><subsection number="3.5" title="New POS tags for measure units"><p>Table 3 shows examples of DTG tokens that were miscategorised by Qtag. We can see that most of these are in fact genuine DTGs, which is good news since the classifier rules are not concerned with the type of DTG (DTG_S, DTG_M, or</p><p>DTG_L) but only with the occurrence of a DTG.</p><p>Further analysis of the miscategorisations shown in Table 3 suggests ways in which the performance of the POS Tagger can be improved: §    add additional training data for DTGs with</p><p>different minute information than 0 or 5; §    add POS tags for measure units, such as &lt;WGT&gt; for weights, &lt;DST&gt; for distances, &lt;SPD&gt; for speeds, §    add POS tags for the various different types of Latitude and Longitude information or Grid reference. Some examples are given in (3).</p><doubt alpha="30.3" length="33" tooSmall="False" monospace="0.0">(3)      &lt;w pos= "WGT"&gt;2500KG&lt;/w&gt;</doubt><doubt alpha="40.0" length="60" tooSmall="False" monospace="0.0">&lt;w pos ="DST"&gt;500NM&lt;/w&gt; &lt;w pos = LATLONG&gt;15.35S/151.20E &lt;/w&gt;</doubt><p>In the end, 71 extra tags were added to the tagset, giving a total of 116 POS tags.<footnote anchor="2"/> The new Qtag language model was trained on 80% of the POS tags from the VP-02 data. The remaining 20% (36862 tags from 430 messages from VP-02) were used to test the performance of the POS Tagger. Table 4 shows the results obtained for the DTG_S, DTG_M and DTG_L tags, after Qtag was trained with the additional training data for the new measure units POS tags.</p><p>There were 57 tags for formatting, 3 for DTGs, 3 for measure units and 6 for Lat/Long/Grid. Only 111 out of the 116 different POS tags appear in our test data.</p><table caption="Table 2 Additional training data for DTGs" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>additional DTG_M</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>.................................................</p></td><td class="cell"><p>DTG_S</p></td><td class="cell"><p>DTG_M</p></td><td class="cell"><p>DTG_L</p></td><td class="cell"><p>All Tags</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Rec all</p></td><td class="cell"><p>100.00%</p></td><td class="cell"><p>98.91%</p></td><td class="cell"><p>95.65%</p></td><td class="cell"><p>90.81%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Precision</p></td><td class="cell"><p>80.52%</p></td><td class="cell"><p>98.91%</p></td><td class="cell"><p>95.65%</p></td><td class="cell"><p>85.94%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>additional DTG_S</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>...................................................</p></td><td class="cell"><p>DTG_S</p></td><td class="cell"><p>DTG_M</p></td><td class="cell"><p>DTG_L</p></td><td class="cell"><p>All Tags</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Rec all</p></td><td class="cell"><p>100.00%</p></td><td class="cell"><p>98.91%</p></td><td class="cell"><p>95.65%</p></td><td class="cell"><p>92.75%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Precision</p></td><td class="cell"><p>80.52%</p></td><td class="cell"><p>98.91%</p></td><td class="cell"><p>95.65%</p></td><td class="cell"><p>87.77%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>additional DTG_L</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>..................................................</p></td><td class="cell"><p>DTG_S    DTG_M     DTG_L    All Tags</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Rec all</p></td><td class="cell"><p>100.00%    98.91%      95.65% 90.95%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Precision</p></td><td class="cell"><p>80.52%     98.91%      59.46% 85.94%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 3 Errors in POS tagging for DTGs" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Number</b></p></td><td class="cell"><p><b>Example</b></p></td><td class="cell"><p></p></td><td class="cell"><p><b>Correct Tag(s)</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p>030/02OF170015ZMAY02</p></td><td class="cell"><p>NN</p></td><td class="cell"><p>DTG_L</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p>2100S</p></td><td class="cell"><p>DTG_S</p></td><td class="cell"><p>LAT_LONG_S</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2</p></td><td class="cell"><p>4000FT,</p></td><td class="cell"><p>DTG_S</p></td><td class="cell"><p>DST</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>8</p></td><td class="cell"><p>4000L</p></td><td class="cell"><p>DTG_S</p></td><td class="cell"><p>WGT</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>5</p></td><td class="cell"><p>171659Z</p></td><td class="cell"><p>NN</p></td><td class="cell"><p>DTG_M</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>13</p></td><td class="cell"><p>WEST</p></td><td class="cell"><p>req</p></td><td class="cell"><p>NN</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>8</p></td><td class="cell"><p>5000M</p></td><td class="cell"><p>DTG_S</p></td><td class="cell"><p>DST</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>32</p></td><td class="cell"><p>(2)AT</p></td><td class="cell"><p>NN</p></td><td class="cell"><p>Frm2 at</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>7</p></td><td class="cell"><p>(0.5-0.7</p></td><td class="cell"><p>NN</p></td><td class="cell"><p>CD</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p>A.151206KMAR02</p></td><td class="cell"><p>NN</p></td><td class="cell"><p>FrmA DTG_L</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2</p></td><td class="cell"><p>PD:130800K</p></td><td class="cell"><p>NN</p></td><td class="cell"><p>Pd DTG_M</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p>C.LAND</p></td><td class="cell"><p>NN</p></td><td class="cell"><p>FrmC VB</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 4 DTGs with new language model" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>New Tags</b></p></td><td class="cell"><p><b>Recall</b></p></td><td class="cell"><p><b>Precision</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>DTG_S</p></td><td class="cell"><p>100.00%</p></td><td class="cell"><p>91.18%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>DTG_M</p></td><td class="cell"><p>98.91%</p></td><td class="cell"><p>99.27%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>DTG_L</p></td><td class="cell"><p>97.34%</p></td><td class="cell"><p>100.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="5" global="138"/><p>Table 5 shows the overall results for Qtag using the macroaverage and microaverage statistics as described in (Sebastiani, 2001). Almost half of the POS tags in the test data were &lt;NN&gt;. We believe using the microaverage result without &lt;NN&gt; gives a better indication of performance.</p></subsection></section><section number="4" title="Rule-based Classifier"><p>Unlike most work on document classification (see Jackson and Moulinier, 2002, or Manning and Schütze, 1999), we do not rely on the semantic content of the documents to classify our documents, but take advantage of the very highly constrained structure of the documents. This is an example of <i>Category-Pivoted Text Classification </i>where the classifier is given a classification and must find which messages should be assigned to a given class, as opposed to <i>Document-Pivoted Text Classification, </i>which tries to determine the appropriate classifications for a set of documents (Sebastiani, 2002).</p><p>Quoting from (Jackson and Moulinier, 2002), there are two views of NLP: "Symbolic NLP tends to work top-down by imposing known grammatical patterns and meaning associations upon texts. Empirical NLP tends to work bottom-up from the texts themselves, looking for patterns and associations to model, some of which may not correspond to purely syntactic or semantic relationships." Empirical NLP has been widely used since the early 1990's while Symbolic NLP has been viewed less favourably. The system we describe here is in the tradition of Symbolic NLP, as the categories we use have been pre-defined and do not emerge from the data. However, at this point, classification is mainly performed on the basis of formatting structures, not on linguistic constructs.</p><p>Our first rule-based classifier used an "if, else" structure to parse the tags returned from Qtag one at a time. The document type was determined solely on the basis of the previous tag and the current tag, and only &lt;Start&gt; tags and the last one or two &lt;End&gt; tags were used to classify a message. A large amount of code (in Python) was written to implement this method, which turned out to be neither efficient nor successful.</p><p>Our first approach was too optimistic and too reliant on the document structure given in the SOPs, and our rules did not perform well. Our second implementation of a rule-based classifier uses regular expressions to state the rules. Regular expressions allow us to define more detailed rules and they also allow for more flexibility.</p><subsection number="4.1" title="Regular Expressions as rules"><p>As discussed in (Carr and Estival, 2002), the discrepancy between the format prescribed by the SOPs and the real text input by the operators was one of the main causes of errors. The use of regular expressions as rules allows flexibility in rule definition and result in shorter and more effective code. Several rules can be written to recognise one SOP document type.</p><p>The output of Qtag is read into a string. This string contains the list of POS tags for a message. Each rule recognises the tags for one SOP document type and allows any number of other tags in between. Only those POS tags required by the classifier rules are read into the Classifier. Having all the POS tags in a string also allows message headers and multiple SOPs to be pruned off or recognised differently very easily.</p><p>We give in (4) an example of a classifier rule for document type "P", where there can be any number of tags before &lt;Frm1&gt; and at least one instance of the separator or more tags before &lt;Frm2&gt;. In (5), we give an example of a message containing a document of type "P".</p><doubt alpha="42.9" length="352" tooSmall="False" monospace="0.0">(4) P = ([ a-za-z]|[ A-Za-za-z0-9]|[ A-ZA-Z]|[ DTG_S]|[ DTG_M]|[ DTG_L]|[ a-za-za-za-za-za-za-z]|[ A-Za-za-za-z]|[ ]]){0,}Frm1([ a-za-z]|[ A-Za-za-z0-9]|[ A-ZA-Z]|[ DTG_S]|[ DTG_M]|[ DTG_L]|[ a-za-za-za-za-za-za-z]|[ A-Za-za-za-z]|[ ]]){ 1,}Frm2.([ a-za-z]|[ A-Za-za-z0-9]| [A-ZA-Z]|[ DTG_S]|[ DTG_M]|[ DTG_L]|[ a-za-za-za-za-za-za-z]|[ A-Za-za-za-z]|[</doubt><doubt alpha="0.0" length="7" tooSmall="False" monospace="0.0">]]){0,}</doubt><doubt alpha="61.5" length="39" tooSmall="False" monospace="0.0">(5) CD DTG_M Frm1 CD From at Frm2 DTG_L</doubt></subsection><subsection number="4.2" title="SOP-MRC rules"><p>As mentioned earlier, the classifier rules were first created following the 88 SOP document definitions. They were later derived from a corpus analysis and further refined after analysis of the results on the same corpus.<page local="6" global="139"/> The rules use mostly POS tags relating to formatting, eg. &lt;Frm1&gt; also some content information, with the POS tags for DTGs and &lt;CD&gt; (number). A total of 66 rules were used to recognise the 37 document types that appeared in the VP-02 data. Of these 66 rules, 44 rely on the POS tags for DTGs or &lt;CD&gt;.</p><table caption="Table 5 Overall Performance of Qtag" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Recall and Precision Averages</b></p></td><td class="cell"><p><b>Recall</b></p></td><td class="cell"><p><b>Precision</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Macroaverage</p></td><td class="cell"><p>89.69%</p></td><td class="cell"><p>95.65%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Microaverage</p></td><td class="cell"><p>97.84%</p></td><td class="cell"><p>97.08%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Microaverage (no NN)</p></td><td class="cell"><p>92.67%</p></td><td class="cell"><p>94.75%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><doubt alpha="60.9" length="46" tooSmall="False" monospace="0.0">("formatted 1") or &lt;FrmB&gt; ("formatted B"), but</doubt><p>One disadvantage of using regular expressions to implement classifier rules might be that they can be fairly long. The example in (4) is one of the shortest rules in terms of number of elements. However, this problem can be alleviated by the use of named groups and the Python interpreter is useful to test the regular expressions before they are included into the classifier.</p><p>It is also worth noting that these handcrafted rules were in fact written very quickly, much more quickly than "one rule in two days" as described by (Jackson and Moulinier, 2002).</p><p>Each message is tested against all the rules for SOP document types. If no match is found, then the document is assigned to the document type "Free Text". Some rules are in fact subsets of other rules. This defines a hierarchy of rules which can be used to determine the correct SOP document type, see Section 6.</p></subsection></section><section number="5" title="Results"><p>We present the results obtained by SOP-MRC on two different message databases. The VP-02 database was used for training the POS Tagger and to define the classifier rules. It contains 2328 messages and 37 document types. The TT-01 database contains 3131 messages and 18 document types. The detailed results for each document type, for both VP-02 and TT-01, are given in Appendix 1 and 2.</p><p>The first part of Table 6 shows the overall results of SOP-MRC for VP-02. Since over 75% of the messages are "Free Text", we also show the microaverage result without the "Free Text" category to give a better indication of performance.</p><p>The second part of Table 6 shows the overall results of SOP-MRC for TT-01. In this corpus, over 85% of the messages are "Free Text" and the microaverage result is again given without the</p><doubt alpha="66.7" length="24" tooSmall="False" monospace="0.0">"Free Text" category.___</doubt><p><b><u>Recall and Precision Averages    Recall Precision</u></b></p><p>These results are very encouraging. Although the macroaverage for TT-01 is not very good, this is explained by the fact that there were a number of False Positives for document types which do not occur in this data (see Appendix 2). The microaverage shows that the document types with larger numbers of documents are giving as good results for the new unseen data as for VP-02.</p><p>An explanation for the discrepancies between the document types used in VP-02 and in TT-01 is that the SOP definitions were actually developed at DJFHQ, and that VP-02 was a military exercise which only involved that headquarters, with all the messages coming from DJFHQ, while TT-01 was a four nation exercise, with messages coming from a number of different headquarters.</p><p>Another issue concerns the "Shift Handover" documents. The Shift Handover form is filled in by officers "handing over" their shift to another officer who "watches" the database for outstanding issues, and is a summary of the past 12 or 24 hours. Although this form is essentially free text, because the officers tend to think in terms of formatted documents, they often write it as another formatted document, eg. with numbered items for new paragraphs. If we classified the Shift Handover form as "Free Text", the accuracy would improve. This can be seen as another example of the well-know fact that the operator or human element is a large factor in system success.</p></section><section number="6" title="Multilabel classification"><p>One of the lessons from our earlier work was that we needed to use a multilabel classification rather than a simple multiclass classification. In multiclass classification, each message is assigned to only one of several possible classes, while in a multilabel classification, a message can be assigned to one or more classes (Lewis, 2002). Our new classifier rules now perform a type of multilabel classification, by assigning a complex label to each message.<page local="7" global="140"/> An example of this complex label is shown in (6).<footnote anchor="3"/></p><table caption="Table 7 SOP-MRC for VP-02 and TT01" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>VP-02 </b><b>1</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Macroaverage</p></td><td class="cell"><p>79.99%</p></td><td class="cell"><p>67.94%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Microaverage</p></td><td class="cell"><p>82.49%</p></td><td class="cell"><p>81.52%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Microaverage (no Free Text)</p></td><td class="cell"><p>70.53%</p></td><td class="cell"><p>43.41%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>TT01</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Macroaverage</p></td><td class="cell"><p>12.72%</p></td><td class="cell"><p>13.09%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Microaverage</p></td><td class="cell"><p>86.77%</p></td><td class="cell"><p>83.88%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Microaverage (no Free Text)</p></td><td class="cell"><p>77.39%</p></td><td class="cell"><p>26.81%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><doubt alpha="44.0" length="25" tooSmall="False" monospace="0.0">(6)       C:B:A:Free Text</doubt><p>This example shows the output of a message that contains a document of type C. As mentioned above, some rules are actually subsets of other rules, thereby defining a hierarchy of document types. In this case, the rule for document type C includes the rule for document type B, which includes the rule for document type A; thus A and B are also included in the complex label, as well as Free Text, the default classification.</p><p>In our current implementation, we choose the label returned by the more specific rule in the complex label, and return it as the single label, or multiclass classification, for the message being classified (in this case, C). Although Sebastiani (2002) argues that a multilabel classifier cannot be used as a single label classifier, the complex label that is returned by our classifier component is in fact a multilabel classification in terms of the hierarchical structure of the classifier rules. This hierarchy can be thought of as a set of binary classifiers (implemented as classifier rules) listed in order, from smallest (more general) to largest (more specific). This set is ordered such that a document type is a subset (in terms of structure, not content) of the next document type.<footnote anchor="4"/></p></section><section number="7" title="Conclusions and future work 7.1 Improvements"><p>The expanded POS tagset provides a better coverage for the texts in our domain, and the POS Tagger component is now trained for the real data found in military message databases.</p><p>The new classifier component is much cleaner and more efficient. Python provides high-level methods to implement regular expressions and use</p><p>The names of the document types have been replaced by alphabetical labels for presentation; in the real system, the categories have meaningful labels.</p><footnote label="4">For example, a MEDSITREP (Medical Situation Report) is conceptually a kind of SITREP (Situation Report), but a SITREP is not a kind of "Free Text", even though the rules for "SITREP" and "Free Text" are in a subset relation.</footnote><p>them to search strings of text, which makes it easier to modify the classifier and to add or change the rules.</p><p>The results on a new database of messages, which were not used to create the classifier rules, are encouraging and indicate that we can improve the performance of SOP-MRC with little effort.</p><subsection number="7.2" title="Integration with QBI"><p>QBI is an improved search interface to the Lotus Notes operational log database used at DJFHQ, which is developed by the same DSTO team as SOP-MRC. We aim to incorporate the output of SOP-MRC with QBI by providing a category-pivoted view of the documents as categorised by SOP-MRC. An example of this view is shown in</p><p>Fig. 5.</p><p>The categorised view will allow the users of the QBI to quickly find messages in a Lotus Notes operational log database by using document types to limit their search or to locate the relevant message.</p></subsection><subsection number="7.3" title="Other Improvements to SOP-MRC"><p>The performance of the POS Tagger could be improved by pre-processing the messages. Text such as "10 KM" could be normalised to "10KM" so the POS Tagger can properly tag it &lt;DST&gt; rather than &lt;CD&gt; &lt;NN&gt;. This would also help improve the classifier's performance.</p><p>The current implementation relies on a one-to-one correspondence between classifier rules and document types. We are looking at another approach, in which a classifier rule would be a subset of a number of rules for a few document types, in other words we would have a more general rule for a set of document types. This would correspond to implementing the true multilabel classification mentioned in Section 6, where the hierarchy of rules also correspond to the conceptual hierarchy of document types.</p><page local="8" global="141"/><p><b>Fig. 5 Screenshot of QBI with SOP-MRC</b></p><p>If an incoming message matches the more general rule, it can then be tested against more specific rules. If the message fails to match one of these, then other methods can be used to determine the more specific document type, but at least the more general category can be kept, rather than defaulting to "Free Text", as is currently the case.</p><p>Further analysis to determine the more specific document type would involve the number of certain POS tags, the ordering of these tags or the absence of certain tags.</p><p>Another improvement concerns the addition of further POS tags for our domain, for example, tags for unit names and ranks. This information would also be useful in further work on both document classification and information extraction. For instance, it would allow extracting information about which units are involved or identify the personnel injured from NOTICAS messages.</p><p>We are also investigating the development of a trainable system, using approaches such as TF-IDF, Rocchio Method, Support Vector Machines or hybrid solutions such as Learning from Positive and Unlabeled text documents (Jackson and Moulinier, 2002; Joachims, 1998; Vapnik, 1995; Lee and Lui, 2003).</p><p><i>Retrieval, Extraction and Categorization, </i>Vol. 5, Natural Language Processing, John Benjamin Publishing Company, Philadelphia.</p><p>Joachims, T. (1998) Text Categorization with Support Vector Machines: Learning with Many Relevant Features. <i>Proceedings of the European Conference on Machine Learning, </i>Springer.</p><p>Johnson, D.E., Oles, F.J., Zhang, T., Goetz, T. (2002), A Decision-Tree-Based Symbolic Rule Induction System for Text Categorization, <i>IBM Systems Journal, </i>Vol. 41, No. 3, IBM Corporation.</p><p>Lee, W.S. and Liu, B. (2003) Learning with Positive and Unlabeled Examples using Weighted Logistic Regression. <i>Proceedings of the Twentieth International  Conference  on Machine Learning</i> (ICML-2003), Washington, DC, USA.</p><p>Lewis, David (2002). <i>Machine Learning for Text Classification Applications, </i>Tutorial. 40th Meeting of the Association of Computational Linguistics, University of Pennsylvania. Philadelphia, USA.</p><p>Manning, C.D. and Schütze, H. (1999). <i>Foundations of Statistical Natural Language Processing. </i>MIT Press, Cambridge, Massachusetts.</p><p>Mason, Oliver (2003). <i>Qtag 3.1. </i>Department of English, School of Humanities, University of Birmingham, http://web.bham.ac.uk/O.Mason/software/tagger/</p><p>Sebastiani, Fabrizio. (2002). Machine Learning in Automated Text Categorization. <i>ACM Computing Surveys, </i>Vol. 34, No. 1, pp. 1-47.</p><p>Sutcliffe, Richard, Koch, Heinz-Detlev and McElligott, Anne (eds) (1996), <i>Industrial Parsing of Software Manuals. </i>Amsterdam, Rodopi.</p><p>Tufis, Dan and Mason, Oliver (1998). Tagging Romanian Texts: a Case Study for QTAG, a Language    Independent    Probabilistic Tagger, <i>Proceedings of the First International Conference on Language Resources &amp; Evaluation (LREC), </i>Granada, Spain, 28-30 May 1998, p.589-596.</p><p>Vapnik, V. N. (1995) <i>The Nature of Statistical Learning Theory. </i>Springer-Verlag, New York.</p></subsection><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Document Classification</b></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>Classification                      Date Created                ; From Database                        ; Author                                    ; Size                  ; ±</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>16/01/2002 01:10:10 AM</p></td><td class="cell"><p>1500</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>10/01/2002 12:18:33 PM</p></td><td class="cell"><p>1131</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>»AB</p><p>10/01/2002 07:55:08 AM</p></td><td class="cell"><p>725</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>13/01/2002 08:00:02 AM</p></td><td class="cell"><p>1013</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>11/01/2002 01:00:10 AM</p></td><td class="cell"><p>787</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>15/01/2002 07:57:37 AM</p></td><td class="cell"><p>313</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>15/01/2002 01:32:12 PM</p></td><td class="cell"><p>1223</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>10/01/2002 03:21:10 AM</p></td><td class="cell"><p>2083</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>10/01/2002 01:11:08 PM</p></td><td class="cell"><p>070</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>15/01/2002 07:20:53 PM</p></td><td class="cell"><p>535</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>10/01/2002 02:11:13 AM</p></td><td class="cell"><p>572</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>17/01/2002 03:11:35 AM</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>17/01/2002 01:00:10 PM</p></td><td class="cell"><p>731</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>17/01/2002 07:38:05 PM</p></td><td class="cell"><p>1310</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>»CAA</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>13/01/2002 08:18:23 AM</p></td><td class="cell"><p>1133</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>13/01/2002 00:11:11 PM</p></td><td class="cell"><p>712</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>15/01/2002 10:13:21 AM</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>15/01/2002 12:50:20 PM</p></td><td class="cell"><p>1212</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>15/01/2002 10:01:33 PM</p></td><td class="cell"><p>2000</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>15/01/2002 10:30:07 PM</p></td><td class="cell"><p>1053</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>15/01/2002 11:07:51 PM</p></td><td class="cell"><p>1381</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>10/01/2002 12:10:23 PM</p></td><td class="cell"><p>581</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>17/01/2002 11:31:08 PM</p></td><td class="cell"><p>013</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>18/01/2002 12:15:51 AM</p></td><td class="cell"><p>750</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><references><doubt alpha="66.7" length="54" tooSmall="False" monospace="0.0">Carr, O. and Estival, D. (2002) Text Classification of</doubt><p>Formatted Text Document. <i>Proceedings of the 5th Australasian Natural Language Processing Workshop. </i>Canberra. pp. 49-54.</p><p>Jackson, Peter and Moulinier, Isabelle (2002) <i>Natural Language Processing for Online Applications - Text</i><page local="9" global="142"/></p></references><appendix title="Appendix 2 SOP-MRC on TT01"></appendix><appendix title="Appendix 1 SOP-MRC on VP02"><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Doc Type</p></td><td class="cell"><p>Gold Total</p></td><td class="cell"><p>TP</p></td><td class="cell"><p>FP     FN     Recall Precision</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>A</p></td><td class="cell"><p>1 2</p></td><td class="cell"><p>1</p></td><td class="cell"><p>1       0     100.00% 50.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>B</p></td><td class="cell"><p>1 1</p></td><td class="cell"><p>1</p></td><td class="cell"><p>0       0     100.00% 100.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>C</p></td><td class="cell"><p>10 6</p></td><td class="cell"><p>6</p></td><td class="cell"><p>0       4     60.00% 100.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>D</p></td><td class="cell"><p>2 5</p></td><td class="cell"><p>1</p></td><td class="cell"><p>4       1     50.00% 20.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>E</p></td><td class="cell"><p>14 10</p></td><td class="cell"><p>3</p></td><td class="cell"><p>7      11     21.43% 30.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>F</p></td><td class="cell"><p>1 2</p></td><td class="cell"><p>1</p></td><td class="cell"><p>1       0     100.00% 50.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>G</p></td><td class="cell"><p>6 5</p></td><td class="cell"><p>4</p></td><td class="cell"><p>1       2     66.67% 80.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>H</p></td><td class="cell"><p>1 1</p></td><td class="cell"><p>1</p></td><td class="cell"><p>0       0     100.00% 100.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>I</p></td><td class="cell"><p>2 2</p></td><td class="cell"><p>2</p></td><td class="cell"><p>0       0     100.00% 100.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>J</p></td><td class="cell"><p>1 3</p></td><td class="cell"><p>1</p></td><td class="cell"><p>2       0     100.00% 33.33%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>K</p></td><td class="cell"><p>11 18</p></td><td class="cell"><p>7</p></td><td class="cell"><p>11      4     63.64% 38.89%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>L</p></td><td class="cell"><p>33 16</p></td><td class="cell"><p>9</p></td><td class="cell"><p>7      24    27.27% 56.25%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Free Text</p></td><td class="cell"><p>1846 1657</p></td><td class="cell"><p>1625</p></td><td class="cell"><p>32     221    88.03% 98.07%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>M</p></td><td class="cell"><p>3 2</p></td><td class="cell"><p>2</p></td><td class="cell"><p>0       1     66.67% 100.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>N</p></td><td class="cell"><p>19 28</p></td><td class="cell"><p>8</p></td><td class="cell"><p>20      11    42.11% 28.57%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>O</p></td><td class="cell"><p>2 4</p></td><td class="cell"><p>2</p></td><td class="cell"><p>2       0     100.00% 50.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>P</p></td><td class="cell"><p>164 391</p></td><td class="cell"><p>131</p></td><td class="cell"><p>260     33    79.88% 33.50%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Q</p></td><td class="cell"><p>34 22</p></td><td class="cell"><p>13</p></td><td class="cell"><p>9      21     38.24% 59.09%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>R</p></td><td class="cell"><p>3 3</p></td><td class="cell"><p>3</p></td><td class="cell"><p>0       0     100.00% 100.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>S</p></td><td class="cell"><p>32 27</p></td><td class="cell"><p>27</p></td><td class="cell"><p>0       5     84.38% 100.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>T</p></td><td class="cell"><p>1 4</p></td><td class="cell"><p>1</p></td><td class="cell"><p>3       0     100.00% 25.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>U</p></td><td class="cell"><p>20</p></td><td class="cell"><p>30</p></td><td class="cell"><p>18</p></td><td class="cell"><p>12</p></td><td class="cell"><p>2</p></td><td class="cell"><p>90.00% 60.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>V</p></td><td class="cell"><p>5 5</p></td><td class="cell"><p>5</p></td><td class="cell"><p>0       0     100.00% 100.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>W</p></td><td class="cell"><p>1 3</p></td><td class="cell"><p>1</p></td><td class="cell"><p>2       0     100.00% 33.33%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>X</p></td><td class="cell"><p>1 2</p></td><td class="cell"><p>1</p></td><td class="cell"><p>1       0     100.00% 50.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Y</p></td><td class="cell"><p>5 2</p></td><td class="cell"><p>2</p></td><td class="cell"><p>0       3     40.00% 100.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Z</p></td><td class="cell"><p>1 11</p></td><td class="cell"><p>1</p></td><td class="cell"><p>10      0     100.00% 9.09%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>AA</p></td><td class="cell"><p>1 1</p></td><td class="cell"><p>1</p></td><td class="cell"><p>0       0     100.00% 100.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>BB</p></td><td class="cell"><p>2 3</p></td><td class="cell"><p>2</p></td><td class="cell"><p>1       0     100.00% 66.67%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CC</p></td><td class="cell"><p>1 1</p></td><td class="cell"><p>1</p></td><td class="cell"><p>0       0     100.00% 100.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>DD</p></td><td class="cell"><p>81 31</p></td><td class="cell"><p>31</p></td><td class="cell"><p>0      50    38.27% 100.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>EE</p></td><td class="cell"><p>3 4</p></td><td class="cell"><p>2</p></td><td class="cell"><p>2       1     66.67% 50.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>FF</p></td><td class="cell"><p>1 2</p></td><td class="cell"><p>1</p></td><td class="cell"><p>1       0     100.00% 50.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>GG</p></td><td class="cell"><p>1</p></td><td class="cell"><p>1</p></td><td class="cell"><p>1</p></td><td class="cell"><p>0</p></td><td class="cell"><p>0</p></td><td class="cell"><p>100.00% 100.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>HH</p></td><td class="cell"><p>1 1</p></td><td class="cell"><p>1</p></td><td class="cell"><p>0       0     100.00% 100.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>II</p></td><td class="cell"><p>13 19</p></td><td class="cell"><p>8</p></td><td class="cell"><p>11      5     61.54% 42.11%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>JJ</p></td><td class="cell"><p>4 3</p></td><td class="cell"><p>3</p></td><td class="cell"><p>0       1     75.00% 100.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Total</p></td><td class="cell"><p>2328 2328</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Doc Type Gold Total   TP     FP FN</p></td><td class="cell"><p>Recall Precision</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>A 0 10 10</p></td><td class="cell"><p>0.00% 0.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>B 0 10 10</p></td><td class="cell"><p>0.00% 0.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>C          1       0       0       0 1</p></td><td class="cell"><p>0.00% 0.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>D         2       0       0       0 2</p></td><td class="cell"><p>0.00% 0.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>E         10      11       0       11 10</p></td><td class="cell"><p>0.00% 0.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>F 0 10 10</p></td><td class="cell"><p>0.00% 0.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>G 0 10 10</p></td><td class="cell"><p>0.00% 0.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>J          0       2       0       2 0</p></td><td class="cell"><p>0.00% 0.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>K         0      14      0       14 0</p></td><td class="cell"><p>0.00% 0.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>L         26      10       4        6 22</p></td><td class="cell"><p>15.38% 40.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Free Text 2767 2493   2471    22 296</p></td><td class="cell"><p>89.30% 99.12%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>M 0 10 10</p></td><td class="cell"><p>0.00% 0.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>N          5       26       3       23 2</p></td><td class="cell"><p>60.00% 11.54%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>O          2       2        2        0 0</p></td><td class="cell"><p>100.00% 100.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>P         176    521     167     354 9</p></td><td class="cell"><p>94.89% 32.05%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Q         57      0        0        0 57</p></td><td class="cell"><p>0.00% 0.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>S 5 10 15</p></td><td class="cell"><p>0.00% 0.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>T          0       3        0        3 0</p></td><td class="cell"><p>0.00% 0.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>U 0 10 10</p></td><td class="cell"><p>0.00% 0.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>V          1        0        0        0 1</p></td><td class="cell"><p>0.00% 0.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>W          0       3        0        3 0</p></td><td class="cell"><p>0.00% 0.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Y 0 10 10</p></td><td class="cell"><p>0.00% 0.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Z          0       7        0        7 0</p></td><td class="cell"><p>0.00% 0.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>AA       14      0       0       0 14</p></td><td class="cell"><p>0.00% 0.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CC        10 0 0 1</p></td><td class="cell"><p>0.00% 0.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>DD       50      10       1        9 49</p></td><td class="cell"><p>2.00% 10.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>EE         2      16      0       16 2</p></td><td class="cell"><p>0.00% 0.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>FF         1       0       0       0 1</p></td><td class="cell"><p>0.00% 0.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>II         3       10 13</p></td><td class="cell"><p>0.00% 0.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>JJ          5        110 4</p></td><td class="cell"><p>20.00% 100.00%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Total    3128 3128</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></appendix></body></article>