<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="827"/><title>Bootstrapping Information Extraction from Field Books</title><pubinfo>Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning,pp. 827-836, Prague, June 2007. ©2007 Association for Computational Linguistics</pubinfo><author surname="Canisius" givenname="Sander"><org  name="Tilburg University" country="The Netherlands" city="Tilburg"/></author><author surname="Sporleder" givenname="Caroline"><org  name="Tilburg University" country="The Netherlands" city="Tilburg"/></author></firstpageheader><frontmatter><p><b>Bootstrapping Information Extraction from Field Books</b></p><p><b>Sander Canisius </b>and <b>Caroline Sporleder</b></p><p>ILK / Communication and Information Sciences Tilburg University, P.O. Box 90153, 5000 LE Tilburg, The Netherlands {S.V.M.Canisius,C.Sporleder}@uvt.nl</p></frontmatter><abstract>We present two machine learning ap­proaches to information extraction from semi-structured documents that can be used if no annotated training data are available, but there does exist a database filled with information derived from the type of docu­ments to be processed. One approach em­ploys standard supervised learning for infor­mation extraction by artificially constructing labelled training data from the contents of the database. The second approach com­bines unsupervised Hidden Markov mod­elling with language models. Empirical evaluation of both systems suggests that it is possible to bootstrap a field segmenter from a database alone. The combination of Hid­den Markov and language modelling was found to perform best at this task. </abstract></header><body><section number="1" title="Introduction"><p>Over the past decades much textual data has be­come available in electronic form. Many text types are inherently more or less structured, for example, classified advertisements for appartments, medical records, or logs of archaeological finds or zoological specimens collected during expeditions. Such doc­uments consist of a number of shorter texts (or <i>en­tries), </i>each describing an individual object (e.g., an appartment, or an archaeological find) or event (e.g., a patient presenting to a health care provider). These descriptions in turn typically consist of different seg­ments (or <i>fields) </i>which contain information of a specific type drawn from a more or less given inven­tory. Example (1), for instance, shows two descrip­tions of zoological specimens (a snake and three frogs) collected during an expedition. The descrip­tions contain different segments giving information about the specimens and the circumstances of their collection. For example, in the first description, <i>Lep-tophis </i>and <i>ahaetulla </i>refer, respectively, to the genus and species of the specimen, <i>road to Overtoom </i>men­tions the place of collection, <i>in bush above water </i>en­codes information about the biotope, <i>in the process of eating Hyla minuta </i>is a remark about the circum­stances of collection, <i>16-V-1968 </i>gives the collection date and <i>RMNH15100 </i>the registration number.</p><p>(1) Leptophis ahaetulla, road to Overtoom, in bush</p><doubt alpha="64.3" length="70" tooSmall="False" monospace="0.0">above water in the process of eating Hyla minuta 16-V-1968. RMNH 15100</doubt><p>Hyla minuta 1 $ 2 c? Las Claritas, 9-VI-1978 quak­ing near water 50 cm above water surface, near sec­ondary vegetation, 200 m, M.S. Hoogmoed, RMNH</p><doubt alpha="0.0" length="11" tooSmall="False" monospace="0.0">27217 27219</doubt><p>Unfortunately, this inherent structure is rarely made explicit. While the different object or event descriptions might be indicated by additional white-space or other formatting means, as in the example above, the individual fields within a description are typically not marked in any way. However, knowl­edge of the inherent structure would be very bene­ficial for information extraction and retrieval. For instance, texts in their raw form only allow key word search. To retrieve all entries describing specimens of type <i>Hyla minuta </i>from a zoological field report, one can only search for occurrences of that string anywhere in the document.  This can return false positives, such as the first description in (1) above, which does contain the string but is not about a <i>Hyla minuta </i>specimen but about a specimen of type <i>Lep-tophis ahaetulla </i>(the string <i>Hyla minuta </i>just hap­pens to occur in the special remarks field).<page local="2" global="828"/> On the other hand, if the genus and species information in an entry was explicitly marked, it would be pos­sible to query specifically for entries whose genus is <i>Hyla </i>and whose species is <i>minuta, </i>thus avoiding the retrieval of entries in which this string occurs in another ield.</p><p>The task of automatically finding and labelling segments in object or event descriptions has been referred to as <i>field segmentation </i>(Grenager et al., 2005).<footnote anchor="1"/> It can be seen as a sequence labelling prob­lem, where each text is viewed as a sequence of tokens and the aim is to assign each token a label indicating to what segment the token belongs (e.g., biotope or location). If training data in the form of texts annotated with segment information was readily available, the problem could be approached by training a sequence labeller in a supervised ma­chine learning set-up. However, manually annotated data is rarely available. creating it from scratch is not only time consuming but usually also requires a certain amount of expert knowledge. Moreover, the sequence labeller has to be re-trained for each new domain (e.g., natural history vs. archaeology) and possibly also each sub-domain (e.g., insects vs. mammals) due to the fact that the inventory of ields varies.</p><p>Thus, fully supervised machine learning is not feasible for this task. In this paper, we explore two approaches which require no or only a very small amount of manually labelled training data. Both approaches exploit the fact that there are often re­sources <i>derived </i>from the original documents that can potentially be utilised to bootstrap a sequence labeller in the absence of labelled training data. It is common practice, for example, that information contained in (semi-structured) ield reports or medi­cal records is manually entered into a database, usu­ally in an attempt to make the data more accessible and easier to search. In such databases, each row corresponds to an entry in the original docu­ment (e.g., a zoological specimen) and the database columns correspond to the ields one would like to discern in the original document. Manually convert­ing raw text documents into databases is a labori­ous task though, and it is rather common that the database covers only a small fraction of the objects described in the original texts. The research question we address in this paper is whether it is possible to bootstrap a domain-speciic ield segmentation sys­tem from an existing, manually created database for that domain. Such a system could then be applied to the remaining texts in that domain, which could then be segmented (semi-)automatically and possibly be added to the original database.</p><footnote label="1">The task differs from many other information extraction problems in which the aim is to extract short pieces of relevant information from larger text of largely irrelevant information. Infield segmentation, all or most of the information in the input document is assumed to be relevant and the task is to segment it into fields containing different types of information.</footnote><p>A database does not make perfect training mate­rial for a ield segmenter though, as it is only de­rived from the original document and there are typ­ically signiicant (and sometimes systematic) differ­ences between the two data sources: First, while the ordering of the segments in a semi-structured text document is often not entirely ixed, some or-derings are more likely than others. This informa­tion is lost in the derived databases. Second, the databases may contain information that is not nor­mally present in the underlying text documents, for example information relating to the storage of an object in a collection. conversely, some of the de­tails present in the texts might be omitted from the database, e.g., the special remarks field might be signiicantly shortened in the database. Third, pieces of information are frequently re-written when entered in the database, in some cases these differ­ences may be systematic, e.g., dates, person names, or registration numbers might be written in a differ­ent format. Also, ield boundaries in the text docu­ments are sometimes indicated by punctuation, such as commas, and ields sometimes start with explicit key words, such as <i>collector. </i>Both of these features are missing from the database.</p><p>Despite of this, these databases will provide cer­tain clues about the structure and content of differ­ent segments in the text documents. We exploit this in two different ways: (i) by concatenating database ields to artiicially create annotated training data for a supervised machine learner, and (ii) by using the database to build language models for the ield segmentation task.<page local="3" global="829"/></p></section><section number="2" title="Related Work"><p>Most approaches to ield segmentation and related information extraction tasks, such as illing tem­plates with information about speciic events, have been supervised. Freitag and Kushmerick (2000) combine a pattern learner with boosting to perform ield segmentation in raw texts and in highly struc­tured texts such as web pages and test this approach on a variety of ield segmentation and template ill­ing tasks. Kushmerick et al. (2001) address the prob­lem of extracting contact information from busi­ness cards. They mainly focus on ield <i>labelling, </i>bypassing the segmentation step by assuming that each line on a business card only contains one ield (though a field like address may span several lines). Their method combines a text classiier, for assigning likely labels to each ield, with a trained Hidden Markov Model (HMM) for learning order­ing constraints between ields. Borkar et al. (2001) identify ields in international postal addresses and bibliographic records by nesting HMMs: an outer HMM for modelling ield transitions and a num­ber of inner HMMs for modelling token transitions within fields. Viola and Narasimhand (2005) also deal with address segmentation but employ a trained context-free grammar.</p><p>One of the few unsupervised approaches is pro­vided by Grenager et al. (2005), who perform ield segmentation on bibliographic records and classiied advertisements, using EM to it an HMM to the data. They show that an unconstrained model does not learn the ield structure very well and propose aug­menting the model with a limited amount ofdomain-unspeciic background knowledge, for example, by modifying the transition model to bias it towards recognising larger-scale patterns.</p></section><section number="3" title="Learning Field Segmentation from Databases"><doubt alpha="50.0" length="8" tooSmall="False" monospace="0.0">3.1 Data</doubt><p>We tested our approach on two datasets provided by Naturalis, the Dutch National Museum of Nat­ural History<footnote anchor="2"/> Each dataset consists of (i) a number of ield book entries describing the circumstances under which animal specimens were collected, and (ii) a database containing similar information about the same group of animals but in a more structured form. The latter were used for training, the former for testing. While the databases were manually cre­ated from the corresponding ield books, we made sure that the ield book entries we selected for test­ing did not overlap with the database entries. The two data sets are described below. Table 1 lists the main properties of the data.</p><footnote label="2">http://www.naturalis.nl</footnote><p><b>Reptiles and Amphibians (RA) </b>This dataset de­scribes a number of reptile and amphibian speci­mens. The database consists of 16,670 entries and 41 columns. The columns relate, for example, to the circumstances of a specimen's collection, its taxo-nomic classification, how and where it is stored, who entered the entry into the database and when. Many database cells are empty. Those that are illed come in a variety of format, i.e., numbers, dates, individ­ual words, and free text of various lengths. 22 of the columns contained information that was miss­ing from the ield books, e.g., information relating to the storage of the specimens; these columns were excluded from the experiments.</p><p>From the corresponding ield books, 210 entries were selected randomly and manually annotated with segment information. To test the reliability of the manual annotation, 50 entries were labelled by two annotators. The inter-annotator accuracy on the token level was 92.84% and the kappa .92. The num­ber of distinct field types found in the entries was 19, some of which only occurred in two entries, others occurred in virtually every entry. The average ield length was four tokens, with a maximum average of 21 for the special remarks field, and a minimum of one for fields such as species. The average num­ber of tokens per entry was 60. punctuation marks that did not clearly belong to any ield were labelled as other. In the experiments, 200 entries were used for testing and 10 for parameter tuning.</p><p><b>Pisces </b>The second dataset contains information about the stations where ish specimens were caught. The database consists of 1,375 entries and four columns which provide information on the location of the stations. From the corresponding ield books, we manually labelled 100 entries. compared to the<page local="4" global="830"/></p><p>RA Pisces</p><doubt alpha="28.2" length="39" tooSmall="False" monospace="0.0"># entries in DB            16,670 1,375</doubt><doubt alpha="43.4" length="83" tooSmall="False" monospace="0.0"># fields 19 4 entry length (avg.) 60.17 39.79 segment length (avg.)       4.08 4.75</doubt><table caption="Table 1: Properties of the two datasets"></table><p>first data set, this set is much more regular, with less variation in the number of segments per entry and in the average segment length. The field book entries are also much shorter and there are fewer segments (see Table 1).</p><subsection number="3.2" title="Baselines"><p>In order to get a sense of the difficulty of the task, we implemented five baseline approaches. For the first, <b>Majority (MajB), </b>we always assign the field label that occurs most frequently in the manually labelled test data, namely special remarks. The other four baselines implement different look-up strate­gies, using the database to determine which label should be assigned to a token or token sequence.</p><p><b>Exact (ExactB) </b>looks for substrings in a field book entry which exactly match the content of a database cell and then assigns each token in the matched string the corresponding column label from the database. There are normally several ways to match a field book entry to the database cells; we employed a greedy search, labelling the longest matching substrings first. All tokens that could not be matched in this way were assigned the label</p><p>OTHER.</p><p><b>Unigram (UniB) </b>assigns each token the column label of the database cell in which it occurs most frequently. If a token is not found in the database, it is assigned the label other.</p><p><b>Trigram (TriB) </b>assigns each token the most fre­quent column label of the trigram centred on it. If a trigram is not found in the database, the baseline backs off to the two bigrams covering the token and then to the unigram. If the token is not found in the database, other is assigned.</p><p><b>Trigram+Voting (TriB+Vote) </b>is based on a tech­nique proposed by Van den Bosch and Daelemans (2005) for sequence labelling tasks. The main idea is to assign labels to trigrams in the sequence using a sliding window. Because each token, except the boundary tokens, is contained in three different trigrams (i.e., the one centred on the token to its left, the one centred on itself, and the one centred on the token to its right), each token gets three labels as­signed to it, over which voting can be performed. In our case the labels are assigned by database look-up. If a trigram is not found in the database, no label is assigned to it. If the labels assigned to a given token differ, majority voting is used to resolve the conflict. If this does not break the tie (i.e., because all three trigrams assign different labels), the label of the tri-gram that occurs most frequently in the database is assigned. We also implemented two post-processing rules: (i) turning the label other between two iden­tical neighbouring labels into the surrounding labels, and (ii) labelling commas as other if the neigh­bouring labels are not identical.</p><p><b>3.3   Supervised Learning from Automatically Generated Training Data</b></p><p>Our first strategy was to automatically generate training data for a supervised machine learner from the database. since the rows in the database corre­spond to field book entries and the columns corre­sponds to the fields that we want to identify, train­ing data can be obtained by concatenating the cells in each database row. The order of the fields in the field book entries is not fixed and this should also be reflected in the artificially generated training data. However, the field sequence is not entirely random, i.e., not all sequences are equally likely. If a small amount of manually annotated data is available, the field transition probabilities can be estimated from this, otherwise the best one can do is to assume uni­form probabilities for all possible orderings. We experimented with both strategies, creating two dif­ferent training sets, one in which the database cells were concatenated randomly with uniform probabil­ities, and another in which the cells were concate­nated to reflect the field ordering probabilities esti­mated from ten entries in the manually labelled de­velopment set.<footnote anchor="3"/> When estimating the field transition probabilities, we computed a probability distribution over the initial fields of an entry as well as the condi­tional probability distributions of a field <i>x </i>following a field <i>y </i>for all seen segment pairs in the ten entries.<page local="5" global="831"/> To account for unobserved events, we used Laplace smoothing.</p><footnote label="3">We found that 10 annotated entries are enough for this pur­pose; the field segmentation results we obtained by estimating the sequence probabilities for the training set from 100 entries were not significantly different. This is probably because the probabilities are only used indirectly, i.e. to bias the field order-ings for the generated training data. If the probabilities were used directly in the model, the amount of manually annotated data would probably matter much more.</footnote><p>The artificially created training data were then converted to a token-based representation in which each token corresponds to an instance to be labelled with the field to which it belongs. On the whole, we had just under 700,000 instances (i.e., tokens) in our training data. We implemented 107 features, falling in three classes:</p><p>• the neighbouring tokens (in a window of 5 cen­tering on the token in focus)</p><p>• the typographic properties of the focus token (word vs. number, capitalisation, number of characters in the token etc.)</p><p>• the tfidf weight of the focus token in its con­text with respect to each of the columns in the database (i.e., the fields)</p><p>The tfidf-based features were computed for a win­dow of three, centering on the token in focus. For all n-grams in this window covering the token in focus (i.e., the trigram, the two bigrams, and the unigram of the focus token), we calculated the <i>tfidf </i>simi­larity with the columns in the database, where the similarity between an n-gram <i>U </i>and a column <i>colx</i>is defined as:</p><p><i>tfidfti,colx = tfti,colx </i>log <i>idfti</i></p><p>The term frequency, tfti&gt;colx is the number of oc­currences of t j in <i>colx </i>divided by the number of oc­currences of all n-grams of length <i>n </i>in <i>colx </i>(0 if the <i>n</i>-gram does not occur in the column). The in­verse document frequency, <i>idfti</i>, is the number of all columns in the database divided by the number of columns containing tj. A high tfidf weight for a given <i>n</i>-gram in a given column means that it fre­quently occurs in that column but rarely in other columns, thus it is a good indicator for that column.</p><p>The training data was then used to train a memory-based machine learner (TiMBL (Daele-mans et al., 2004), default settings, <i>k </i>= 3, numeric features declared) to determine which field each to­ken belongs to.<footnote anchor="4"/></p><footnote label="4">We chose TiMBL because it has been applied successfully</footnote></subsection><subsection number="3.4" title="Hidden Markov Models"><p>Our second approach combines language modelling and Hidden Markov Models (HMMs) (Rabiner, 1989). Hidden Markov Models have been in use for information extraction tasks for a long time. A prob­abilistic model is trained to assign a label, or <i>state </i>to each of a sequence of observations, where both labels and observations are expected to be sequen­tially correlated; hence the popularity of HMMs in natural language processing and information extrac­tion. Recently, a large number of more sophisticated learning techniques have largely replaced HMMs for information extraction; however unlike most of those newer techniques, HMMs offer the advantage of having a well-established unsupervised training procedure: the Baum-Welch algorithm (Baum et al., 1970).</p><p>Training a Hidden Markov Model, whether su­pervised or unsupervised, comes down to estimating three probability distributions.</p><p>1. An initial state distribution n, which models the probability of the first observation of a se­quence to have a certain label.</p><p>2. A state-transition distribution A, modelling the conditional probability of being in a certain state s, given that the previous state was s'.</p><p>3. A state-emission distribution B, which models the conditional probability of observing a cer­tain object <i>o </i>given some state <i>s.</i></p><p>For information extraction tasks, the typical in­terpretation of an <i>observation </i>as referred to above, is that of a token, where the entire observation se­quence commonly corresponds to one sentence. In the current study, we chose to apply HMMs on a somewhat higher level, where an observation corre­sponds to a <i>segment </i>of the field book entry. Ideally, one such segment maps one-to-one to a cell in the specimen database, though we leave open the possi­bility ofmerging several segments into one database cell.</p><p>Provided that a field book entry can be segmented reliably, we have turned one part of the learn­ing problem, that of estimating the state-emission</p><p>to sequence labelling tasks (Van den Bosch and canisius, 2006; Van den Bosch and Daelemans, 2005).</p><page local="6" global="832"/><p>distribution, into one for which we have (almost) perfect supervised training data: the contents of the database cells. The general form of a Hid­den Markov Model's state-emission distribution is <i>P</i>(o|s), where <i>s </i>is the state, i.e. a field type in our case, and <i>o </i>is the observation. As mentioned be­fore, we treat a segment of tokens as one observa­tion, therefore our state-emission distribution will look like <i>P(o = t\,t2,...,t</i><b><i>nl</i></b><i>s). </i>Essentially, what we have here is a language model, conditioned on the current state. Since the specimen database pro­vides a large amount of labelled segment sequences, any probabilistic language modelling method can be used to estimate the state-emission distribution.</p><p>Whereas the specimen database provides suffi­cient information to estimate the state-emission dis­tribution in a fully supervised way, the initial-state and state-transition distributions cannot be derived from the database alone. Columns in a database are either unordered or ordered in a way that does not necessarily reflect the order they had in the field book entries they were extracted from. However, the original field book entries do show a rather system­atic structure. Often, using information about the order fields typically occur in, seems to be the only way to distinguish certain field types from one an­other. To estimate the two missing probability dis­tributions, the Baum-Welch algorithm was used, up­dating the initial-state and state-transition distribu­tions, while keeping the state-emission distributions unchanged.</p><subsubsection number="3.4.1" title="Segmentation of Field Book Entries"><p>In our setup, the Hidden Markov Model expects the input texts to be pre-segmented. To come up with a good initial segmentation of an input entry, we again chose a language-modelling approach. It is expected that segment boundaries can best be recog­nised by looking for unusual token subsequences; that is, token sequences that are highly unlikely to occur within a field according to the information we obtained from the specimen database about what a typical segment does look like. A bigram language model has been trained on the contents of <i>all </i>the columns of the specimen database. Using this lan­guage model and the Viterbi algorithm, the globally most-likely segmentation of the input text is pre­dicted.</p></subsubsection><subsubsection number="3.4.2" title="The State-emission Model"><p>The state-emission model is constructed by train­ing a separate bigram language model for each col­umn of the specimen database. Combining those gives us the conditional distribution required for a Hidden Markov Model. However, in a database, not every column has necessarily been filled for every record. For example, in the Reptiles and Amphib­ians database, there are columns that only contain actual data as infrequently as in 5% of the records. Relative to columns that contain data more often, these sparsely-filled columns tend to be overesti­mated when simply computing a likelihood accord­ing to the language model. For this reason, a penalty term is added to the state-emission distribution cor­responding to the probability that a record contains data for the given column. The likelihood com­puted by the language model and the corresponding penalty term are then simply multiplied.</p></subsubsection><subsubsection number="3.4.3" title="Language Modelling"><p>For building both types of language model pre­sented in the two previous sections, we used <i>n-</i>gram language modelling as implemented by the SRI Language Modelling Toolkit (Stolcke, 2002). With this toolkit, high-order <i>n</i>-gram models can be built, where the sparsity problem often encountered with such models is tackled by various smoothing methods. We supplemented this built-in <i>n</i>-gram smoothing, with our own smoothing on the token level by replacing low-frequent words with symbols reflecting certain orthographic features of the origi­nal word, and numbers with a symbol only encoding the number of digits in the original number.</p><p>In addition to these general measures to deal with sparsity, we also applied a small number of knowledge-driven modifications to the training data for the language models. The need for those is caused by the fact that the contents of the specimen database are almost, but not entirely extracted liter­ally from the original field book entries. For exam­ple, for the second entry ofExample 1, the following information is stored in the database.</p><p><b>Genus </b>Hyla <b>Species </b>minuta <b>Gender </b>1f + 2 m <b>Place </b>Las Claritas<page local="7" global="833"/></p><doubt alpha="60.9" length="23" tooSmall="False" monospace="0.0">Collection date9-6-1978</doubt><p><b>Biotope </b>quaking near water 50 cm above water sur­face, near secondary vegetation <b>Height </b>200 m <b>Collector </b>M.S. Hoogmoed <b>Registration number </b>27217 27219</p><p>Comparing just this single field book entry with its corresponding database record, one can already see several mismatches. The gender symbols and $ in the field book texts are stored as m and f in the database. The collection date 9-VI-1978, has been converted to 9-6-1978 before adding it to the database, i.e. the Roman numeral for the month number has been mapped to the corresponding Ara­bic numeral. As a final example, in the field book en­try the registration number for the specimen is pre­ceded by the symbol RMNH; in the database this stan­dard symbol is stripped of and only the number is stored. Each of these differences, while only small, will hinder the performance of a language model trained on the contents of the database and applied to field book texts.</p><p>As a simple illustration of this, when encounter­ing the symbol RMNH in a field book entry, this most likely indicates the start of a new (registration num­ber) segment. However, in the database, on which all language models are trained, RMNH never occurs as a symbol in the registration number column; it does occur a few times in the column for special re­marks but never at the start of the text. As a result, a segmentation model trained on the contents of the database, on encountering the symbol RMNH will al­ways opt for continuing the existing segment as op­posed to starting a new one, which is most likely the better choice.</p><p>Fortunately, many such mismatches between the text in field books and the database are systematic and can easily be covered by a small number of man­ually constructed rules that modify the training data for the language models. Among others, we added the RMNH symbol in front of registration numbers, and randomly changed some month numbers from Arabic numerals to Roman numerals.</p><p>Another difference between the field books and the database that turned out to be rather crucial is the fact that many segments in the field book en­tries are separated by commas. Such commas used</p><p>Table 2: Performance of all baseline and learning approaches on the Reptiles and Amphibians data, expressed in token accuracy, precision, recall, F-score, and WindowDiff. For WindowDiff, lower scores are better.</p><p>as delimiters <i>between fields </i>do not appear in the database, where fields correspond to columns and boundaries between fields consequently do not have to be explicitly marked by punctuation. For exam­ple, the comma between Las Claritas and 9-VI-1978 only serves to separate the <i>Place </i>segment from the <i>Collection date </i>segment; the comma is not copied to the database. However, commas do occur <i>field-internally </i>in the database, especially in longer fields such as special remarks. Hence a language model trained on the database in its original form will never have encountered a comma functioning as a segment boundary marker and thus will not recog­nise that commas may be used for this purpose in the field book entries. To deal with this, we modified the training data for the segment model by randomly in­serting commas at the end ofsome segments. Exper­imental results point out that this modification has a large impact on the performance of the segmentation model.</p></subsubsection></subsection><subsection number="3.5" title="Results and Discussion"><p>To evaluate the performance of the two approaches, we applied them to the Reptiles and Amphibians database. First we computed baseline scores using the approaches described in Section 3.2. All result­ing scores are listed in Table 2.</p><p>Performance of the systems was measured us­ing a number of different metrics, each reflecting different qualities of the output. The most basic one, token accuracy, simply measures the percent­age of tokens that were assigned the correct field type.<page local="8" global="834"/> It has the disadvantage that it does not reflect the quality of the segments that were found. For a more segment-oriented evaluation, we used preci­sion, recall and F-score on correctly identified and labelled segments. As a last measure for segmen­tation quality we used WindowDiff (Pevzner and Hearst, 2002), which only evaluates segment bound­aries not the labels assigned to them. In comparison with F-score, it is more forgiving with respect to seg­ment boundaries that are slightly off.</p><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Token Acc.</p></td><td class="cell"><p>Prec.</p></td><td class="cell"><p>Segment Rec.</p></td><td class="cell"><p>F&gt;=i</p></td><td class="cell"><p>WDiff</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MajB</p></td><td class="cell"><p>24.8</p></td><td class="cell"><p>0.0</p></td><td class="cell"><p>0.0</p></td><td class="cell"><p>0.0</p></td><td class="cell"><p>.346</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ExactB</p></td><td class="cell"><p>16.0</p></td><td class="cell"><p>25.7</p></td><td class="cell"><p>23.1</p></td><td class="cell"><p>24.3</p></td><td class="cell"><p>.425</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>UniB</p></td><td class="cell"><p>27.0</p></td><td class="cell"><p>8.9</p></td><td class="cell"><p>22.8</p></td><td class="cell"><p>12.8</p></td><td class="cell"><p>.818</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>TriB</p></td><td class="cell"><p>43.8</p></td><td class="cell"><p>12.9</p></td><td class="cell"><p>24.8</p></td><td class="cell"><p>16.9</p></td><td class="cell"><p>.582</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>TriB+Vote</p></td><td class="cell"><p>45.1</p></td><td class="cell"><p>14.9</p></td><td class="cell"><p>27.8</p></td><td class="cell"><p>19.4</p></td><td class="cell"><p>.536</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MBL rand.</p></td><td class="cell"><p>44.6</p></td><td class="cell"><p>7.1</p></td><td class="cell"><p>19.2</p></td><td class="cell"><p>10.4</p></td><td class="cell"><p>.568</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MBL bias</p></td><td class="cell"><p>53.4</p></td><td class="cell"><p>12.1</p></td><td class="cell"><p>32.0</p></td><td class="cell"><p>17.6</p></td><td class="cell"><p>.533</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>HMM</p></td><td class="cell"><p>56.9</p></td><td class="cell"><p>62.7</p></td><td class="cell"><p>58.1</p></td><td class="cell"><p>60.3</p></td><td class="cell"><p>.177</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>The baseline performance scores support our as­sumption that the contents of the database can be used to learn how to segment and label field book entries, i.e. the increasingly more sophisticated database matching strategies each cause a substan­tial performance improvement up to 45.1 token ac­curacy for the trigram lookup with voting strategy. The biggest problem of all baseline approaches is that their performance with respect to the segment-oriented measures is disappointing. Even trigram lookup with voting only reaches an F-score of 19.4.</p><p>Looking at the performance of the two memory-based learners in Table 2 <i>(MBL rand. </i>was trained on randomly concatenated training data, <i>MBL bias </i>on data modelled after 10 training sequences), we see that the small amount of prior knowledge used for generating the artificial training data results in a sub­stantial improvement compared with the memory-based learner that was trained on randomly concate­nated training data with uniform probabilities.</p><p>As can be seen in the last row of Table 2, the Hid­den Markov Model outperforms all other approaches in all aspects; it attains both the best token accuracy (56.9), and by far the best F-score (60.3). The most probable explanation for the superior performance of the HMM-based approach is that this approach models sequential constraints between different seg­ments, whereas the baselines and the memory-based models are predominantly local.</p><p>In Table 3, we consider the effect that the knowledge-based rewriting rules discussed in Sub­section 3.4.3 have on the performance of both the segmentation and the labelling step. We evaluate both (i) the performance of the two processing steps separately—for labelling this presupposes perfectly segmented input— and (ii) the performance of the cascade of segmentation and labelling. As before, the performance of the labelling and the cascade is expressed in F-score on segments. Performance of the segmentation is measured in F-score on inserted segment <i>boundaries.</i></p><p>The first row of the table shows the scores if no modification rules are used. This proves detrimen­tal for the segmentation, only attaining an F-score of 28.4. With 62.3, the F-score for labelling is rea­sonable; however, the weakness ofthe segmentation causes the output of the entire cascade to be use­less. Modifying the training data for the segmen­tation model by randomly inserting a comma at the end of segments gives a substantial improvement in segmentation performance, and as a result the qual­ity of the cascade improves with it, as can be seen in the second row. All remaining rows list the scores of a single modification rule applied to the training data <i>in addition to </i>the comma rule. Each of the rules gives a slight performance increase. Using all rules together makes a big difference: the F-score of the cascade increases from 44.7 with the comma rule only, to 60.3.</p><p>Table 3: The effect of systematically modifying the training data for both the segmentation and labelling models. The comma rule is used only in the train­ing of the segmentation model. The other rules are named after the database field they are applied to. The scores reflect their performance when applied in conjunction with the comma rule.</p><p>To confirm that the HMM-based approach carries over to other datasets, we also tested it on the Pisces data. The results of this experiment, as well as all baseline scores are presented in Table 4.<footnote anchor="5"/> The fact that this data set is more regular and contains fewer segments is reflected by the relatively high token ac­curacies attained by the baseline approaches. With the simple database lookup strategies, however, al­most no entire segments are predicted correctly.<page local="9" global="835"/> At­taining a token accuracy of 94.4 and an F-score of 86.9, the performance of the Hidden Markov Model is again more than satisfactory, confirming the re­sults observed with the Reptiles and Amphibians data.</p><footnote label="5">We did not test the memory-based approaches as these led to significantly worse results than the HMM-based model on the Reptiles and Amphibians data set.</footnote><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Rule</p></td><td class="cell"><p>Boundaries</p></td><td class="cell"><p>Labels</p></td><td class="cell"><p>Cascade</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>None</p></td><td class="cell"><p>28.4</p></td><td class="cell"><p>62.3</p></td><td class="cell"><p>17.2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Comma</p></td><td class="cell"><p>69.7</p></td><td class="cell"><p>62.3</p></td><td class="cell"><p>44.7</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Comma+Collection Date</p></td><td class="cell"><p>69.7</p></td><td class="cell"><p>64.4</p></td><td class="cell"><p>46.8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Comma+Reg. Number</p></td><td class="cell"><p>72.9</p></td><td class="cell"><p>68.6</p></td><td class="cell"><p>50.9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Comma+Gender</p></td><td class="cell"><p>71.5</p></td><td class="cell"><p>65.0</p></td><td class="cell"><p>49.5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Comma+Collector</p></td><td class="cell"><p>70.4</p></td><td class="cell"><p>65.8</p></td><td class="cell"><p>45.3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>All</p></td><td class="cell"><p>72.0</p></td><td class="cell"><p>78.3</p></td><td class="cell"><p>60.3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Table 4: Performance of Hidden Markov Model and all baseline approaches on the Pisces data, expressed in token accuracy, precision, recall, F-score, and WindowDiff. For WindowDiff, lower scores are bet­ter.</p></subsection></section><section number="4" title="Conclusion"><p>Information extraction is often used to automate the process of filling a structured database with content extracted from written texts. Supervised machine learning approaches have been successfully applied for creating systems capable ofperforming this task. However, the supervised nature of these approaches requires large amounts of annotated training data; the acquisition of which is often a laborious and time-consuming process. In this study, we experi­mented with two machine learning techniques that do not require such annotated training data, but can be trained on a database containing information de­rived from the type ofdocuments targeted by the ap­plication.</p><p>The first approach is an attempt to employ a stan­dard supervised machine learning algorithm, train­ing it on artificial labelled training data. These data are created by concatenating the contents of the cells of the database records in random order. Experi­ments with this approach pointed out that truly ran­dom concatenation of database fields results in weak performance; a rather simple baseline approach, which only matches substrings of a field book en­try with the contents of the database, leads to better results. However, if a small amount of annotated field book entries is available —in this study, 10 en­tries turned out to be sufficient— one can estimate field ordering probabilities that can be used to gener­ate more realistic training data from the database. A machine learner trained on these data labelled 10% more tokens correctly than the system trained on the randomly generated data.</p><p>Our second approach is based on unsupervised Hidden Markov modelling. First, an n-gram lan­guage model is used to divide the field book en­tries into unlabelled segments. Then, a Hidden Markov Model is trained on these segmented entries using the Baum-Welch algorithm to estimate state­transition probabilities. The resulting HMM labels the segments found in the preceding segmentation step. The HMM state-emission distributions are es­timated by training n-gram language models on the contents of the database columns.</p><p>The performance of the HMM proved to be supe­rior to the other approaches, outperforming the su­pervised learner by labelling 56.9% of the tokens correctly, as well as attaining good results in terms of segment-level F-score (60.3). Experiments with the HMM approach on a second, independent data set confirmed its generality.</p></section><section title="Acknowledgements"><p>The research reported in this paper was funded by the Netherlands Organisation for Scientific Re­search (NWO) as part of the IMIX and CATCH pro­grammes. We are grateful to Antal van den Bosch, Marieke van Erp, Steve Hunt, and the staff at Natu­ralis, the Dutch National Museum for Natural His­tory, for interesting discussions and help in prepar­ing the data.</p><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Token Acc.</p></td><td class="cell"><p>Prec.</p></td><td class="cell"><p>Segment Rec.</p></td><td class="cell"><p></p></td><td class="cell"><p>WDiff</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>MajB</p></td><td class="cell"><p>50.0</p></td><td class="cell"><p>0.0</p></td><td class="cell"><p>0.0</p></td><td class="cell"><p>0.0</p></td><td class="cell"><p>.279</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ExactB</p></td><td class="cell"><p>9.3</p></td><td class="cell"><p>0.0</p></td><td class="cell"><p>0.0</p></td><td class="cell"><p>0.0</p></td><td class="cell"><p>.565</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>UniB</p></td><td class="cell"><p>53.7</p></td><td class="cell"><p>1.5</p></td><td class="cell"><p>3.6</p></td><td class="cell"><p>2.1</p></td><td class="cell"><p>.588</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>TriB</p></td><td class="cell"><p>68.6</p></td><td class="cell"><p>0.2</p></td><td class="cell"><p>0.2</p></td><td class="cell"><p>0.2</p></td><td class="cell"><p>.359</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>TriB+Vote</p></td><td class="cell"><p>67.1</p></td><td class="cell"><p>2.2</p></td><td class="cell"><p>2.8</p></td><td class="cell"><p>2.4</p></td><td class="cell"><p>.384</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>HMM</p></td><td class="cell"><p>94.4</p></td><td class="cell"><p>87.6</p></td><td class="cell"><p>86.3</p></td><td class="cell"><p>86.9</p></td><td class="cell"><p>.049</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><references><p>Leonard E. Baum, Ted Petrie, George Soules, and Nor­man Weiss. 1970. A Maximization Technique Occur­ring in the Statistical Analysis of Probabilistic Func­tions of Markov Chains. <i>The Annals of Mathematical Statistics, </i>41(1):164-171.</p><p>Vinayak Borkar, Kaustubh Deshmukh, and Sunita Sarawagi. 2001. Automatic segmentation of text into structured records. In <i>Proceedings of the 2001 ACM</i> <i>SIGMOD International Conference on Management of Data, </i>pages 175-186.</p><page local="10" global="836"/><p>Walter Daelemans, Jakub Zavrel, Ko Van der Sloot, and Antal Van den Bosch, 2004. <i>TiMBL: Tilburg Memory Based Learner, version 5.1, Reference Guide. </i>ILK Re­search Group Technical Report Series no. 04-02.</p><p>Dayne Freitag and Nicholas Kushmerick. 2000. Boosted wrapper induction. In <i>Proceedings ofthe 17th National Conference on Artificial Intelligence (AAAl/JAAl-2000), </i>pages 577-583.</p><p>Trond Grenager, Dan Klein, and Christopher D. Man­ning. 2005. Unsupervised learning of field segmen­tation models for information extraction. In <i>Proceed­ings of the 43nd Annual Meeting of the Association for Computational Linguistics (ACL 2005), </i>pages 371­378.</p><p>Nicholas Kushmerick, Edward Johnston, and Stephen McGuinness. 2001. Information extraction by text classification. In <i>Proceedings of the 1JCA1-01 Work­shop on Adaptive Text Extraction and Mining.</i></p><p>Lev Pevzner and Marti A. Hearst. 2002. A critique and improvement of an evaluation metric for text segmen­tation. <i>Computational Linguistics, </i>28(1):19-36.</p><p>Lawrence R. Rabiner. 1989. A tutorial on Hidden Markov Models and selected applications in speech recognition. <i>Proceedings ofthe 1EEE, </i>77(2):257-286.</p><p>Andreas Stolcke. 2002. SRILM - an extensible language modeling toolkit. <i>Proceedings ofthe 1nternational Conference on Spoken Language Processing (1CSLP 2002), </i>2:901-904.</p><p>Antal Van den Bosch and Sander Canisius. 2006. Improved morpho-phonological sequence processing with constraint satisfaction inference. In <i>Proceed­ings ofthe Eighth Meeting ofthe ACL Special 1nterest Group in Computational Phonology (S1GPHON '06)</i>.</p><p>Antal Van den Bosch and Walter Daelemans. 2005. Im­proving sequence segmentation learning by predicting trigrams. In <i>Proceedings ofthe Ninth Conference on Natural Language Learning, CoNLL-2005, </i>pages 80­87.</p><p>Paul Viola and Mukund Narasimhand. 2005. Learning to extract information from semi-structured text using a discriminative context free grammar. In <i>Proceedings of the 28th Annual International ACM </i><i>S1G1R</i><i> Confer­ence on Research and Development in 1nformation Re­trieval, </i>pages 330-337.</p></references></body></article>