<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="238"/><title>A Sequence Alignment Model Based on the Averaged Perceptron</title><pubinfo>Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning,pp. 238-247, Prague, June 2007. ©2007 Association for Computational Linguistics</pubinfo><author surname="Freitag" givenname="Dayne"><org  name="Fair Isaac Corporation" country="USA" city="San Diego"/></author><author surname="Khadivi" givenname="Shahram"><org  name="RWTH Aachen University" country="Germany" city="Aachen"/></author></firstpageheader><frontmatter><p><b>A Sequence Alignment Model Based on the Averaged Perceptron</b></p><p><b>Dayne Freitag</b></p><p>Fair Isaac Corporation 3661 Valley Centre Drive San Diego, CA 92130, USA DayneFreitag@fairisaac.com</p><p><b>Shahram Khadivi</b></p><p>Lehrstuhl für Informatik 6 Computer Science Department RWTH Aachen University D-52056 Aachen, Germany khadivi@cs.rwth-aachen.de</p></frontmatter><abstract>We describe a discriminatively trained se­quence alignment model based on the av­eraged perceptron. In common with other approaches to sequence modeling using per-ceptrons, and in contrast with comparable generative models, this model permits and transparently exploits arbitrary features of input strings. The simplicity of perceptron training lends more versatility than compa­rable approaches, allowing the model to be applied to a variety of problem types for which a learned edit model might be useful. We enumerate some of these problem types, describe a training procedure for each, and evaluate the model's performance on sev­eral problems. We show that the proposed model performs at least as well as an ap­proach based on statistical machine transla­tion on two problems of name translitera­tion, and provide evidence that the combina­tion of the two approaches promises further improvement. </abstract></header><body><section number="1" title="Introduction"><p>Sequence alignment is a problem that crops up in many forms, both in computational linguistics (CL) and in other endeavors. The ability to find an op­timal alignment between two sequences has found application in a number of areas of CL, includ­ing phonetic modeling (Ristad and Yianilos, 1998), name transcription (Huang et al., 2004), and dupli­cate detection or information integration (Bilenko and Mooney, 2003; McCallum et al., 2005). Se­quence alignment is a member of a broader class of problems which we might call <i>sequence transduc­tion, </i>to which one of the core CL challenges, ma­chine translation, belongs.</p><p>Under the assumption that one string (the <i>target) </i>is produced through a series of local edits to another string (the <i>source), </i>and given an edit cost matrix, the optimal sequence of edits can be efficiently com­puted through dynamic programming (Needleman andWunsch, 1970). While the cost matrix tradition­ally has been set by hand, several recent papers have proposed determining edit costs empirically. These proposals arise from a variety oflearning paradigms, including generative models (Ristad and Yianilos, 1998; Bilenko and Mooney, 2003), conditional ran­dom fields (McCallum et al., 2005), maximum-margin methods (Joachims, 2003), and gradient boosting (Parker et al., 2006). While approaches based on generative models support only limited fea­ture engineering, discriminative approaches share the advantage of allowing arbitrary features of the input sequences.</p><p>We describe a new sequence alignment model based on the averaged perceptron (Collins, 2002), which shares with the above approaches the ability to exploit arbitrary features of the input sequences, but is distinguished from them by its relative sim­plicity and the incremental character of its training procedure. The fact that it is an online algorithm makes it straightforward to adapt to a range ofprob-lems. To show this, we evaluate the approach on several different tasks, some of them merely illus­trative, but some with clear practical significance, particularly the problem of named entity transcrip­tion.<page local="2" global="239"/></p></section><section number="2" title="The Algorithm 2.1   The Formalism"><p>Suppose we are given two sequences, sm e X* and <i>tn</i><i> </i>e X*. We desire a real-valued function <i>A</i><i>(s, t) </i>which assigns high scores to pairs <i>s, t </i>with high <i>affinity, </i>where affinity is an application-specific notion (e.g., t is a likely phoneme sequence repre­sented by the letter sequence s). If we stipulate that this score is the sum of the individual scores of a series of edits, we can find the highest-scoring such series through a generalization of the standard edit distance:</p><doubt alpha="27.3" length="11" tooSmall="False" monospace="0.0">A(s\, t\) =</doubt><doubt alpha="42.9" length="28" tooSmall="False" monospace="0.0">( ae,tj(s,i,t,j)+A(s\, tj-1)</doubt><doubt alpha="47.1" length="34" tooSmall="False" monospace="0.0">majJasi,,{s,i,t,j) +A(sj-1,tj) (1)</doubt><doubt alpha="44.8" length="29" tooSmall="False" monospace="0.0">{asi,t3(s,i,t,j)+A(sj-1,tj-1)</doubt><p>with <i>A(0,</i><i> </i>0) = 0. The function <i>aSi</i>;tj <i>(s, </i><i>i,</i><i> </i><i>t, </i><i>j)</i><i> </i>rep­resents the score of substituting <i>t</i><i>j </i>for s»; a£;tj and <i>aSi</i>)£ represent insertion and deletion, respectively. If we assume constant-time computation of primitive edit costs, this recursive definition of <i>A </i>allows us to find the highest scoring series of edits for a given sequence pair in time proportional to the product of their lengths. Note that <i>a </i>is indexed by the charac­ters involved in an edit (i.e., inserting 'e' generally has a different cost than inserting 's'). Note further that the score associated with a particular operation may depend on any features computable from the respective positions in the two sequences.</p><p>In the experiments reported in this paper, we as­sume that each local function <i>a </i>is defined in terms of <i>p</i><i> </i>+ <i>q</i><i> </i>features, <i>{f</i><i>i</i><i>,</i><i> </i><i>•••,</i><i> </i><i>f</i><i>p,</i><i> </i><i>f</i><i>p+i</i><i>,</i><i> </i><i>•••,</i><i> </i><i>f</i><i>p+q</i>}, and that these features have the functional form X* xN«R. In other words, each feature takes a sequence and an index and returns a real value. The first <i>p </i>features are defined over sequences from the source alphabet, while the remaining <i>q </i>are de­fined over the target alphabet.<footnote anchor="1"/> In this paper we use character n-gram indicator features.</p><p>'Of course, features that depend jointly on both sequences may also be ofinterest.</p><p>1: Given a set <i>S</i><i> </i>of source sequences 4: <b>for </b>some number of iterations <b>do</b> 5: <b>for </b>s in S <b>do</b> 6: Pick t, <i>t'</i>, t having higher affinity with s 12: Append <i>a </i>to <i>V</i> 15: Return the mean <i>a </i>from <i>V</i></p><doubt alpha="52.2" length="23" tooSmall="False" monospace="0.0">2:V &lt;—[], an empty list</doubt><doubt alpha="63.6" length="22" tooSmall="False" monospace="0.0">3:a —0,a weight vector</doubt><doubt alpha="35.3" length="17" tooSmall="False" monospace="0.0">7:(e,v)— Aa(s, t)</doubt><doubt alpha="33.3" length="18" tooSmall="False" monospace="0.0">8:&lt;e',v')—Aa(s,t')</doubt><doubt alpha="61.5" length="13" tooSmall="False" monospace="0.0">9:ifV &gt; vthen</doubt><doubt alpha="26.7" length="30" tooSmall="False" monospace="0.0">10:a — a+®(s, t, e)-®(s,t',e')</doubt><doubt alpha="55.6" length="9" tooSmall="False" monospace="0.0">11:end if</doubt><doubt alpha="60.0" length="10" tooSmall="False" monospace="0.0">13:end for</doubt><doubt alpha="60.0" length="10" tooSmall="False" monospace="0.0">14:end for</doubt><p>Table 1: The training algorithm. <i>Aa </i>is the affinity function under model parameters <i>a,</i><i> </i>returning edit sequence <i>e </i>and score <i>v.</i></p><p>The score of a particular edit is a linear combina­tion of the corresponding feature values:</p><doubt alpha="60.0" length="5" tooSmall="False" monospace="0.0">p p+q</doubt><doubt alpha="48.6" length="37" tooSmall="False" monospace="0.0">a(s,i,t,j) =J2®k•fk(s,i)+J2ak•fk(t,j)</doubt><doubt alpha="55.6" length="9" tooSmall="False" monospace="0.0">k=i k=p+i</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(2)</doubt><p>The weights <i>a</i><i>k </i>are what we seek to optimize in or­der to tune the model for our particular application.</p><subsection number="2.2" title="A Perceptron-Based Edit Model"><p>In this section we present a general-purpose exten­sion of perceptron training for sequence labeling, due to Collins (2002), to the problem of sequence alignment. Take <i>a </i>to be a model parameterization, and let <i>Aa</i><i>(s, t) </i>return an optimal edit sequence e, with its score <i>v,</i><i> </i>given input sequences <i>s </i>and <i>t </i>un­der a. Elements of sequence <i>e </i>are character pairs <i>(c</i><i>s</i><i>,c</i><i>t</i>), with <i>C</i><i>s </i><i>e X</i><i>s </i>U <i>{e}</i><i> </i>and <i>c</i><i>t </i><i>e X</i><i>t </i>U {e}, where <i>e </i>represents the empty string. Let <i>$(s, t, e) </i>be a feature vector, having the same dimensional­ity as <i>a,</i><i> </i>for a source, target, and corresponding edit sequence. This feature vector is the sum of feature vectors at each point in <i>e </i>as it is played out along input sequences <i>s </i>and <i>t.</i></p><p>Table 1 shows the basic algorithm. Starting with a zero parameter vector, we iterate through the col­lection of source sequences. For each sequence, we pick two target sequences having unequal affinity with the source sequence (Line 6).<page local="3" global="240"/> If the scores re­turned by our current model (Lines 7 and 8) agree with our ordering, we do nothing. Otherwise, we update the model using the perceptron training rule (Line 10). Ultimately, we return <i>a</i><i> </i>averaged over all datapoint presentations.</p></subsection><subsection number="2.3" title="Training Modes"><p>The algorithm presented in Table 1 does not specify how the two target sequences <i>t</i><i> </i>and <i>t'</i><i> </i>are to be cho­sen in Line 6. The answer to this question depends on the application. There are fundamentally two set­tings, depending on whether or not target strings are drawn from the same set as source strings; we will call the setting in which source and target strings in­habit the same set the <i>affinity </i>setting, and refer to the the case where they form different sets as the <i>trans-duction </i>setting. Here, we sketch four problem sce­narios, two from each setting, and specify a target selection procedure appropriate for each.</p><p><b>Affinity, ranking. </b>The task poses a latent affinity between strings, but we can measure it only indi­rectly. In particular, we can order <i>some </i>of the target sequences according to their affinity with a source sequence <i>s.</i><i> </i>In this case, we train as follows: Order a sample of the target sequences according to this partial order. Let <i>t</i><i> </i>and <i>t'</i><i> </i>be two sequences from this order, such that <i>t</i><i> </i>is ordered higher than <i>t'.</i></p><p><b>Affinity, classification. </b>The sequences in X* can be grouped into classes, and we wish the model to assign high affinity to co-members of a class and low affinity to members of different classes. Train as follows: For each <i>s,</i><i> </i>sample <i>t</i><i> </i>from among its co-members and <i>t'</i><i> </i>from among the members of other classes.</p><p><b>Transduction, ranking. </b>The data is presented as source-target pairs, where each <i>t</i><i> </i>is a transduction of the corresponding <i>s.</i><i> </i>We wish to learn a model which, given a novel <i>s,</i><i> </i>will enable us to rank can­didate transductions. Train as follows: Given <i>s,</i><i> </i>let <i>t</i><i> </i>be the target sequence provided to us. Sample <i>t'</i><i> </i>from among the other target sequences.</p><p><b>Transduction, generation. </b>We are again given source-target pairs. We wish to learn to generate a probable target string, given a novel source string. Train as follows: <i>Generate </i>a t' that is approximately optimal according to the current model. Note that since edit decisions are based in part on (arbitrary)</p><p><b>editing   — STRINGS</b></p><doubt alpha="58.3" length="12" tooSmall="False" monospace="0.0">/s,it_ft,TR-</doubt><doubt alpha="55.0" length="20" tooSmall="False" monospace="0.0">fs,t- ft,R-fs,J,n f$</doubt><doubt alpha="60.0" length="5" tooSmall="False" monospace="0.0">fs,_i</doubt><table caption="Table 2: Features with non-zero value for an exam­ple string pair and a model of order 2."></table><p>features of the target sequence, and since generation involves construction of the target sequence, it is not uncommon for a greedy generator to make edit de­cisions which are locally optimal, but which result several edits later in a partially constructed sequence in which no good edits are available. Thus, the prob­lem of generation does not correspond to a simple recurrence relation like Equation 1. Consequently, we experimented with several heuristic approaches to generation and found that a beam search works well.</p></subsection></section><section number="3" title="Evaluation"><p>To establish the effectiveness of the model, we trained it on a range ofproblems, including instances of each of the four settings enumerated above. Prob­lems ranged from the merely illustrative to a non­trivial application of computational linguistics.</p><subsection number="3.1" title="Feature Construction"><p>Without exception, the features we provide to the algorithm are the same in all experiments. Given a user-specified <i>order </i><i>k,</i><i> </i>we define a Boolean fea­ture for every distinct character gram observed in the data of length <i>k </i>or smaller. Recall that there are two disjoint sets of features, those defined over strings drawn from the source and target alphabets, respec­tively. Given a source string and index, those fea­tures have value 1 whose corresponding grams (of size <i>k </i>or smaller) are observed preceding or follow­ing the index (preceding features are distinct from following ones); given a target string and index, we observe only preceding grams. Although it is pos­sible to observe following grams in the target string in some settings, it is not possible in general (i.e., not when generating strings). We therefore adhere to this restriction for convenience and uniformity. An example will make this clear. In Table 2 we are midway through the conversion of the source string "editing" into the target string "STRINGS".<page local="4" global="241"/> Below the two strings are those gram features which have non-zero value at the indicated cursors. The underbar character encodes on which side ofthe cur­sor a gram is observed. Note that an empty-gram feature, which always tests true, is also included, al­lowing us to experiment with 0-order models.</p></subsection><subsection number="3.2" title="Illustrative Problems"><p>To test the ability of the model to recover known edit affinities, we experimented with a simple artifi­cial problem. Using a large list of English words, we define an edit affinity that is sensitive only to consonants. Specifically, the affinity between two words is the maximum number of consonant self-substitutions, with any substitutions involving the first five consonants counting for five normal substi­tutions. Thus, substituting 'b' for 'b' contributes 5 to the score, substituting 'z' for 'z' contributes 1, while operations other than self-subsitutions, and any op­erations involving vowels, contribute 0.</p><p>One epoch of training is conducted as follows. For each word <i>s </i>in the training set, we choose 10 other words from the set at random and sort these words according to both the true and estimated affin­ity. Let <i>t </i>be the string with highest true affinity; let <i>t' </i>(the decoy) be the string with highest estimated affinity. We performed 3-fold cross-validation on a collection of 33,432 words, in each fold training the model for 5 epochs.<footnote anchor="2"/></p><p>Our performance metric is <i>ranking accuracy, </i>the fraction of target string pairs to which the estimated ranking assigns the same order as the true one. Dur­ing testing, for each source string, we sample at ran­dom 1000 other strings from the hold-out data, and count the fraction of all pairs ordered correctly ac­cording to this criterion.</p><p>A 0-order model successfully learns to rank strings according to this affinity with 99.3% ac­curacy, while ranking according to the unmodified Levenshtein distance yields 76.4%. Table 3 shows the 6 averaged weights with the highest magnitude</p><footnote label="2">Here and in other experiments involving the edit model, the number of epochs was set arbitrarily, and not based on perfor­mance on a development set. Beyond the number of epochs re­quired for convergence, we have not observed much sensitivity in test accuracy to the number of epochs.</footnote><doubt alpha="20.7" length="29" tooSmall="False" monospace="0.0">(d, d): f 61.1 (c, c): f 60.6</doubt><doubt alpha="21.4" length="14" tooSmall="False" monospace="0.0">(g,g): f0 60.3</doubt><doubt alpha="23.1" length="13" tooSmall="False" monospace="0.0">(b,b): f 59.1</doubt><doubt alpha="21.4" length="14" tooSmall="False" monospace="0.0">(f, f): f 57.0</doubt><doubt alpha="23.1" length="13" tooSmall="False" monospace="0.0">(t,t): f 18.6</doubt><p>Table 3: Largest weights in a consonant-preserving edit affinity in which the first five consonants are given 5 times as much weight as others.</p><p>from a model trained on one of the folds. In pre­senting weights, we follow the formatting conven­tion <i>edit: feature. </i>Since the model in question is of order 0, all features in Table 3 are the "empty fea­ture." Note how idempotent substitutions involving the 5 highly weighted consonants are weighted sig­nificantly higher than the remaining operations.</p></subsection><subsection number="3.3" title="Rhyming"><p>While the above problem illustrates the ability of the proposed algorithm to learn latent alignment affinities, it is expressible as a order-0 model. A somewhat more interesting problem is that of mod­eling groups of rhyming words. This problem is an instance of what we called the "classification" scenario in Section 2.3. Because English letters have long since lost their direct correspondence to phonemes, the problem of distinguishing rhyming English words is difficult for a knowledge-lean edit model. What's more, the importance of a letter is dependent on context; letters near the end of a word are more likely to be significant.</p><p>We derived groups of rhyming words from the CMU pronouncing dictionary (CMU, 1995), dis­carding any singleton groups. This yielded 21,396 words partitioned into 3,799 groups, ranging in size from 464 words (nation, location, etc.) down to 2. We then divided the words in this data set at random into three groups for cross-validation.</p><p>Training was conducted as follows. For each word in the training set, we selected at random up to 5 rhyming words and 5 non-rhyming words. These words were ranked according to affinity with the source word under the current model. Let <i>t </i>be the lowest scoring rhyming word, and let <i>t' </i>be the highest-scoring non-rhyming word.</p><page local="5" global="242"/><p>For each word in the hold-out set, we scored and ranked all rhyming words in the same set, as well as enough non-rhyming words to total 1000. We then recorded the precision at the point in this rank­ing where recall and precision are most nearly equal. Our summary statistic is the micro-averaged break­even precision.</p><p>Table 4 presents the performance of the proposed model and compares it with two simple baselines. Not surprisingly, performance increases with in­creasing order. The simple heuristic approaches fare quite poorly by comparison, reflecting the subtlety of the problem.</p></subsection><subsection number="3.4" title="Transcription"><p>Our work was motivated by the problem of named entity transcription. Out-of-vocabulary (OOV) terms are a persistent problem in statistical machine translation. Often, such terms are the names of en­tities, which typically have low corpus frequencies. In translation, the appropriate handling of names is often to transcribe them, to render them idiomati­cally in the target language in a way that preserves, as much as possible, their phonetic structure. Even when an OOV term is not a name, transcribing it preserves information that would otherwise be dis­carded, leaving open the possibility that downstream applications will be able to make use of it.</p><p>The state of the art in name transcription involves some form of generative model, sometimes in com­bination with additional heuristics. The generative component may involve explicitly modeling pho­netics. For example, Knight and Graehl (1998) employ cascaded probabilistic finite-state transduc­ers, one of the stages modeling the orthographic-to-phonetic mapping. Subsequently, Al-Onaizan and Knight (2002) find they can boost perfor­mance by combining a phonetically-informed model</p><p>Table 5: Characteristics ofthe two transcription data sets, Arabic-English (A-E) and Mandarin-English (M-E), including number of training, development, and evaluation pairs <i>(Train, Dev, </i>and <i>Eval), </i>and mean length in characters of English and foreign strings <i>(ELen </i>and <i>FLen).</i></p><p>with one trained only on orthographic correspon­dences. Huang et al. (2004), construct a probabilis­tic Chinese-English edit model as part of a larger alignment solution, setting edit weights in a heuris­tic bootstrapped procedure.</p><p>In rendering unfamiliar written Arabic words or phrases in English, it is generally impossible to achieve perfect performance, because many sounds, such as short vowels, diphthong markers, and dou­bled consonants, are conventionally not written in Arabic. We calculate from our experimental datasets that approximately 25% of the characters in the En­glish output must be inferred. Thus, a character error rate of 25% can be achieved through simple translit­eration.</p><subsubsection number="3.4.1" title="Transcribing names"><p>We experimented with a list of 10,084 personal names distributed by the Linguistic Data Consor­tium (LDC). Each entry in the database includes an arabic name in transliterated ASCII (SATTS method) and its English rendering. The Arabic names appear as they would in conventional writ­ten Arabic, i.e., lacking short vowels and other di­acritics. We randomly segregated 1000 entries for evaluation and used the rest for training. The <i>A-E </i>row in Table 5 summarizes some of this data set's characteristics.</p><p>We trained the edit model as follows. For each training pair the indicated English rendering con­stitutes our true target <i>(t),</i><i> </i>and we use the current model to generate an alternate string <i>(t'),</i><i> </i>updating the model in the event <i>t' </i>yields a higher score than <i>t.</i><i> </i>This was repeated for 10 epochs. We experimented with a model of order 3.</p><p>Under this methodology, we observed a 1-best ac-<page local="6" global="243"/></p><table caption="Table 4: Micro-averaged break-even precision on the task ofgrouping rhyming English words." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Model</i></p></td><td class="cell"><p><i>Precision</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Levenshtein</p></td><td class="cell"><p>0.126</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Longest common suffix</p></td><td class="cell"><p>0.130</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PTEM, Order 0</p></td><td class="cell"><p>0.505</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PTEM, Order 3</p></td><td class="cell"><p>0.790</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Task</i></p></td><td class="cell"><p><i>Train</i></p></td><td class="cell"><p><i>Dev</i></p></td><td class="cell"><p><i>Eval</i></p></td><td class="cell"><p><i>ELen</i></p></td><td class="cell"><p><i>FLen</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>A-E</p></td><td class="cell"><p>8084</p></td><td class="cell"><p>1000</p></td><td class="cell"><p>1000</p></td><td class="cell"><p>6.5</p></td><td class="cell"><p>4.9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>M-E</p></td><td class="cell"><p>2000</p></td><td class="cell"><p>430</p></td><td class="cell"><p>1557</p></td><td class="cell"><p>16.3</p></td><td class="cell"><p>23.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 6: Some of the weights governing the han­dling of thetah marbouta( ) in an order-3 Arabic-"></table><p>English location name transcription model. Buck­walter encoding of Arabic characters is used here for purposes of display. The symbol "&lt;e&gt;" represents end of string.</p><p>curacy of 0.552. It is difficult to characterize the strength of this result relative to those reported in the literature. Al-Onaizan and Knight (2002) report a 1-best accuracy of 0.199 on a corpus of Arabic <i>person </i>names (but an accuracy of 0.634 on <i>English </i>names), using a "spelling-based" model, i.e., a model which has no access to phonetic information. However, the details of their experiment and model differ from ours in a number of respects.</p><p>It is interesting to see how a learned edit model handles ambiguous letters. Table 6 shows the weights of some of the features governing the han­dling of the character <i>(tah marbouta) </i>from exper­iments with Arabic place names. This character, which represents the "t" sound, typically appears at the end of words. It is generally silent, but is spoken in certain grammatical constructions. In its silent form, it is typically transcribed "ah" (or "a"); in its spoken form, it is transcribed "at". The weights in the table reflect this ambiguity and illustrate some of the criteria by which the model chooses the ap­propriate transcription. For example, the negative weight on the feature <i>f</i><i>s,jp&lt;e&gt; </i>inhibits the produc­tion of "t" at the end of a phrase, where "h" is almost always more appropriate. Similarly, "h" is more common following "ya" in the target string (often as part of the larger suffix "iyah"). However, the pre­ceding context "rya" is usually observed in the word "qaryat", meaning "village" as in "the village of ... " In this grammatical usage, the tah marbouta is spo­ken and therefore rendered with a "t". Consequently, the corresponding weight in the "h" interpretation is inhibitory.</p><p>The Al-Onaizan and Knight spelling model can be regarded as a statistical machine translation (SMT) system which translates source language characters to target language characters in the ab­sence of phonetic information. For comparison with state of the art, we used the RWTH phrase-based SMT system (Zens et al., 2005) to build an Arabic-to-English transliteration system. This sys­tem frames the transcription problem as follows. We are given a sequence of source language charac­ters <i>sY </i>representing a name, which is to be trans­lated into a sequence of target language characters tn. Among all possible target language character se­quences, we will choose the character sequence with the highest probability:</p><doubt alpha="41.4" length="29" tooSmall="False" monospace="0.0">%   =   argmax{Pr(tn|sm)} (3)</doubt><p>The posterior probability Pr(tn|sm) is modeled di­rectly using a log-linear combination of several models (Och and Ney, 2002), including a character-based phrase translation model, a character-based lexicon model, a 4-gram character sequence model, a character penalty and a phrase penalty. The first two models are used for both directions: Arabic to English and English to Arabic. We do not use any reordering model because the target character sequence is always monotone with respect to the source character sequence. More details about the baseline system can be found in (Zens et al., 2005).</p><p>We remark in passing that while the perceptron-based edit model is a general algorithm for learn­ing sequence alignments using simple features, the above SMT approach combines several models, some of which have been the subject of research in the fields of speech recognition and machine trans­lation for several years. Furthermore, we made an effort to optimize the performance of the SMT ap­proach on the tasks presented here.</p><p>Table 7 compares this system with the edit model. The difference between the 1-best accuracies of the two systems is significant at the 95% level, using the bootstrap for testing. However, we can improve on both systems by combining them. We segregated 1000 training documents to form a development set, and used it to learn linear combination coefficients over our two systems, resulting in a combined sys­tem that scored 0.588 on the evaluation set—a sta-<page local="7" global="244"/></p><table caption="Table 6: Some of the weights governing the handling of thetah marbouta( ) in an order-3 Arabic-" class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(p, h)</p></td><td class="cell"><p><i>ft,a-</i></p></td><td class="cell"><p>38.5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>(p,</i><i> </i>t)</p></td><td class="cell"><p><i>ft</i><i>,a-</i></p></td><td class="cell"><p>30.8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(P, h)</p></td><td class="cell"><p><i>ft,ya-</i></p></td><td class="cell"><p>11.8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(p, t)</p></td><td class="cell"><p><i>fs,jp&lt;e&gt;</i></p></td><td class="cell"><p>-8.6</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(P, h)</p></td><td class="cell"><p><i>ft,rya-</i></p></td><td class="cell"><p>-12.1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(p, t)</p></td><td class="cell"><p><i>ft,uba-</i></p></td><td class="cell"><p>-14.4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Table 7: 1-best and 5-best transcription accuracies. The successive improvements in 1-best accuracy are significant at the 95% confidence level.</p><p>tistically significant improvement over both systems at the 95% confidence level.</p></subsubsection><subsubsection number="3.4.2" title="Ranking transcriptions"><p>In some applications, instead of transcribing a name in one language into another, it is enough just to rank candidate transcriptions. For example, we may be in possession of comparable corpora in two languages and the means to identify named entities in each. If we can rank the likely transcriptions of a name, we may be able to align a large portion of the transliterated named entities, potentially extend­ing the coverage of our machine translation system, which will typically have been developed using a smaller parallel corpus. This idea is at the heart of several recent attempts to improve the handling of named entities in machine translation (Huang et al., 2004; Lee and Chang, 2003). A core component of all such approaches is a generative model simi­lar in structure to the "spelling" model proposed by Al-Onaizan and Knight.</p><p>When ranking is the objective, we can adopt a training procedure that is much less expensive than the one used for generation. Let <i>t </i>be the correct tran­scription for a source string <i>(s).</i><i> </i>Sample some num­ber of strings at random (200 in the following exper­iments) from among the transcriptions in the training set of strings other than <i>s.</i><i> </i>Let <i>t' </i>be the string having highest affinity with s, updating the model, as usual, if <i>t' </i>scores higher than <i>t.</i></p><p>In addition to the Arabic-English corpus, we also experiment with a corpus distributed by the LDC of full English names paired with their Mandarin spelling. The <i>M-E </i>row of Table 5 summarizes char­acteristics of this data set. Because we are inter­ested in an approximate comparison with similar ex­periments in the literature, we selected at random 2430 for training and 1557 for evaluation, which are the data sizes used by Lee and Chang (2003) for their experiments. In these experiments, the Chinese names are represented as space-separated pinyin without tonal markers.</p><p>Note that this problem is probably harder than the Arabic one, for several reasons. For one thing, the letters in a Mandarin transcription of a foreign name represent syllables, leading to a somewhat lossier rendering of foreign names in Mandarin than in Ara­bic. On a more practical level, this data set is noisier, occasionally containing character sequences in one string for which corresponding characters are lack­ing from its paired string. On the other hand, the Mandarin problem contains full names, rather than name components, which provides more context for ranking.</p><p>We trained the edit model on both data sets us­ing both the sampling procedure outlined above and the self-generation training regime, in each case for 20 epochs, producing models of orders from 1 to 3. However, we found that the efficiency of the phrase-based SMT system described in the previous section would be limited for this task, mainly due to two reasons: the character-based phrase models due to possible unseen phrases in an evaluation corpus, and the character sequence model as all candidate tran­scriptions confidently belong to the target language. Therefore, to make the phrase-based SMT system robust against data sparseness for the ranking task, we also make use of the IBM Model 4 (Brown et al., 1993) in both directions. The experiments show that IBM Model 4 is a reliable model for the ranking task. For each evaluation pair, we then ranked all available evaluation transcriptions, recording where in this list the true transcription fell.</p><p>Table 8 compares the various models, showing the fraction of cases for which the true transcription was ranked highest, and its mean reciprocal rank (MRR). Both the phrase-based SMT model and the edit model perform well on this task. While the best configuration of PTEM out-performs the best SMT model, the differences are not significant at the 95% confidence level. However, compare these perfor­mance scores to those returned by the system ofLee and Chang (2003), who reported a peak MRR of 0.82 in similar experiments involving data different from ours.</p><p>The PTEM rows in the table are separated into<page local="8" global="245"/></p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Model</i></p></td><td class="cell"><p><i>lbest</i></p></td><td class="cell"><p><i>5best</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SMT</p></td><td class="cell"><p>0.528</p></td><td class="cell"><p>0.824</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PTEM, Order 3</p></td><td class="cell"><p>0.552</p></td><td class="cell"><p>0.803</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Linear combination</p></td><td class="cell"><p>0.588</p></td><td class="cell"><p>0.850</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Table 8: Performance on two transcription ranking tasks, showing fraction of cases in which the correct transcription was ranked highest, accuracy (ACC) and mean reciprocal rank of the correct transcription (MRR).</p><p>those in which the model was trained using the same procedure as for generation (PTEM<i>G), </i>and those in which the quicker ranking-specific train­ing regime was used (PTEM<i>R). </i>The comparison is interesting, inasmuch it does not support the con­clusion that one regime is uniformly superior to the other. While generation regime yields the best per­formance on Arabic (using a high-order model), the ranking regime scores best on Mandarin (with a low-order model). When training a model to generate, it seems clear that more context in the form of larger n-grams is beneficial. This is particularly true for Mandarin, where an order-1 model probably does not have the capacity to generate plausible decoys.</p></subsubsection></subsection></section><section number="4" title="Discussion"><p>This paper is not the first to show that perceptron training can be used in the solution of problems involving transduction. Both Liang, et al (2006), and Tillmann and Zhang (2006) report on effective machine translation (MT) models involving large numbers of features with discriminatively trained weights. The training of these models is an in­stance of the "Generation" scenario outlined in Sec­tion 2.3. However, because machine translation is a more challenging problem than name transcrip­tion (larger vocabularies, higher levels of ambiguity, non-monotonic transduction, etc.), our general­purpose approach to generation training may be in­tractable for MT. Instead, much ofthe focus ofthese papers are the heuristics that are required in order to train such a model in this fashion, including feature selection using external resources (phrase tables), staged training, and generating to BLEU-maximal sequences, rather than the reference target.</p><p>Klementiev and Roth (2006) explore the use of a perceptron-based ranking model for the purpose of finding name transliterations across comparable cor­pora. They do not calculate an explicit alignment be­tween strings. Instead, they decompose a string pair into a collection of features derived from charac­ter n-grams heuristically paired based on their loca­tions in the respective strings. Thus, Klementiev and Roth, in common with the two MT approaches de­scribed above, carefully control the features used by the perceptron. In contrast to these approaches, our algorithm discovers <i>latent </i>alignments, essentially selecting those features necessary for good perfor­mance on the task at hand.</p><p>As noted in the introduction, several previous pa­pers have proposed general, discriminatively trained sequence alignment models, as alternatives to the generative model proposed by Ristad and Yianilos. McCallum, et al. (2005), propose a conditional ran­dom field for sequence alignment, designed for the important problem of duplicate detection and infor­mation integration. Comprising two sub-models, one for matching strings and one for non-matching, the model is trained on sequence pairs explicitly labeled "match" or "non-match," and some care is apparently needed in selecting appropriate non-matching strings. It is therefore unclear how this model would be extended to problems involving ranking or generation.</p><p>Joachims (2003) proposes SVM-align, a sequence alignment model similar in structure to that de­scribed here, but which sets weights through di­rect numerical optimization. Training involves ex­posing the model to sequence pairs, along with the correct alignment and some number of "decoy" se­quences. The reliance on an explicit alignment and hand-chosen decoys yields a somewhat less flexi­ble solution than that presented here. It is not clear whether these features of the training regime are in­dispensable, or whether they might be generalized to increase the approach's scope.<page local="9" global="246"/> Note that where di­rectly maximizing the margin is feasible, it has been shown empirically to be superior to perceptron train­ing (Altun et al., 2003).</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Model</i></p></td><td class="cell"><p>C-E Task</p></td><td class="cell"><p>A-E Task</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ACC</p></td><td class="cell"><p>MRR</p></td><td class="cell"><p>ACC</p></td><td class="cell"><p>MRR</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SMT</p></td><td class="cell"><p>0.795</p></td><td class="cell"><p>0.797</p></td><td class="cell"><p>0.982</p></td><td class="cell"><p>0.985</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SMT w/o LM</p></td><td class="cell"><p>0.797</p></td><td class="cell"><p>0.798</p></td><td class="cell"><p>0.983</p></td><td class="cell"><p>0.985</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>IBM_4</p></td><td class="cell"><p>0.961</p></td><td class="cell"><p>0.971</p></td><td class="cell"><p>0.978</p></td><td class="cell"><p>0.987</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SMT + IBM_4</p></td><td class="cell"><p>0.971</p></td><td class="cell"><p>0.977</p></td><td class="cell"><p>0.991</p></td><td class="cell"><p>0.994</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PTEMG, Ord. 1</p></td><td class="cell"><p>0.843</p></td><td class="cell"><p>0.877</p></td><td class="cell"><p>0.959</p></td><td class="cell"><p>0.975</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PTEMG, Ord. 2</p></td><td class="cell"><p>0.970</p></td><td class="cell"><p>0.978</p></td><td class="cell"><p>0.968</p></td><td class="cell"><p>0.980</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PTEMG, Ord. 3</p></td><td class="cell"><p>0.975</p></td><td class="cell"><p>0.982</p></td><td class="cell"><p>0.971</p></td><td class="cell"><p>0.983</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PTEM/?, Ord. 1</p></td><td class="cell"><p>0.961</p></td><td class="cell"><p>0.973</p></td><td class="cell"><p>0.992</p></td><td class="cell"><p>0.995</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PTEM/?, Ord. 2</p></td><td class="cell"><p>0.960</p></td><td class="cell"><p>0.972</p></td><td class="cell"><p>0.989</p></td><td class="cell"><p>0.993</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>PTEM/?, Ord. 3</p></td><td class="cell"><p>0.960</p></td><td class="cell"><p>0.972</p></td><td class="cell"><p>0.989</p></td><td class="cell"><p>0.994</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Parker et al. (2006), propose to align sequences by gradient tree boosting. This approach has the attrac­tive characteristic that it supports a factored repre­sentation of edits (a characteristic it shares with Mc-Callum et al.). Although this paper does not evaluate the method on any problems from computational lin­guistics (the central problem is musical information retrieval), gradient tree boosting has been shown to be an effective technique for other sorts of sequence modeling drawn from computational linguistics (Di-</p><doubt alpha="62.9" length="35" tooSmall="False" monospace="0.0">etterich et al., 2004).5 Conclusion</doubt><p>Motivated by the problem of Arabic-English tran­scription of names, we adapted recent work in per-ceptron learning for sequence labeling to the prob­lem of sequence alignment. The resulting algorithm shows clear promise not only for transcription, but also for ranking of transcriptions and structural clas­sification. We believe this versatility will lead to other successful applications of the idea, both within computational linguistics and in other fields involv­ing sequential learning.</p><p><b>Acknowledgment of support</b></p><p>This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA/IPTO) under Contract HR0011-06-C-0023. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the Defense Advanced Research Projects</p><p>Agency (DARPA).</p></section><references><p>Y. Al-Onaizan and K. Knight. 2002. Machine translit­eration of names in Arabic text. In <i>Proceedings of theACL-02 workshop on computational approaches to semitic languages.</i></p><p>Y. Altun, I. Tsochantaridis, and T. Hofmann. 2003. Hid­den Markov support vector machines. In <i>Proceedings ofICML-2003.</i></p><p>M. Bilenko and R. Mooney. 2003. Adaptive duplicate detection using learnable string similarity measures.</p><p>In <i>Proceedings ofKDD-2003.</i></p><p>P. F. Brown, S. A. Della Pietra, V. J. Della Pietra, and R. L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. <i>Computa­tional Linguistics, </i>19(2), June.</p><p>CMU. 1995. The CMU pronouncing dictionary. http://www.speech.cs.cmu.edu/cgi-bin/cmudict. Ver­sion 0.6.</p><p>M. Collins. 2002. Discriminative training methods for hidden Markov models: theory and experiments with perceptron algorithms.   In <i>Proceedings ofEMNLP-</i> <i>2002.</i><i></i></p><p>T. Dietterich, A. Ashenfelter, and Y. Bulatov. 2004. Training conditional random fields via gradient tree boosting. In <i>Proceedings ofICML-2004.</i></p><p>F. Huang, S. Vogel, and A. Waibel. 2004. Improving named entity translation combining phonetic and se­mantic similarities.  In <i>Proceedings ofHLT-NAACL</i> <i>2004.</i><i></i></p><p>T. Joachims. 2003. Learning to align sequences: a maximum-marginapproach. Technical report, Cornell University.</p><p>A. Klementiev and D. Roth. 2006. Weakly supervised named entity transliteration and discovery from mul­tilingual comparable corpora. In <i>Proceedings ofCol-ing/ACL 2006.</i></p><p>K. Knight and J. Graehl. 1998. Machine transliteration.</p><p><i>Computational Linguistics, </i>24(4).</p><p>C.-J. Lee and J.S. Chang. 2003. Acquisition of English-Chinese transliterated word pairs from parallel-aligned texts using a statistical machine transliteration model.</p><p>In <i>Proceedings ofthe HLT-NAACL 2003 Workshopon</i> <i>Building and Using Parallel Texts.</i><i></i></p><p>P. Liang, A. Bouchard-Cote, D. Klein, and B. Taskar. 2006. An end-to-end discriminative approach to machine translation.    In <i>Proceedings of COLING</i> <i>2006/ACL 2006.</i><i></i></p><p>A. McCallum, K. Bellare, and F. Pereira. 2005. A condi­tional random field for discriminatively-trained finite-state string edit distance. In <i>Proceedings ofUAI-2005.</i></p><p>S.B. Needleman and C.D. Wunsch. 1970. A general method applicable to the search for similarities in the amino acid sequence of two proteins. <i>Journal of Molecular Biology, </i>48.</p><p>F.J. Och and H. Ney. 2002. Discriminative training and maximum entropy models for statistical machine translation. In <i>ACL02, </i>pages 295-302, Philadelphia,</p><p>PA, July.</p><page local="10" global="247"/><p>C. Parker, A. Fern, and P Tadepalli. 2006. Gradient boosting for sequence alignment. In <i>Proceedings of</i> <i>AAAI-2006.</i><i></i></p><p>E.S. Ristad and P.N. Yianilos. 1998. Learning string-edit distance. <i>IEEE Transactions on Pattern Analysis and Machine Intelligence, </i>20.</p><p>C. Tillmann and T. Zhang. 2006. A discriminative global training algorithm for statistical MT. In <i>Proceedings</i> <i>ofColing/ACL 2006.</i><i></i></p><p>R. Zens, O. Bender, S. Hasan, S. Khadivi, E. Matusov, J. Xu, Y. Zhang, and H. Ney. 2005. The RWTH phrase-based statistical machine translationsystem. In <i>Proceedings ofthe International Workshop on Spoken Language Translation (IWSLT), </i>pages 155-162, Pitts­burgh, PA, October.</p></references></body></article>