<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="468"/><title>Smoothed Bloom Filter Language Models: Tera-Scale LMs on the Cheap</title><pubinfo>Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning,pp. 468-476, Prague, June 2007. ©2007 Association for Computational Linguistics</pubinfo><author surname="Talbot" givenname="David"><org  name="University of Edinburgh" country="United Kingdom" city="Edinburgh"/></author><author surname="Osborne" givenname="Miles"><org  name="University of Edinburgh" country="United Kingdom" city="Edinburgh"/></author></firstpageheader><frontmatter><p><b>Smoothed Bloom filter language models: Tera-Scale LMs on the Cheap</b></p><p><b>David Talbot and Miles Osborne</b></p><p>School of Informatics, University of Edinburgh 2 Buccleuch Place, Edinburgh, EH8 9LW, UK d.r.talbot@sms.ed.ac.uk, miles@inf.ed.ac.uk</p></frontmatter><abstract>A Bloom filter (BF) is a randomised data structure for set membership queries. Its space requirements fall significantly below lossless information-theoretic lower bounds but it produces false positives with some quantifiable probability. Here we present a general framework for deriving smoothed language model probabilities from BFs. We investigate how a BF containing n-gram statistics can be used as a <i>direct replacement </i>for a conventional n-gram model. Recent work has demonstrated that corpus statistics can be stored efficiently within a BF, here we consider how smoothed language model probabilities can be derived efficiently from this randomised representation. Our pro­posal takes advantage of the one-sided error guarantees of the BF and simple inequali­ties that hold between related n-gram statis­tics in order to further reduce the BF stor­age requirements and the error rate of the derived probabilities. We use these models as replacements for a conventional language model in machine translation experiments. </abstract></header><body><section number="1" title="Introduction"><p>Language modelling (LM) is a crucial component in statistical machine translation (SMT). Standard n-gram language models assign probabilities to trans­lation hypotheses in the target language, typically as smoothed trigram models (Chiang, 2005). Al­though it is well-known that higher-order language models and models trained on additional monolin­gual corpora can significantly improve translation performance, deploying such language models is not trivial. Increasing the order of an n-gram model can result in an exponential increase in the number of parameters; for the English Gigaword corpus, for instance, there are 300 million distinct trigrams and over 1.2 billion distinct five-grams. Since a language model is potentially queried millions of times per sentence, it should ideally reside locally in memory to avoid time-consuming remote or disk-based look­ups.</p><p>Against this background, we consider a radically different approach to language modelling. Instead of explicitly storing all distinct n-grams from our corpus, we create an implicit randomised represen­tation of these statistics. This allows us to drastically reduce the space requirements of our models. In this paper, we build on recent work (Talbot and Os-borne, 2007) that demonstrated how the <i>Bloom filter </i>(Bloom (1970); BF), a space-efficient randomised data structure for representing sets, could be used to store corpus statistics efficiently. Here, we propose a framework for deriving smoothed n-gram models from such structures and show via machine trans­lation experiments that these <i>smoothed Bloom filter language models </i>may be used as direct replacements for standard n-gram models in SMT.</p><p>The space requirements of a Bloom filter are quite spectacular, falling significantly below information-theoretic error-free lower bounds. This efficiency, however, comes at the price of <i>false positives: </i>the fil­ter may erroneously report that an item not in the set is a member. False negatives, on the other hand, will never occur:<page local="2" global="469"/> the error is said to be <i>one-sided. </i>Our framework makes use of the <i>log-frequency Bloom filter </i>presented in (Talbot and Osborne, 2007), and described briefly below, to compute smoothed con­ditional n-gram probabilities on the fly. It takes advantage of the one-sided error guarantees of the Bloom filter and certain inequalities that hold be­tween related n-gram statistics drawn from the same corpus to reduce both the error rate and the compu­tation required in deriving these probabilities.</p></section><section number="2" title="The Bloom filter"><p>In this section, we give a brief overview of the Bloom filter (BF); refer to Broder and Mitzenmacher (2005) for a more in detailed presentation. A BF rep­resents a set S = (xi, x2,xn} with n elements drawn from a universe U of size N. The structure is attractive when N » n. The only significant stor­age used by a BF consists of a bit array of size m. This is initially set to hold zeroes. To train the filter we hash each item in the set k times using distinct hash functions h1, h2,hk. Each function is as­sumed to be independent from each other and to map items in the universe to the range 1 to m uniformly at random. The k bits indexed by the hash values for each item are set to 1; the item is then discarded. Once a bit has been set to 1 it remains set for the life­time of the filter. Distinct items may not be hashed to k distinct locations in the filter; we ignore col-lisons. Bits in the filter can, therefore, be <i>shared </i>by distinct items allowing significant space savings but introducing a non-zero probability of false positives at test time. There is no way of directly retrieving or ennumerating the items stored in a BF.</p><p>At test time we wish to discover whether a given item was a member of the original set. The filter is queried by hashing the test item using the same k hash functions. If all bits referenced by the k hash values are 1 then we <i>assume </i>that the item was a member; if any of them are 0 then we <i>know </i>it was not. True members are always correctly identified, but a false positive will occur if all k corresponding bits were set by other items during training and the item was not a member of the training set.</p><p>The probability of a false postive, f, is clearly the probability that none of k randomly selected bits in the filter are still 0 after training. Letting p be the proportion of bits that are still zero after these n elements have been inserted, this gives, f = (1 - P)k.</p><p>As n items have been entered in the filter by hashing each k times, the probability that a bit is still zero is, which is the expected value of p. Hence the false positive rate can be approximated as, f = (1 - p)k w (1 - p')k w (1 - e"<b>m</b>)" .</p><doubt alpha="33.3" length="9" tooSmall="False" monospace="0.0">, / 1_fcn</doubt><doubt alpha="50.0" length="8" tooSmall="False" monospace="0.0">p=1--wem</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">m</doubt><p>By taking the derivative we find that the number of functions k* that minimizes f is, which leads to the intuitive result that exactly half the bits in the filter will be set to 1 when the optimal number of hash functions is chosen.</p><doubt alpha="45.5" length="11" tooSmall="False" monospace="0.0">k*=ln2•m, n</doubt><p>The fundmental difference between a Bloom fil­ter's space requirements and that of any lossless rep­resentation of a set is that the former does not depend on the size of the (exponential) universe N from which the set is drawn. A lossless representation scheme (for example, a hash map, trie etc.) must de­pend on N since it assigns a distinct representation to each possible set drawn from the universe.</p></section><section number="3" title="Language modelling with Bloom filters"><p>Recent work (Talbot and Osborne, 2007) presented a scheme for associating static frequency information with a set of n-grams in a BF efficiently.<footnote anchor="1"/></p><subsection number="3.1" title="Log-frequency Bloom filter"><p>The efficiency of the scheme for storing n-gram statistics within a BF presented in Talbot and Os-borne (2007) relies on the Zipf-like distribution of n-gram frequencies: most events occur an extremely small number of times, while a small number are very frequent. We assume that raw counts are quan­tised and employ a logarithmic codebook that maps counts, c(x), to quantised counts, qc(x), as follows, <b>Algorithm 1 </b>Training frequency BF<page local="3" global="470"/></p><doubt alpha="36.0" length="25" tooSmall="False" monospace="0.0">qc(x)= 1 + [log6c(x)J.(1)</doubt><footnote label="1">Note that as described the Bloom filter is not an associative data structure and provides only a Boolean function character­ising the set that has been stored in it.</footnote><doubt alpha="60.6" length="33" tooSmall="False" monospace="0.0">Input:Strain,{hi,...hk}and BF = 0</doubt><p>Output: BF <b>for all </b>x G <i>S</i><i>train </i><b>do</b> c(x) — frequency of n-gram x in <i>Strain</i>qc(x) — quantisation of c(x) (Eq. 1) <b>for </b>j = 1 to qc(x) <b>do for </b>i = 1 to k <b>do</b> hi(x) — hash of event {x, j} under <i>hiBF </i>[hi(x)] — 1 <b>end for end for end for return </b>BF</p><p>The precision of this codebook decays exponentially with the raw counts and the scale is determined by the base of the logarithm b; we examine the effect of this parameter on our language models in experi­ments below.</p><p>Given the quantised count qc(x) for an n-gram x, the filter is trained by entering composite events consisting of the n-gram appended by an integer counter j that is incremented from 1 to qc(x) into the filter. To retrieve an n-gram's frequency, the n-gram is first appended with a counter set to 1 and hashed under the k functions; if this tests positive, the counter is incremented and the process repeated. The procedure terminates as soon as any of the k hash functions hits a 0 and the previous value of the counter is reported. The one-sided error of the BF and the training scheme ensure that the actual quan­tised count cannot be larger than this value. As the counts are quantised logarithmically, the counter is usually incremented only a small number of times.</p><p>We can then approximate the original frequency of the n-gram by taking its expected value given the quantised count retrieved,</p><doubt alpha="30.3" length="33" tooSmall="False" monospace="0.0">b7"1+bj-1E[c(x)|gc(x)=j]=--2-.(2)</doubt><p>These training and testing routines are repeated here as Algorithms 1 and 2 respectively.</p><p>As noted in Talbot and Osborne (2007), errors for this log-frequency BF scheme are one-sided: fre­quencies will never be underestimated. The prob­ability of overestimating an item's frequency decays <b>Algorithm 2 </b>Test frequency BF</p><p>Input: x, MAXQCOUNT, {hi, ...hk} and BF Output: Upper bound on c(x) G <i>Strain</i><b>for </b>j = 1 to MAXQCOUNT <b>do for </b>i = 1 to k <b>do</b> exponentially with the size of the overestimation er­ror d (i.e. as <i>fd</i><i> </i>for d &gt; 0) since each erroneous increment corresponds to a single false positive and d such independent events must occur together.</p><doubt alpha="65.4" length="52" tooSmall="False" monospace="0.0">hi(x)— hash of event{x, j} underhiifBF[hi(x)]= 0then</doubt><doubt alpha="59.3" length="54" tooSmall="False" monospace="0.0">returnE[c(x)|qc(x)=j— 1] (Eq. 2)end if end for end for</doubt><p>The efficiency of the log-frequency BF scheme can be understood from an entropy encoding per­spective under the distribution over frequencies of n-gram types: the most common frequency (the sin­gleton count) is assigned the shortest code (length k) while rarer frequencies (those for more common n-grams) are assigned increasingly longer codes (k x qc(x)).</p></subsection><subsection number="3.2" title="Smoothed BF language models"><p>A standard n-gram language model assigns condi­tional probabilities to target words given a certain context. In practice, most standard n-gram language models employ some form of interpolation whereby probabilities conditioned on the most specific con­text consisting usually of the n — 1 preceding to­kens are combined with more robust estimates based on less specific conditioning events. To compute smoothed language model probabilities, we gener­ally require access to the frequencies of n-grams of length 1 to n in our training corpus. Depending on the smoothing scheme, we may also need auxiliary statistics regarding the number of distinct suffixes for each n-gram (e.g., Witten-Bell and Kneser-Ney smoothing) and the number of distinct prefixes or contexts in which they appear (e.g., Kneser-Ney). We can use a single BF to store these statistics but need to distinguish each type of event (e.g., raw counts, suffix counts, etc.). Here we use a distinct set of k hash functions for each such category. Our motivation for storing the corpus statistics directly rather than precomputed probabilities is twofold:<page local="4" global="471"/> (i) the efficiency of the scheme described above for storing frequency information together with items in a BF relies on the frequencies hav­ing a Zipf-like distribution; while this is definitely true for corpus statistics, it may well not hold for probabilities estimated from them; (ii) as will be­come apparent below, by using the corpus statistics directly, we will be able to make additional savings in terms of both space and error rate by using simple inequalities that hold for related information drawn consistently from the same corpus; it is not clear whether such bounds can be established for proba­bilities computed from these statistics.</p><subsubsection number="3.2.1" title="Proxy items"><p>There is a potential risk of redundancy if we rep­resent related statistics using the log-frequency BF scheme presented in Talbot and Osborne (2007). In particular, we do not need to store information ex­plicitly that is necessarily implied by the presence of another item in the training set, if that item can be identified efficiently at query time when needed. We use the term <i>proxy item </i>to refer to items whose presence in the filter implies the existence of another item <i>and </i>that can be efficiently queried given the im­plied item. In using a BF to store corpus statistics for language modelling, for example, we can use the event corresponding to an n-gram and the counter set to 1 as a proxy item for a distinct prefix, suffix or context count of 1 for the same n-gram since (ignor­ing sentence boundaries) it must have been preceded and followed by at least one distinct type, i.e., where s( ) is the number of the distinct types follow­ing this n-gram in the training corpus. We show be­low that such lower bounds allow us to significantly reduce the memory requirements for a BF language model.</p><doubt alpha="45.2" length="31" tooSmall="False" monospace="0.0">qc(wi,Wn)&gt;1G BF ==s(wi,w,n) &gt;1,</doubt></subsubsection><subsubsection number="3.2.2" title="Monotonicity of n -gram event space"><p>The error analysis in Section 2 focused on the false positive rate of a BF; if we deploy a BF within an SMT decoder, however, the actual error rate will also depend on the <i>a priori </i>membership probability of items presented to it. The error rate Err is, <i>Err</i><i> </i><i>=</i><i> </i><i>Pr(x</i><i> </i><i>£</i><i> </i><i>S</i><i>train</i><i>\Decoder)f.</i><i></i></p><p>This implies that, unlike a conventional lossless data structure, the model's accuracy depends on other components in system and how it is queried.</p><p>Assuming that statistics are entered consistently from the same corpus, we can take advantage of the monotonicity of the <i>n</i>-gram event space to place up­per bounds on the frequencies of events to be re­trieved from the filter prior to querying it, thereby reducing the <i>a priori </i>probability of a negative and consequently the error rate.</p><p>Specifically, since the log-frequency BF scheme will never underestimate an item's frequency, we can apply the following inequality recursively and bound the frequency of an n-gram by that of its least frequent subsequence, c(Wi, ...,Wn) &lt; min {c(Wi,      Wn-l), c(W2, ...,Wn)}.</p><p>We use this to reduce the error rate of an interpolated BF language model described below.</p></subsubsection></subsection><subsection number="3.3" title="Witten-Bell smoothed BF LM"><p>As an example application of our framework, we now describe a scheme for creating and querying a log-frequency BF to estimate n-gram language model probabilities using Witten-Bell smoothing (Bell et al., 1990). Other smoothing schemes, no­tably Kneser-Ney, could be described within this framework using additional proxy relations for infix and prefix counts.</p><p>In Witten-Bell smoothing, an n-gram's probabil­ity is discounted by a factor proportional to the num­ber of times that the n — 1-gram preceding the cur­rent word was observed preceding a novel type in the training corpus. It is defined recursively as, <i>P</i><i>ml</i><i>(W</i><i>i </i>\Wi where <i>Ax </i>is defined via, and <i>P</i><i>mi </i>(•) is the maximum likelihood estimator cal­culated from relative frequencies.</p><doubt alpha="58.3" length="12" tooSmall="False" monospace="0.0">Pwb (Wi\Wl~_</doubt><doubt alpha="25.0" length="4" tooSmall="False" monospace="0.0">= A,</doubt><doubt alpha="45.8" length="24" tooSmall="False" monospace="0.0">+(1—Awi-1)Pwb(Wi\Wi-n+2)</doubt><footnote label="1"> —  A x =</footnote><doubt alpha="35.3" length="17" tooSmall="False" monospace="0.0">c(x) s(x) + c(x)'</doubt><p>The statistics required to compute the Witten-Bell estimator for the conditional probability of an n-gram consist of the counts of all n-grams of length 1 to n as well as the counts of the number of distinct types following all n-grams of length 1 to n — 1.<page local="5" global="472"/> In practice we use the c(w1, Wi) = 1 event as a proxy for s(Wi, ... , Wi) = 1 and thereby need not store singleton suffix counts in the filter.</p><p>Distinct suffix counts of 2 and above are stored by subtracting this proxy count and converting to the log quantisation scheme described above, i.e.,</p><doubt alpha="42.3" length="26" tooSmall="False" monospace="0.0">qs(x)= 1 + Llogb(s(x)— 1)J</doubt><p>In testing for a suffix count, we first query the item c(Wl,...,Wn) = 1 as a proxy for s(Wl,...,Wn) = 1 and, if found, query the filter for incrementally larger suffix counts, taking the reconstructed suffix count of an n-gram with a non-zero n-gram count to be the expected value, i.e.,</p><doubt alpha="29.0" length="31" tooSmall="False" monospace="0.0">E[s(x)|qs(x)=jnj &gt;0] = 1 + ^-^-</doubt><p>Having created a BF containing these events, the algorithm we use to compute the interpolated WB estimate makes use of the inequalities described above to reduce the <i>a priori </i>probability of querying for a negative. In particular, we bound the count of each numerator in the maximum likelihood term by the count of the corresponding denominator and the count of distinct suffixes of an n-gram by its respec­tive token frequency.</p><p>Unlike more traditional LM formulations that back-off from the highest-order to lower-order mod­els, our algorithm works up from the lowest-order model. Since the conditioning context increases in specificity at each level, each statistic is bound from above by its corresponding value at the previous less specific level. The bounds are applied by passing them as the parameter MAXQCOUNT to the fre­quency test routine shown as Algorithm 2. We ana­lyze the effect of applying such bounds on the per­formance of the model within an SMT decoder in the experiments below. Working upwards from the lower-order models also allows us to truncate the computation before the highest level if the denomi­nator in the maximum likelihood term is found with a zero count at any stage (no higher-order terms can be non-zero given this).</p></subsection></section><section number="4" title="Experiments"><p>We conducted a range of experiments to explore the error-space trade-off of using a BF-based model as a replacement for a conventional n-gram model within an SMT system and to assess the benefits of specific features of our framework for deriving lan­guage model probabilities from a BF.</p><subsection number="4.1" title="Experimental set-up"><p>All of our experiments use publically available re­sources. Our main experiments use the French-English section of the <i>Europarl </i>(EP) corpus for par­allel data and language modelling (Koehn, 2003). Decoding is carried-out using the Moses decoder (Koehn and Hoang, 2007). We hold out 1,000 test sentences and 500 development sentences from the parallel text for evaluation purposes. The parame­ters for the feature functions used in this log-linear decoder are optimised using minimum error rate (MER) training on our development set unless other­wise stated. All evaluation is in terms of the BLEU score on our test set (Papineni et al., 2002).</p><p>Our baseline language models were created us­ing the SRILM toolkit (Stolcke, 2002). We built 3, 4 and 5-gram models from the Europarl corpus us­ing interpolated Witten-Bell smoothing (WB); no n-grams are dropped from these models or any of the BF-LMs. The number of distinct n-gram types in these baseline models as well as their sizes on disk and as compressed by <i>gzip </i>are given in Table 1; the <i>gzip </i>figures are given as an approximate (and opti­mistic) lower bound on lossless representations of these models.<footnote anchor="2"/></p><p>The BF-LM models used in these experiments were all created from the same corpora following the scheme outlined above for storing n-gram statistics. Proxy relations were used to reduce the number of items that must be stored in the BF; in addition, un­less specified otherwise, we take advantage of the bounds described above that hold between related statistics to avoid presenting known negatives to the filter. The base of the logarithm used in quantization is specified on all figures.</p><p>The SRILM and BF-based models are both queried via the same interface in the Moses decoder.</p><footnote label="2">Note, in particular, that gzip compressed files do not sup­port direct random access as required by in language modelling.</footnote><page local="6" global="473"/><p>We assign a small cache to the BF-LM models (be­tween 1 and 2MBs depending on the order of the model) to store recently retrieved statistics and de­rived probabilities. Translation takes between 2 to 5 times longer using the BF-LMs as compared to the corresponding SRILM models.</p></subsection><subsection number="4.2" title="Machine translation experiments"><p>Our first set of experiments examines the relation­ship between memory allocated to the BF-LM and translation performance for a 3-gram and a 5-gram WB smoothed BF-LM. In these experiments we use the log-linear weights of the baseline model to avoid variation in translation performance due to differ­ences in the solutions found by MER training: this allows us to focus solely on the quality of each BF-LM's approximation of the baseline. These exper­iments consider various settings of the base for the logarithm used during quantisation (b in Eq. (1)).</p><p>We also analyse these results in terms of the re­lationships between BLEU score and the underlying error rate of the BF-LM and the number of bits as­signed per n-gram in the baseline model.</p><p>MER optimised BLEU scores on the test set are then given for a range of BF-LMs.</p></subsection><subsection number="4.3" title="Mean squared error experiments"><p>Our second set of experiments focuses on the accu­racy with which the BF-LM can reproduce the base­line model's distribution. Unfortunately, perplex­ity or related information-theoretic quantities are not applicable in this case since the BF-LM is not guar­anteed to produce a properly normalised distribu­tion. Instead we evaluate the mean squared error (MSE) between the log-probabilites assigned by the baseline model and by BF-LMs to n-grams in the English portion of our development set; we also con­sider the relation between MSE and the BLEU score from the experiments above.</p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">32</doubt><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">30</doubt><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">28</doubt><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">26</doubt><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">24</doubt><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">22</doubt><p>WB-smoothed BF-LM 3-gram model</p><doubt alpha="32.0" length="25" tooSmall="False" monospace="0.0">^ ' '      BF-LM base 1.1</doubt><doubt alpha="61.7" length="60" tooSmall="False" monospace="0.0">BF-LM base 1.5 BF-LM base 3 SRILM Witten-Bell 3-gram (174MB)</doubt><doubt alpha="15.2" length="66" tooSmall="False" monospace="0.0">0.0025 0.005 0.0075   0.01   0.0125 0.015 0.0175 0.02 Memory in GB</doubt><figure caption="Figure 1: WB-smoothed 3-gram model (Europarl)."></figure></subsection><subsection number="4.4" title="Analysis of BF-LM framework"><p>Our third set of experiments evaluates the impact of the use of upper bounds between related statistics on translation performance. Here the standard model that makes use of these bounds to reduce the <i>a pri­ori </i>negative probability is compared to a model that queries the filter in a memoryless fashion.<footnote anchor="3"/></p><p>We then present details of the memory savings ob­tained by the use of proxy relations for the models used here.</p></subsection></section><section number="5" title="Results"><subsection number="5.1" title="Machine translation experiments"><p>Figures 1 and 2 show the relationship between trans­lation performance as measured by BLEU and the memory assigned to the BF respectively for WB-smoothed 3-gram and 5-gram BF-LMs. There is a clear degradation in translation performance as the memory assigned to the filter is reduced. Models using a higher quantisation base approach their opti­mal performance faster; this is because these more coarse-grained quantisation schemes store fewer items in the filter and therefore have lower underly­ing false positive rates for a given amount of mem­ory.</p><p>Figure 3 presents these results in terms of the re­lationship between translation performance and the false positive rate of the underlying BF. We can see that for a given false positive rate, the more coarse­grained quantisation schemes (e.g., base 3) perform<page local="7" global="474"/></p><footnote label="3">In both cases we apply 'sanity check' bounds to ensure that none of the ratios in the WB formula (Eq. 3) are greater than 1.</footnote><doubt alpha="0.0" length="2" tooSmall="True" monospace="0.0">0)</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">oÜ</doubt><doubt alpha="33.3" length="3" tooSmall="True" monospace="0.0">C/3</doubt><doubt alpha="60.0" length="10" tooSmall="True" monospace="0.0">D LU —I CO</doubt><table caption="Table 1: WB-smoothed SRILM baseline models." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>n</p></td><td class="cell"><p>Types</p></td><td class="cell"><p>Mem.</p></td><td class="cell"><p>Gzip'd</p></td><td class="cell"><p>BLEU</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3</p></td><td class="cell"><p>5.9M</p></td><td class="cell"><p>174Mb</p></td><td class="cell"><p>51Mb</p></td><td class="cell"><p>28.54</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>4</p></td><td class="cell"><p>14.1M</p></td><td class="cell"><p>477Mb</p></td><td class="cell"><p>129Mb</p></td><td class="cell"><p>28.99</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>5</p></td><td class="cell"><p>24.2M</p></td><td class="cell"><p>924Mb</p></td><td class="cell"><p>238Mb</p></td><td class="cell"><p>29.07</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">3</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">5</doubt><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">24</doubt><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">22</doubt><p>WB-smoothed BF-LM 3-gram model</p><doubt alpha="0.0" length="2" tooSmall="True" monospace="0.0">0)</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">oÜ</doubt><doubt alpha="33.3" length="3" tooSmall="True" monospace="0.0">C/3</doubt><doubt alpha="60.0" length="10" tooSmall="True" monospace="0.0">D LU —I CO</doubt><doubt alpha="9.1" length="11" tooSmall="False" monospace="0.0">32 3C 28 26</doubt><doubt alpha="63.3" length="49" tooSmall="False" monospace="0.0">WB-smoothed BF-LM 5-gram model ' ' BF-LM base 1.1</doubt><doubt alpha="57.1" length="14" tooSmall="False" monospace="0.0">BF-LM base 1.5</doubt><doubt alpha="64.4" length="45" tooSmall="False" monospace="0.0">BF-LM base 3 SRILM Witten-Bell 5-gram (924MB)</doubt><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">C.C1</doubt><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">C.C2</doubt><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">C.C6</doubt><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">C.C7</doubt><figure caption="Figure 2: WB-smoothed 5-gram model (Europarl)."></figure><doubt alpha="0.0" length="20" tooSmall="False" monospace="0.0">30 29 28 27 26 25 24</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">23 22</doubt><doubt alpha="0.0" length="10" tooSmall="False" monospace="0.0">0.01 0.1 1</doubt><p>False positive rate (probability)</p><figure caption="Figure 3: False positive rate vs. BLEU ."></figure><p>worse than the more fine-grained schemes.<footnote anchor="4"/></p><p>Figure 4 presents the relationship in terms of the number of bits per n-gram in the baseline model. This suggests that between 10 and 15 bits is suf­ficient for the BF-LM to approximate the baseline model. This is a reduction of a factor of between 16 and 24 on the plain model and of between 4 and 7 on <i>gzip </i>compressed model.</p><p>The results of a selection of BF-LM models with decoder weights optimised using MER training are given in Table 2; these show that the models perform consistently close to the baseline models that they approximate.</p></subsection><subsection number="5.2" title="Mean squared error experiments"><figure caption="Figure 5 shows the relationship between memory as­signed to the BF-LMs and the mean squared error"></figure><footnote label="4">Note that in this case the base 3 scheme will use approxi­mately two-thirds the amount of memory required by the base 1.5 scheme.</footnote><doubt alpha="10.0" length="20" tooSmall="False" monospace="0.0">32 3C 28 26 24 22 2C</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">7</doubt><doubt alpha="0.0" length="25" tooSmall="False" monospace="0.0">9    11   13   15   17 19</doubt><p>Bits per n-gram</p><figure caption="Figure 4: Bits pern-gram vs. BLEU."></figure><p>(MSE) of log-probabilities that these models assign to the development set compared to those assigned by the baseline model. This shows clearly that the more fine-grained quantisation scheme (e.g. base 1.1) can reach a lower MSE but also that the more coarse-grained schemes (e.g., base 3) approach their minimum error faster.</p><p>Figure 6 shows the relationship between MSE between the BF-LM and the baseline model and BLEU. The MSE appears to be a good predictor of BLEU score across all quantisation schemes. This suggests that it may be a useful tool for optimising BF-LM parameters without the need to run the de­coder assuming a target (lossless) LM can be built and queried for a small test set on disk. An MSE of below 0.05 appears necessary to achieve translation performance matching the baseline model here.</p></subsection><subsection number="5.3" title="Analysis of BF-LM framework"><p>We refer to (Talbot and Osborne, 2007) for empiri­cal results establishing the performance of the log-frequency BF-LM: overestimation errors occur with</p><p>Memory in GB<page local="8" global="475"/></p><doubt alpha="0.0" length="19" tooSmall="False" monospace="0.0">0.03      0.04 0.05</doubt><table caption="Table 2: MERT optimised WB-smoothed BF-LMS." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>n</p></td><td class="cell"><p>Memory</p></td><td class="cell"><p>Bits / n-gram</p></td><td class="cell"><p>base</p></td><td class="cell"><p>BLEU</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3</p></td><td class="cell"><p>10MB</p></td><td class="cell"><p>14 bits</p></td><td class="cell"><p>1.5</p></td><td class="cell"><p>28.33</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3</p></td><td class="cell"><p>10MB</p></td><td class="cell"><p>14 bits</p></td><td class="cell"><p>2.0</p></td><td class="cell"><p>28.47</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>4</p></td><td class="cell"><p>20MB</p></td><td class="cell"><p>12 bits</p></td><td class="cell"><p>1.5</p></td><td class="cell"><p>28.63</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>4</p></td><td class="cell"><p>20MB</p></td><td class="cell"><p>12 bits</p></td><td class="cell"><p>2.0</p></td><td class="cell"><p>28.63</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>5</p></td><td class="cell"><p>40MB</p></td><td class="cell"><p>14 bits</p></td><td class="cell"><p>1.5</p></td><td class="cell"><p>28.53</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>5</p></td><td class="cell"><p>40MB</p></td><td class="cell"><p>14 bits</p></td><td class="cell"><p>2.0</p></td><td class="cell"><p>28.72</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>5</p></td><td class="cell"><p>50MB</p></td><td class="cell"><p>17 bits</p></td><td class="cell"><p>1.5</p></td><td class="cell"><p>29.31</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>5</p></td><td class="cell"><p>50MB</p></td><td class="cell"><p>17 bits</p></td><td class="cell"><p>2.0</p></td><td class="cell"><p>28.67</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">24</doubt><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">22</doubt><p>WB-smoothed BF-LM 3-gram model</p><doubt alpha="0.0" length="2" tooSmall="True" monospace="0.0">0)</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">oÜ</doubt><doubt alpha="33.3" length="3" tooSmall="True" monospace="0.0">C/3</doubt><doubt alpha="60.0" length="10" tooSmall="True" monospace="0.0">D LU —I CO</doubt><doubt alpha="9.1" length="11" tooSmall="False" monospace="0.0">32 3C 28 26</doubt><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">C.C1</doubt><p>MSE between WB S-gram SRILM and BF-LMs</p><doubt alpha="0.0" length="8" tooSmall="False" monospace="0.0">0.5 0.25</doubt><doubt alpha="0.0" length="14" tooSmall="False" monospace="0.0">0.1 0.05 0.025</doubt><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">0.01</doubt><doubt alpha="54.2" length="24" tooSmall="False" monospace="0.0">Base S Base 1.5 Base 1.1</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">0.001</doubt><doubt alpha="11.1" length="9" tooSmall="False" monospace="0.0">0.02 0.0S</doubt><doubt alpha="0.0" length="20" tooSmall="False" monospace="0.0">0.0025    0.005 0.01</doubt><p>Memory in GB</p><figure caption="Figure 5: MSE between SRILM and BF-LMs"></figure><p>WB-smoothed BF-LM s-gram model</p><doubt alpha="11.5" length="26" tooSmall="False" monospace="0.0">sc 29 28 27 26 25 24 2s 22</doubt><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">c.c1</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">1</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">0.1</doubt><p>Mean squared error</p><figure caption="Figure 6: MSE vs. BLEU for WB 3-gram BF-LMs"></figure><p>a probability that decays exponentially in the size of the overestimation error.</p><p>Figure 7 shows the effect of applying upper bounds to reduce the <i>a priori </i>probability of pre­senting a negative event to the filter in our in­terpolation algorithm for computing WB-smoothed probabilities. The application of upper bounds im­proves translation performance particularly when the amount of memory assigned to the filter is lim­ited. Since both filters have the same underlying false positive rate (they are identical), we can con­clude that this improvement in performance is due to a reduction in the number of negatives that are presented to the filter and hence errors.</p><p>Table 3 shows the amount of memory saved by the use of proxy items to avoid storing singleton suffix counts for the Witten-Bell smoothing scheme. The savings are given as ratios over the amount of memory needed to store the statistics without proxy items. These models have the same underlying false positive rate (0.05) and quantisation base (2). Sim­ilar savings may be anticipated when applying this framework to infix and prefix counts for Kneser-Ney smoothing.</p><doubt alpha="50.0" length="6" tooSmall="False" monospace="0.0">C.CC25</doubt><figure caption="Figure 7: Effect of upper bounds on BLEU"></figure></subsection></section><section number="6" title="Related Work"><p>Previous work aimed at reducing the size of n-gram language models has focused primarily on quanti­sation schemes (Whitaker and Raj, 2001) and prun­ing (Stolcke, 1998). The impact of the former seems limited given that storage for the n-gram types them­selves will generally be far greater than that needed for the actual probabilities of the model. Pruning on the other hand could be used in conjunction with the framework proposed here. This holds also for compression schemes based on clustering such as (Goodman and Gao, 2000). Our approach, however, avoids the significant computational costs involved in the creation of such models.</p><p>Other schemes for dealing with large language models include per-sentence filtering of the model or its distribution over a cluster. The former requires time-consuming adaptation of the model for each sentence in the test set while the latter incurs sig­nificant overheads for remote calls during decoding. Our framework could, however, be used to comple­ment either of these approaches.</p><doubt alpha="0.0" length="12" tooSmall="False" monospace="0.0">0.005 0.0075</doubt><table caption="Table 3: Space savings via proxy items ." class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>n-gram order</p></td><td class="cell"><p>Proxy space saving</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3</p></td><td class="cell"><p>0.885</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>4</p></td><td class="cell"><p>0.783</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>5</p></td><td class="cell"><p>0.708</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="9" global="476"/></section><section number="7" title="Conclusions and Future Work"><p>We have proposed a framework for computing smoothed language model probabilities efficiently from a randomised representation of corpus statis­tics provided by a Bloom filter. We have demon­strated that models derived within this framework can be used as direct replacements for equivalent conventional language models with significant re­ductions in memory requirements. Our empirical analysis has also demonstrated that by taking advan­tage of the one-sided error guarantees of the BF and simple inequalities that hold between related n-gram statistics we are able to further reduce the BF stor­age requirements and the effective error rate of the derived probabilities.</p><p>We are currently implementing Kneser-Ney smoothing within the proposed framework. We hope the present work will, together with Talbot and Os-borne (2007), establish the Bloom filter as a practi­cal alternative to conventional associative data struc­tures used in computational linguistics. The frame­work presented here shows that with some consider­ation for its workings, the randomised nature of the Bloom filter need not be a significant impediment to is use in applications.</p><p><b>Acknowledgements References</b></p><p>T.C. Bell, J.G. Cleary, and I.H. Witten. 1990. <i>Text Compres­sion. </i>Prentice Hall, Englewood Cliffs, NJ.</p><p>B. Bloom. 1970. Space/time tradeoffs in hash coding with allowable errors. <i>CACM, </i>13:422-426.</p><p>A. Broder and M. Mitzenmacher. 2005. Network applications of Bloom filters: A survey. <i>Internet Mathematics, </i>1(4):485-509.</p><p>David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In <i>Proceedings of the 43rd Annual Meeting of the Association for Computational Lin­guistics (ACL'05), </i>pages 263-270, Ann Arbor, Michigan, June. Association for Computational Linguistics.</p><p>J. Goodman and J. Gao. 2000. Language model size reduction by pruning and clustering. In <i>ICSLP'00, </i>Beijing, China.</p><p>Philipp Koehn and Hieu Hoang. 2007. Factored translation models. In <i>Proc. of the 2(007 Conference on Empirical Meth­ods in Natural Language Processing (EMNLP/Co-NLL).</i></p><p>P. Koehn. 2003. Europarl: A multilingual corpus for evaluation of machine translation, draft. Available at:http://people.csail.mit.edu/koehn/publications/europarl.ps.</p><doubt alpha="56.9" length="58" tooSmall="False" monospace="0.0">K. Papineni, S. Roukos, T. Ward, and W.J. Zhu. 2002. BLEU:</doubt><p>a method for automatic evaluation of machine translation. In <i>ACL-20002: 40th Annual meeting of the Association for Computational Linguistics.</i></p><p>Andreas Stolcke. 1998. Entropy-based pruning of back-off lan­guage models. In <i>Proc. DARPA Broadcast News Transcrip­tion and Understanding Workshop, </i>pages 270-274.</p><p>A. Stolcke. 2002. SRILM - an extensible language modeling toolkit. In <i>Proc. Intl. Conf. on Spoken Language Processing.</i></p><p>D. Talbot and M. Osborne. 2007. Randomised language mod­elling for statistical machine translation. In <i>45th Annual Meeting ofthe Association ofComputational Linguists (To appear).</i></p><p>E. Whitaker and B. Raj. 2001. Quantization-based language model compression (tr-2001-41). Technical report, Mit­subishi Electronic Research Laboratories.</p></section></body></article>