In probabilities this article, to the term over language model is used to refer to Language any models description that important assigns (Manning in strings a certain language processing, alphabet. and in particular, in have applications natural Sch 1999). speech recognition systems Language models and (FA) often ütze or consist of a symbolic description (CFG), of a language, probability such as a finite automaton for a context-free grammar we to, probabilistic example, the transitions of the FA or extended the rules of by the CFG, a assignment obtain (PFA) or probabilistic by which (PCFG), respectively. a For finite automaton of applications, one may context-free grammar or certain in first determine to the symbolic probability part the automaton grammar or and The a second phase try problem, transitions FAs rules. current article is involved find reliable estimates for the process of or CFGs to become PFAs or PCFGs. with that We the second extending . refer to this Training as training If is often done in on the basis of a is corpus of actual language of use in a certain domain. FA each sentence or this corpus for annotated CFG by a list transitions of an recognizing may the sentence of simply in a parse tree a generating relative frequency estimation . This means the sentence, that we then training probabilities consist transitions of or by of counting frequencies in estimate rules their or the corpus, respectively. transitions By to the frequencies of relative to the frequencies of the start states of the left-hand side is maximized. nonterminals rules, this estimation, the likelihood the corpus The technique of we introduce in this of article input is different model. in that training Our is the finite corpus, goal is done on basis not a estimations probabilities of but an or language rules of the input FA or CFG to find for the transitions such that ∗ Faculty E-mail: markjan@let.rug.nl. of Arts, Humanities Computing, P.O. Box 716, NL-9700 AS Groningen, The Netherlands. Submission received: 19th September 20th January 2004; Revised submission received: 5th August 2004; Accepted for publication: 2004 © 2005 Association for Computational Linguistics Computational Linguistics Volume 31, Number 2 the or resulting more PFA or specifically, PCFG approximates the input language Kullback-Leibler (KL) model distance as (or well as possible, input such may model that the the trained model is minimized. The relative input FA entropy) between the and to input model. or CFG to be trained of This be structurally unrelated the technique work on has several approximation of CFGs One language applications. is means an extension of FAs with probabilities existing motivation for of work was by which is that application important of FAs is (Nederhof 2000). The this CFGs, when generally input less is costly than application is often in, an benefit the work example, speech The very large, for recognition systems. practical relevance of as the this was case limited, probabilities of however, by in the fact that in practice Boolean one distinction is more interested in the sentences than a purely between grammatical and Several ungrammatical work approaches sentences. approximation were of discussed PCFGs means by Mohri of PFAs. and Nederhof (2001) approach is to extend to first to directly map this with probabilities by to with A rules is attached approach, transitions attached PFA probabilities. probability easiest distribution the resulting may be a very inaccurate Although this approximation computationally of the lar, more may the be assignments of approximating probabilities to described by the input models. the transitions of PCFG. In the same FA particu- there that lead to accurate A approach was is language input second PCFG itself obtained to train the approximating on FA by means However, training a corpus, then we of a corpus. If the material. Furthermore, may by not always be the case, and no already possess training may this approximating FA training may material be available. input PCFG, as a determinized for the sparse-data 1 Hence, problem if may be more be material severe was for much larger than the it was grammar. the automaton than CFG, it may the even sufficient train be sufficient to accurately the FA. available to the A approach is not train (pseudo)random third to construct of a training corpus from the PCFG by a such that sentences that are more means of PCFG generator sentences, (1994), for generated with greater of likelihood. This has been proposed likely according Jurafsky to the are by et Zue al. (1991). the It special is not case clear, however, bigrams, whether extending a nonprobabilistic technique by of et al. models this The idea is feasible for training finite-state would that are larger in order than bigrams. obtain reason probability is that very large corpora estimates PFA. Note have to be generated PFAs. square of number of to the parameters of accurate for the that the size of the lexicon; such a bound a does bigram not model is bounded by the exist for general The approach current article discusses if infinite a fourth approach. were In the limit, it on is which equivalent PFA to the third we above, as an idea have found a way infinite to corpus constructed avoid considering of sentences individually. the PCFG is The is trained, but key that allows us to handle an set a grammar represents intersection strings generated construct that the of by the that we input PCFG new FA. Within grammar, we the languages described by the of and the of FA, this new can compute of PCFGs. the expected frequencies transitions the using to a fairly standard analysis transitions of FA then allow us determine the assignment of probabilities These expected frequencies PFA. the that minimizes KL between the PCFG to the distance and the resulting Nederhof Training Models on Models The mean only requirement input is that the FA to be trained be most unambiguous, one by which The be recognized by at computation of we that each of string can n-grams Stolcke the case been formulated by and Segal FA. special idea has already (P)FA by Rimon one and Herz (1991). An n-gram (1994), realizing an previously envisioned here each possible history of model is seen as a that contains exactly state for the n − our It is that such an FA is unambiguous (even 1 previously read symbols. clear (1994), technique FA way properly deterministic) and that therefore that the two subsumes the technique techniques are formulated is by Stolcke and Segal although the n-gram model input rather different. note underlying string over Also that which the hold for an (unambiguous) accepts FAs. any the alphabet, does not general PFAs. Another application Mohri of our (1997), work PFAs involves determinization and minimization of As shown by to minimize cannot always be determinized, (P)FAs. and when arbitrary nondeterministic This no practical algorithms problem are known or small PFAs We however, can be a offer a minimal deterministic deterministic FA are required. equivalent FA to an input FA. The can, always compute way PFA new it results in a this determinized to a such that approximates this article probability distribution to extend the KL distance. described by the input PFA as well as possible, in terms of the Although is the proposed technique it is no has means some limitations, in particular, to trained unambiguous, by restricted to language models that the model be or other probabilistic based on finite automata context-free in grammars, manner. as several grammatical formalisms The can be 2. Section of treated a similar in Section structure 3 this discusses article is how as follows. We provide some This in the expected frequency of preliminary a rule in a PCFG definitions is an step the algorithms to be discussed below. Section can be computed. definesawaytocombineaPFAandaPCFGintoanewPCFGthatextendsawell-known auxiliary 4 representation we merge input of the model intersection of model a regular and a context-free into single language. Thereby is the and the respectively, foundation of a of to be trained a FA on presented in structure. structure for number algorithms, section 5, which This allow, the of training CFG an unambiguous on of PFA the (section basis of 5.2), a PCFG (section training an the basis a and training of 5.1), unambiguous FA on of PFA (section 5.3). an unambiguous the basis a

Many (1972) of the Booth definitions Thompson on probabilistic (1973), context-free grammars on are probabilistic based on Santos and and on Paz (1971) Starke and (1972). the definitions finite automata are based A of grammar is and context-free G a 4-tuple , ( respectively, Σ , N, S, R), where S N is Σ and N are two finite R disjoint is sets terminals of , and nonterminals of form , where ∈ a the A α A ∈ N the start symbol and α ∈ ( Σ ∪ N) , and finite set rules each → ∗ is G is a 5-tuple R ( Σ ,N,S,R,p G ), where Σ ,N,Sand .A probabilistic context-free grammar from in probabilities. R are as In above, , what and follows, p G a function rules A,B over over to symbol a ranges N, the set Σ X , symbols over w, v range over the set Σ ∗ symbols α , , range γ over the the set ( set Σ ∪ N) ∗ , symbol ranges symbol ρ over the set the set R, Σ ∪ N, symbols d,e β range over R . With of ranges range set ∗ (A α ∈ R the slight abuse → atomic symbol when it occurs within notation, we and symbols ) treat a rule ρ as an a string d ρ e ∈ R ∗ = operator The denotes the empty string. String concatenation is . symbol or represented by · by empty space. Computational Linguistics Volume 31, Number 2 , For is of ( a fixed Σ ∪ N) (P)CFG G , we define form and a rule ρ ∈ R the relation by α ρ β , if ⇒ on triples α ∈ and only some and δ ∈ Σ ∪ N) if consisting α is of of the form two strings β , for ( , and (A ). wA δ and β the w γδ w ∈ Σ ρ = → derivation (in G ) is string ρ m ,m ≥ 0, γ A leftmost , N) d 1 ··· such that α 0 ρ 1 α 1 ρ ρ 2 ρ a m = some , ∈ ( Σ ∪ ;d is always leftmost derivation. In ··· α m , for α 0 α m = of ... we a to the remainder this ified article, otherwise. If let the term derivation some refer , , leftmost ( derivation, α for α 0 α m ∈ Σ ∪ N) , unless then we spec- 0 ρ α 1 ρ m from derives α from ··· m ρ α 0 , we S write ... w, for α 0 α m ; say 1 m and d itself. derives , is any α 0 ∈ ( that d = ··· ρ m Σ ∪ N) A derivation We d such is that d if some w ∈ Σ called ,S a complete derivation. one R . say that G unambiguous for each w ∈ Σ d w for at most Let d ∈ 0, G be a fixed PCFG ( Σ ,N,S,R,p m G ). ) For if α , β ,and ∈ ( Σ ∪ N) and d = m we ( ) ( ( ) 0 otherwise. ρ 1 ··· ρ m ∈ R p ∏ 1 p ρ d β p d β The , ≥ define G α d β = i G i α G α = probability PCFG p (w)ofastring = w ∈ Σ is (A if defined ) with (A to be ∑ G to ) d p proper p ρ G α 1 G (S d G probabilities is of said be , = ρ α for w). ∑ sum all to one. A ∈ N, PCFG that is, if issaidto the all rules (w) ρ = → if 1. Consistency α left-hand be implies side A that the PCFG defines probability G consistent ∑ p G = on w is of (Booth There is a distribution the set terminal strings. Thompson 1973). a practical sufficient condition for consistency ,w PCFG that , is said decidable A to and ( be reduced 1 N) if for each (S nonterminal d ∈ such 1 wA 1 β ) A, (w there are d1 ,d β that 1 A β d2 1 ) 2 ∈ w 2 and ∈ Σ ∪ p p ww 2 > 0. R , words, if Σ PCFG nonzero is G a then for each nonterminal A, · G there from is at S least one In 2 with reduced, one is with probability derivation d 1 d left-hand side that A. derives PCFG a string some A G that is ww 1 2 and that includes rule probability not distribution, reduced can provided be turned into that (w) reduced p 0. This and that describes the reduction consists in same removing from the nonterminal that ∑ w G which > the conditions do not hold, together with grammar any A for nonterminal; above Ullman (1972) of any CFGs, rule which that contains such similar. a see Aho and for reduction is very A finite of automaton M is a 5-tuple respectively, ( Σ , Q, q 0 , ,q q f , T), Q where Σ and Q are two finite sets respectively, terminals T and is states, of q 0 f ∈ of are the form initial and states, finite transitions, each the a s, where final and a set r → r ∈ ,q Q ,T,p −{ q } ), ,s where ∈ Q,and ,Q,q a ∈ ,q Σ .A 2 f ,and probabilistic T finite automaton is M is in f M Σ 0 f areasabove,and p M a function from a 6-tuple ( Σ ,Q, q 0 T probabilities. transitions In to what follows, symbols over q,r,s T range . over the set Q, symbol τ ranges over the set T, and For symbol c ranges a (P)FA on , we the set fixed M define a configuration of two to be an element of Q × Σ ,and we define (r,w) the relation (s,w ) if only triples if consisting is of form configurations , for and a transition , (r τ ∈ s). T by τ (r (in and M some ) is the aw (r a c = τ 1 ··· τ m ,m Q ≥ 0, some a ∈ Σ A string such ··· for 0 0 , (r m ,w m ) ; that is (r and a w 0 ,w τ = → ), 0 ) τ computation 1 m ,w ,w ), (r 1 ,w 1 ) τ 2 τ m m 1 m )forsome (r ,w ∈ × Σ c = If (r 0 ,w ) (r ,w ... m 0 0 ), ,(r m ,w m ) Q always a compu- τ tation. T , we 0 τ write ··· ( 0 ,w m ) c (r ,w m ). We ∈ × if Σ (q and 0 ,w) c = ... m m (q τ then that c τ 0 recognizes , 1 ··· ∈ r say c w f ). Nederhof Training Models on Models Let M be a fixed FA ( Σ [(q , Q, ,w) q 0 , q (q f , , T). )] . The We language is L( M ) accepted ,  most one f } T . say We M unambiguous is if by M is defined (r,w) (q be { w ∈ (q Σ |∃ c to ,w) , ) for c Σ one c ∈ say of M T deterministic (s,w Q if for each w 0 c ∈ Q , f (r,w) ). is at ∈ there at Turning most given combination FA into one τ is ∈ and ) for each × Σ (s,w ∈ × Σ such that τ is a . that FAs deterministic and accepts language (deterministic) into determinization smallest All (deterministic) can FA be determinized. Turning the same called FA a given is minimization the . There that for accepts the minimization same of language called FAs. are effective algorithms deterministic Let M be T a , fixed we PFA ( Σ , Q, ((r,w) q 0 , q (s,v)) f , T, p M ). For m (r,w),(s,v) ( ) if (r,w) ∈ Q × (s,v), Σ and c 1 τ m ∈ define c p ∏ c = ((r,w) τ ··· (s,v)) 0 M = otherwise. The probability (w) i = 1 of p M τ string i ,w) p M a w ∈ Σ is and c p M = PFA ((q is 0 c to c p M (q f , )). defined be ∑ M said to be proper if ∑ ,a,s: (r a s) T p M ( τ ) = 1 for all r ∈ Q −{ q f } . τ τ =

Let occur G in be a PCFG ( right-hand Σ ,N,S,R,p side of G ). We assume from without R. For loss of generality , we that S does not the any rule each rule ρ define E( ) ∑ (S d ρ d ′ ρ = p G ⇒ w) d,d ′ ,w (1) If G is Each proper and consistent, (1) is the expected written frequency of ,with ρ in a complete derivation. complete derivation d ρ d can be as d ρ d d d = d d , where S d d ′′ ⇒ wA β ,A ρ ⇒ , ⇒ w , β d ′′′ α α ⇒ w (2) for some A, α , β ,w ,w ,and w . Therefore

E( ρ ) = outer(A) · p G ( ρ ) · inner( α ) (3) where we define outer(A) = d,w ′ , β ∑ ,d ′′′ ,w ′′′ p G (S ⇒ d wA β ) · p G ( β d ⇒ ′′′ w ) (4) inner( α ) = d ∑ ′′ ,w ′′ p G ( α ⇒ d ′′ w ) (5) for each A ∈ N and α ∈ ( Σ ∪ N) . From the definition of inner, we can easily derive the following equations: inner(a) = 1 inner(A) =

∑ ρ (A , α : ρ = → α ) inner(X β ) = inner(X) · inner( β ) p G ( ρ ) · inner( α ) (6) (7) (8) Computational Linguistics Volume 31, Number 2 This can be taken as a recursive definition of inner, assuming = of outer: in (8). Similarly, we can derive a recursive definition outer(S) = 1 outer(A) = ρ = → α for A = In S. is, general, for there may nonterminals be cyclic A, dependencies inner(A) in the outer(A) equations may for inner and of and be defined in outer; that certain There may approximate to closed-form expression for inner(A). means However, terms one themselves. may even be no iteration. the solutions arbitrary precision by of fixed-point ∑ ρ ,B, (B α , β : A β ) outer(B) · p G ( ) · inner( ) · inner( ) (9) (10)

We intersection recall a construction CFG of from Bar-Hillel, Perles, and Shamir (1964) a language ( and ,Q,q a regular ,q , T); language. The we input that computes consists of the ( ,N,S,R)andanFA context-free = 0 note that assume, without a G = of generality, Σ M Σ The output of M the is same CFG set of f that share terminals ( ,N ,S Σ . loss G and ( S the ,S,q construction R G ∩ = Σ ∩ ∩ ,R ∩ ), where N ∩ = ∪ N) Q, (q ), and consists of the set of rules that is obtained Q × Σ × ∩ = 0 f ∩ as follows: For bein , each ,r rule = Q, (A → X 1 ··· X m rule ((r ) ∈ ,A,r R,m m ) ≥ 0, 0 → ((r (r and 0 ,A,r ,X each 1 ,r ) 1 ) sequence m the ··· (r ) for m 1 ,X of states r ∈ let ∩ = R ;for m 0,R 0 − m ,r m )) ∩ = ∩ contains a rule ∩ = 0 0 → . each state r 0 For each transition = (r a → s) ∈ T, let the rule ∩ = ((r,a,s) → in R . a) be ∩ Note that for each X rule X ( from r 0 ,A,r R m from ) → (r which 0 ,X 1 ,r it 1 ) ··· (r m − 1 ,X m ,r m ) from R ∩ , there is a unique rule A → Similarly, ( 1 r,a,s) ··· m identifies has been constructed s. This by means the above. we each rule → uniquely we a transition a in a r → a ) d ∩ G ∩ , can extract a sequence h 1 (d ∩ ) of rules from that if take derivation sequence (d of from M ,where h and h are string homomorphisms G and a we h pointwise 2 ∩ transitions that define as h 1 ( ∩ ) = h 2 ( ∩ ) = if ∩ = ((r 0 ,A,r m ) → (r 0 ,X 1 ,r 1 ) ··· (r m − 1 ,X m ,r m )) and = (A → X 1 ··· X m ) if ∩ = ((r,a,s) → a) if if ∩ ((r,a,s) a) and (r a = → = → s) ∩ = ((r 0 ,A,r m ) → (r 0 ,X 1 ,r 1 ) ··· (r m − 1 ,X m ,r m )) (11) (12) (13) (14) Nederhof Training Models on Models We define h(d ) = (h 1 (d ),h w, 2 (d )). It can S be easily shown that ). if h(d ) = S w, we wand(q ,w) (q , Conversely, if (d,c) and d ∩ then for the have d w,d,and wehave ,w) (q , ), 0 c same c S and (q 0 c then there is f precisely one for some d h(d ) w (d,c)and S w. f derivation d such that d = ∩ It was observed by Lang of (1994) that G can be seen as a parse forest , that is, a compact . representation The all parse trees according for to G that derive strings recognized by M construction (Vijay-Shanker Weir can be 1993) generalized to, example, tree-adjoining (Boullier grammars Bertsch and Nederhof and The range concatenation for grammars 2000; and 2001). construction (Seki the 1991). latter also has implications for linear context-free The rewriting systems et al. PCFG construction ( ( , N, S, has R, been ) extended and PFA by Nederhof N ( ,S , Q, and = Σ q , Satta ,R a = R q f , (2003) T, p M ). to The apply to G M 0 output a PCFG Σ = ,N ,S p G , ), where , is a G Σ p and are as before, and p is defined by p ((r 0 ,A,r m ) → (r 0 ,X 1 ,r 1 ) ··· (r m 1 ,X m ,r m )) = p G (A → X 1 ··· X m ) ((r,a,s) a) (r a p → = p M → s) (15) (16) Ifd , d, and c are such that h(d ) = (d,c), then clearly p (d ) = p G (d) · p M (c).

We restrict of ourselves model. to a few cases of the general technique of training a model on the basis another

Let us ( assume ,Q,q ,q we , have T) a proper is and consistent This PCFG FA may G = ( Σ ,N,S,R,p from G approximation of unambiguous. , N, S, R), it may have resulted (nonprob- ) and an FA M = Σ 0 f that abilistic) Note an FA Our is CFG ( Σ but is to to probabilities if it is also be totally unrelated that be unambiguous deterministic; any FA to G . guaranteed obtain assign to the transitions from FA can be determinized. proper PFA goal approximates now well that the probability distribution M to a possible. described by G as Let as us define that r, w, 1 as the s, function ((r,w) that c (s, maps )) each 1 if transition (r,w) c (s, from ), T to and ((r,w) one. This for c c 1 1 (s, means each and = )) = otherwise. 0 Of the set of strings generated most by one G , a subset is recognized of there can be at in such computation is for by computations each string. The of M ; note again that expected frequency a transition such computations given by E( ) (w) ((q ,w) c τ c ′ = ∑ p G · 1 0 (q f , w,c,c ′ )) (17) PFA Now ( ,Q,q we construct ,q , T, the ). Let PCFG (r G as s) explained T in section ((r,a,s) 4 from f → ∈ = → a). the On PCFG G and 1 and the basis of the Σ 0 a = the Computational Linguistics Volume 31, Number 2 properties of function h, we can now rewrite E( )as E( ) ∑ (S w) ((q ,w) c τ c ′ d = p 1 0 (q f , )) d,w,c,c ′ = ∑

e,d,w,c,c ′ : h(e) = (d,c τ c (S w) ((q ,w) c τ c ′ d p 1 0 (q f , )) ′ ) e ρ e ′ = ∑ p ∩ (S ∩ w) e,e ′ ,w = E( ) (18) Hereby we have expressed the of expected frequency ((r,a,s) of a) transition in (r a a = It of in frequency rule = was in derivations PCFG s) in terms the expected section 3 such value be computed. Note that since ∩ . explained inner(a) 1. 1 ( ) = 1, how by Therefore, also p ∩ ( ) = 1. Furthermore, a can definition for the right-hand side a of , = E( ) = outer((r,a,s)) p ∩ ( ) inner(a) = outer((r,a,s)) (19)

To obtain the required PFA ( Σ ,Q,q 0 ,q f ,T,p ), we now define the probability function for each (r a p = s) T as ( ) outer((r,a,s)) p = [BAR] ∑ outer((r,a ′ ,s ′ ,s :(r a ′ ) T )) a ′ ′ → s ′ ∈ That on such a domain relative L( frequency ) is proven estimator in p minimizes the KL distance between p and p the An example with is the finite languages given appendix. in Figure 1. We have, for example, (q 0 a 1 ) outer((q 0 ,a,q 1 )) p q = [BAR] outer((q 0 ,a,q 1 )) + outer((q 0 ,c,q 1 )) (20)

Similarly , T, ) to section CFG 5.1, we ( now , N, assume S, R) we have is a proper PFA = Our is ( Σ , Q, q 0 , q f p and a = well p that lets proper Σ that and consistent PCFG unambiguous. ( Σ , N, S, R, p ) approximate goal to find a function as possible. may Although CFGs in other used for natural in which language we may processing are usually ambiguous, there be cases fields assume grammars are unambiguous. as Nederhof Training Models on Models Figure 1 Example of input PCFG PCFG , G , with rule probabilities PFA. between square brackets, input FA M , the reduced G ∩ and the resulting trained Let us define 1 as the , function that maps each in rule . from The R to one. Of the strings in M a subset can be derived G expected frequency of set of recognized by is a rule ρ those derivations given by E( ) ∑ (w) (S d ρ d ′ ρ = p M · 1 ⇒ w) d,d ′ ,w (22) Now we construct the PCFG G ∩ from the PCFG G = PFA in 4. 5.1, ( Σ we , N, (A as X explained X section Analogously to section obtain S, R, for 1 ) and the M ) each ρ = → 1 ··· E( ρ ) = = To obtain the (A required )as Σ G now define the function p G for each ρ = → α p G ( ρ ) = ∑ ρ ′ = (A → α ′ ) ∈ R E( ρ ′ ) The proof that on this relative domain frequency L( ) is estimator identical p G minimizes p G almost to the proof the in KL distance between p M and G from the 5.1. the appendix for a similar claim section E( ρ ) [BAR] (24)

We now ( ,Q assume ,q we ,q have ,T ) a proper is PFA M 1 = Our ( Σ ,Q is 1 , q 0,1 , q f,1 ,T 1 ,p 1 ) and an FA M 2 = Σ 2 0,2 f,2 2 that unambiguous. goal to find a function p 2 so that m ∑ E((r 0 ,A,r m ) → (r 0 ,X 1 ,r 1 ) ··· (r m − 1 ,X m ,r m )) r 0 ,r 1 , ... ,r m ∑ outer((r 0 ,A,r m )) · inner((r 0 ,X 1 ,r 1 ) ··· (r m − 1 ,X m ,r m )) r 0 ,r 1 , ... ,r m (23) PCFG ( ,N,S,R,p ), we probability Computational Linguistics Volume 31, Number 2 proper KL PFA ( Σ distance ,Q 2 ,q 0,2 ,q f,2 ,T 2 ,p 2 ) approximates on domain M L( 1 as ). well as possible, minimizing the between Onewaytosolve p1 and p 2 the thisproblemistomap M 2 from 5.2. M The 2 to an equivalent obtained right-linear probability CFG G and then to apply the algorithm section from to an appropriate function . For function p G can be translated back 4 “cross-product” p2 this special case, can to the construction of the construction (see, section be simplified from 3 Ullman 1972). The simplified forms of finite automata e.g., Aho and the functions , inner commonly backward respectively, and outer section are of called forward and opposed As a result, we and they are defined by to systems linear approximate equations. solutions iteration. can compute exact solutions, as by Appendix We Leibler now prove distance that the choice of p M in section , 5.1 is to such that it minimizes KL and p M restricted the domain L( M ). the Kullback- between distance is p G given Without this restriction, the by D(p ‖ ) ∑ (w) log p G (w) G p M = p G · [BAR] p M (w)

w This approximating can be used a CFG for many is applications mentioned in section be such that L( 1. M ) For G ⊇ L( example, G ) in an FA the of most M approximation guaranteed (25) is infinite, However, to if case practical (w) 0, algorithms. of there are strings of w . We suchthat w ∈ / L( M ) and p G domain > then L( ) regardless it to obtain the choice p M therefore restrict p G to the M and normalize p G|M (w) p G (w) = [BAR] Z , if w ∈ L( M ) (25) (26) (27) where Z = ∑ w:w ∈ L( M ) p G (w). Note that p G|M = our of minimizes p G if L( M ) ⊇ L( G ). Our goal is now to show that choice p M 0, otherwise D(p G|M ‖ p M ) = w:w ∑ ∈ L( M ) p G|M (w) · log [BAR] p p G|M M (w) (w) (28) As Z is independent of p M , it is sufficient to show that our choice of p M minimizes ∑ (w) log p G (w) p G · [BAR] w:w L( ) p M (w) ∈ M (29) Now consider the expression ∏ p ( τ ) E( τ ) M τ (30) Nederhof Training Models on Models By the usual of proof in technique 5.1, with given Lagrange multipliers, it is easy to show that our choice p M section by E( τ ) [BAR] p M ( τ ) = ∑ τ ′ ,a ′ ,s ′ : τ ′ = (r → a ′ s ′ ) ∈ T E( τ ′ ) (31) for each properness. τ = (r a → s) ∈ T, is such that it maximizes (30), under the constraint of For τ ∈ Tand of occurrences w ∈ Σ ∗ , we of define in # τ (w)tobezero,if (unique) w ∈ / L( M ), and otherwise w. to be # the (w) number ((q ,w) (q , τ )). We the (30) computation that recognizes Formally, c τ c ′ τ = ∑ c,c ′ 1 0 f rewrite as ∏ p ( τ ) E( τ ) ∏ p ( τ ) ∑ w p G (w) · # τ (w) M = M τ τ ∏ ∏ p ( τ ) p G (w) · # τ (w) = M

w τ ( ) p G (w) ∏∏ p ( τ ) # τ (w) = M w τ ∏ p (w) p G (w) = M w:p M (w) > 0 ∏ 2 p G (w) · logp M (w) = w:p M (w) > 0 ∏ 2 p G (w) · logp M (w) − p G (w) · logp G (w) + p G (w) · logp G (w) = w:p M (w) > 0 − 2 G (w) log p G (w) ∏ p · [BAR] p (w) + p G (w) · logp G (w) = M w:p M (w) > 0 − 2 ∑ w:p M (w) 0 G (w) · log p G (w) p [BAR] > p (w) 2 ∑ w:p (w) > 0 p G (w) · logp G (w) = M · M (32) (31) We have impossible implies already (w) seen 0 that for the choice of p M that all of that w ∈ L( maximizes M )and p G (w) (30) is p w such > 0. given Since by (31), > p M (w) and M L( ), > 0is for w ∈ / M the value is determined L( ) solely (w) by 0. p G This and implies by the condition (30) is that maximized p M (w) > 0 for all w such that w ∈ M and p G > that by choosing p M such that (33) ∑ w:p (w) 0 (w) log p G (w) − p G · [BAR] 2 M > p M (w) (34) Computational Linguistics Volume 31, Number 2 is maximized, or alternatively that ∑ (w) log p G (w) p G · [BAR] w:p (w) 0 p M (w) M > (35) is minimized, (w) 0. For under the constraint of , that (29) p M (w) (35). > 0 for all w such that w ∈ L( M ) and p G > this Conversely, if choice choice of p M equals a L( )and p M minimizes (w) 0, (29), we may otherwise (29) assume is that w w ∈ M p > since infinite. p M (w) > 0 for all such choiceof that , (29) (35). It follows G of Again, p that the choice that minimizes (29) for this with M of equals maximizes (30), which p M our proof. concurs the choice p M that concludes Acknowledgments Comments Yuval by Khalil Krymolowski, Sima’an, Giorgio Satta, and anonymous reviewers is are gratefully acknowledged. PIONIER The author supported for Linguistic by the Processing, Project Algorithms NWO (Dutch Organization for Scientific funded by Research). References Aho, Parsing, Alfred volume V. and 1 Jeffrey of The D. Theory Ullman. Parsing, 1972. Translation Englewood Cliffs, Compiling NJ. . Prentice of and Hall, Bar-Hillel, E. Shamir. Yehoshua, 1964. On M. Perles, properties and formal Yehoshua Bar-Hillel, structure editor, grammars. Language In of simple phrase Information: Selected Essays Their Theory and . Addison-Wesley, on and MA, Application 116–150. Reading, Bertsch, pages On Eberhard and Mark-Jan of RCG of Nederhof. 2001. the complexity some extensions Seventh International parsing. Workshop In Proceedings of Parsing the Technologies , 66–77, Beijing, on October. pages Booth, 1973. Taylor L. and probabilistic Richard A. measures Thompson. Applying IEEE Transactions to abstract Computers languages. , C-22(5):442–450. on Boullier, Pierre. In 2000. International Proceedings Range concatenation grammars. Workshop Parsing of the Sixth Technologies , 53–64, on February. pages Trento, Italy, Jurafsky, Tajchman, Daniel, Jonathan Chuck Segal, Wooters, Gary Stolcke, The Eric Fosler, Restaurant and Nelson Andreas 1994. Berkeley Project. Morgan. Proceedings International Conference In Spoken Language of the Processing , on Yokohama, (ICSLP-94) pages 2139–2142, Japan. Lang, Bernard. 1994. Recognition Computational can be harder Intelligence than , 10(4):486–494. parsing. Manning, Christopher Schütze. 1999. Foundations D. and Hinrich Natural Language Processing of . Statistical Cambridge, MA. MIT Press, Mohri, Mehryar. in 1997. Finite-state transducers processing. Computational language and Linguistics speech , Mohri, 23(2):269–311. Mehryar Regular approximation and Mark-Jan Nederhof. 2001. through of context-free grammars transformation. Noord, In Robustness J.-C. Junqua Language and G. van Speech Technology editors, . Kluwer in and 153–163. Academic, Nederhof, pages Mark-Jan. with 2000. Practical experiments of regular approximation Linguistics context-free , languages. Computational Nederhof, Mark-Jan 26(1):17–44. Probabilistic and Giorgio intersection. Satta. In 2003. Proceedings parsing Eighth as Workshop of Parsing the Technologies International 137–148, Laboratoire on Lorrain , pages informatique de recherche en (LORIA), Nancy, et France, ses applications Paz, 1971. Introduction April. Azaria. . Press, New to Probabilistic Rimon, Automata Mori Academic J. Herz. 1991. The York. and recognition capacity In Proceedings of local syntactic constraints. Conference European Chapter of the Fifth ACL, of 155–160, the Berlin, of the Santos, Eugene pages S. 1972. Probabilistic April. grammars Control, and automata. Information and Seki, Hiroyuki, 21:27–47. Mamoru Fujii, Takashi Tadao Matsumura, 1991. On multiple and Kasami. Theoretical Computer context-free Science , grammars. 88:191–229. Nederhof Training Models on Models Starke, Peter H. 1972. North-Holland, Abstract Amsterdam. Automata . Stolcke, Andreas Precise and N-gram Jonathan Segal. probabilities from 1994. stochastic context-free Proceedings ACL, the 32nd grammars. In of Annual 74–79, Las Meeting of the NM, June. pages Cruces, Vijay-Shanker, 1993. The K. of and David J. Weir. use shared forests in tree adjoining grammar parsing. In Proceedings European of the Chapter Sixth Conference , of the Utrecht, The of the ACL Netherlands, pages 384–393, Zue, Victor, James Glass, April. Hong Leung, Michael David Goodine, Polifroni, Phillips, Stephanie Joseph and Integration of Seneff. 1991. speech recognition processing in and natural language Voyager In Proceedings the MIT ICASSP-91 system. , Toronto, volume 1, of the 713–716. pages