Computational Linguistics Volume 26, Number 1 When recursive(Ni) = left, Ni consists of only left-recursive nonterminals, which does not mean it cannot also contain right-recursive nonterminals, but in that case right recursion amounts to application of unit rules. When recursive(Ni) = cyclic, it is only such unit rules that take part in the recursion.

That recursive(Ni) = self, for some i, is a sufficient and necessary condition for the grammar to be self-embedding. Therefore, we have to prove that if recursive(Ni) E {left, right, cyclic}, for all i, then the grammar generates a regular language. Our proof differs from an existing proof (Chomsky 1959a) in that it is fully constructive: Figure 2 presents an algorithm for creating a finite automaton that accepts the language generated by the grammar.

The process is initiated at the start symbol, and from there the process descends the grammar in all ways until terminals are encountered, and then transitions are created labeled with those terminals. Descending the grammar is straightforward in the case of rules of which the left-hand side is not a recursive nonterminal: the subautomata found recursively for members in the right-hand side will be connected. In the case of recursive nonterminals, the process depends on whether the nonterminals in the corresponding set from H are mutually left-recursive or right-recursive; if they are both, which means they are cyclic, then either subprocess can be applied; in the code in Figure 2 cyclic and right-recursive subsets Ni are treated uniformly.

We discuss the case in which the nonterminals are left-recursive. One new state is created for each nonterminal in the set. The transitions that are created for terminals and nonterminals not in Ni are connected in a way that is reminiscent of the construction of left-corner parsers (Rosenkrantz and Lewis 1970), and specifically of one construction that focuses on sets of mutually recursive nonterminals (Nederhof 1994, Section 5.8).

An example is given in Figure 3. Four states have been labeled according to the names they are given in procedure make~fa. There are two states that are labeled qB. This can be explained by the fact that nonterminal Bcan be reached by descending the grammar from S in two essentially distinct ways.

The code in Figure 2 differs from the actual implementation i that sometimes, for a nonterminal, a separate finite automaton isconstructed, namely, for those nonterminals that occur as A in the code. A transition in such a subautomaton may be labeled by another nonterminal B, which then represents the subautomaton corresponding to B. The resulting representation is similar to extended context-free grammars (Purdom and Brown 1981), with the exception that in our case recursion cannot occur, by virtue of the construction.

The representation for the running example is indicated by Figure 4, which shows two subautomata, labeled S and B. The one labeled S is the automaton on the top level, and contains two transitions labeled B, which refer to the other subautomaton. Note that this representation is more compact han that of Figure 3, since the transitions that are involved in representing the sublanguage of strings generated by nonterminal B are included only once.

The compact representation consisting of subautomata can be turned into a single finite automaton by substituting subautomata A for transitions labeled A in other automata. This comes down to regular substitution in the sense of Berstel (1979). The advantage of this way of obtaining a finite automaton over a direct construction of a nondeterministic automaton isthat subautomata may be determinized and minimized before they are substituted into larger subautomata. Since in many cases determinized and minimized automata re much smaller, this process avoids much of the combinaNederhof Experiments with Regular Approximation let K = O, A = O, s = fresh_state, f= fresh_state, F = {f}; make_fa( s, S, f). procedure makeffa(qo, a, ql):

if a= e

then let A = A U {(q0,e, ql)}

elseif a = a, some a E ,U

then let A = A U {(q0, a, ql)}

elseif a = Xfl, some X E V, fl C V* such that IflI > 0

then let q = fresh_state;

makeffa(qo, X, q);

makeffa( q, t, ql )

else let A = a; (* a must consist of a single nonterminal *)

if there exists i such that A C Ni

then for each B E Ni do let qB = fresh_state end;

if recursive(Ni) = left

then for each (C-+ XI'.'Xm) E P such that CENi AX1,...,Xm~Ni

do make_fa(qo, XI " . Xm, qc )

end;

for each (C --+ DX1 ... X,~) C P such that

C,D ~ Ni A X1,...,Xm ~ Ni

do make ffa( qD , X I " " X,~ , qc )

end;

let A = A U {(qA, e, ql)}

else (* recursive(g,) C {right, cyclic} *)

for each (C-+ X1...Xm) E P such that CENi A X1,...,Xm~Ni

do make_fa(qc, X1... Xm, ql)

end;

for each (C --~ XI ".. XmD) E P such that

C, D E Ni AXI,...,Xm ~ Ni

do makc_fa(qc, XI ".. Xm, qD)

end;

let A = A U {(qo, e, qa)}

end

else for each (A -+ fl) C P do make_fa(qo,fl, ql) end (* A is not recursive *)

end

end end. procedure fresh_state():

create some object q such that q ~ K;

let K=KU{q};

return q end. Figure 2 Transformation from a grammar G = (E, N,P, S) that is not self-embedding into an equivalent finite automaton 3v = (K, E, A, s, F). Computational Linguistics S A A B B --* --* ~ --* ---* c qB Figure 3 Application of the code from Figure 2 on a small grammar. S B

. . . . . . . . . . . . . . . . . c . . . . . . . . . . . 1 i i i I ,,. d ~ (~) , f w - w = i qB I Figure 4 The automaton from Figure 3 in a compact representation. torial explosion that takes place upon naive construction of a single nondeterministic finite automaton. 1

Assume we have a list of subautomata A1 ..... Am that is ordered from lower-level to higher-level automata; i.e., if an automaton Ap occurs as the label of a transition of automaton Aq, then p < q; Am must be the start symbol S. This order is a natural result of the way that subautomata re constructed uring our depth-first raversal of the grammar, which is actually postorder in the sense that a subautomaton isoutput after all subautomata occurring at its transitions have been output.

Our implementation constructs a minimal deterministic automaton by repeating the following for p = 1,...,m: . Make a copy of Ap. Determinize and minimize the copy. If it has fewer transitions labeled by nonterminals than the original, then replace Ap by its copy. . Replace each transition in Ap of the form (q, Ar, q') by (a copy of) automaton Ar in a straightforward way. This means that new e-transitions connect q to the start state of Ar and the final states of Ar to qt. Aa SB Bb Bc d Volume 26, Number 1 N = {S,A,B} \]kf : {N1, N2} N1 = {S,A} recursive(N1) = left N2 -- {B} recursive(N2) = left __qA a d Nederhof Experiments with Regular Approximation

3. Again determinize and minimize Ap and store it for later reference. The automaton obtained for Am after step 3 is the desired result.

This section describes a number of methods for approximating acontext-free grammar by means of a finite automaton. Some published methods did not mention selfembedding explicitly as the source of nonregularity for the language, and suggested that approximations hould be applied globally for the complete grammar. Where this is the case, we adapt the method so that it is more selective and deals with self-embedding locally.

The approximations are integrated into the construction of the finite automaton from the grammar, which was described in the previous section. A separate incarnation of the approximation process is activated upon finding a nonterminal A such that A E Ni and recursive(Ni) = self, for some i. This incarnation then only pertains to the set of rules of the form B --* c~, where B E Ni. In other words, nonterminals not in Ni are treated by this incarnation of the approximation process as if they were terminals. 4.1 Superset Approximation Based on RTNs The following approximation was proposed in Nederhof (1997). The presentation here, however, differs substantially from the earlier publication, which treated the approximation process entirely on the level of context-free grammars: a self-embedding grammar was transformed in such a way that it was no longer self-embedding. A finite automaton was then obtained from the grammar by the algorithm discussed above.

The presentation here is based on recursive transition networks (RTNs) (Woods 1970). We can see a context-free grammar as an RTN as follows: We introduce two states qA and q~ for each nonterminal A, and m + 1 states q0 ..... qm for each rule A --* X1 .. • Xm. The states for a rule A ~ X 1 . . . X m are connected with each other and to the states for the left-hand side A by one transition (qA, c, q0), a transition (qi-1, Xi, qi) for each i such that 1 < i < m, and one transition (qm, e,q~A). (Actually, some epsilon transitions are avoided in our implementation, but we will not be concerned with such optimizations here.)

In this way, we obtain a finite automaton with initial state qA and final state q~ for each nonterminal A and its defining rules A --* X1 • • • Xm. This automaton can be seen as one component of the RTN. The complete RTN is obtained by the collection of all such finite automata for different nonterminals.

An approximation ow results if we join all the components in one big automaton, and if we approximate the usual mechanism of recursion by replacing each transition (q, A, q') by two transitions (q, c, qA) and (q~, e, q'). The construction is illustrated in Figure 5.

In terms of the original grammar, this approximation can be informally explained as follows: Suppose we have three rules B --* c~Afl, B I ~ c~IAfl ~, and A ~ % Top-down, left-to-right parsing would proceed, for example, by recognizing a in the first rule; it would then descend into rule A ~ % and recognize "y; it would then return to the first rule and subsequently process ft. In the approximation, however, the finite automaton "forgets" which rule it came from when it starts to recognize % so that it may subsequently recognize fl' in the second rule. Computational Linguistics a B Volume 26, Number 1 b (b) d A e (a) A---~ a B b A---~c A B---~dA e B--~ f qB >t~ Y'~ ;i = i ~ q 'B

"'"~"~ f a b (c) ~ ~ t / Figure 5 Application of the RTN method for the grammar in (a). The RTN is given in (b), and (c) presents the approximating finite automaton. We assume A is the start symbol and therefore qA becomes the initial state and q~ becomes the final state in the approximating automaton.

For the sake of presentational convenience, the above describes a construction working on the complete grammar. However, our implementation applies the construction separately for each nonterminal in a set Ni such that recursive(Ni) = self, which leads to a separate subautomaton of the compact representation (Section 3).

See Nederhof (1998) for a variant of this approximation that constructs finite transducers rather than finite automata.

We have further implemented a parameterized version of the RTN approximation. A state of the nondeterministic automaton is now also associated to a list H of length IHI strictly smaller than a number d, which is the parameter to the method. This list represents a history of rule positions that were encountered in the computation leading to the present state.

More precisely, we define an item to be an object of the form \[A ~ a • fl\], where A ~ aft is a rule from the grammar. These are the same objects as the "dotted" productions of Earley (1970). The dot indicates a position in the right-hand side.

The unparameterized RTN method had one state qI for each item/, and two states qA and q~ for each nonterminal A. The parameterized RTN method has one state qrH for each item I and each list of items H that represents avalid history for reaching I, and two states qaH and q~H for each nonterminal A and each list of items H that represents avalid history for reaching A. Such a valid history is defined to be a list Nederhof Experiments with Regular Approximation H with 0 < \[HI < d that represents aseries of positions in rules that could have been invoked before reaching Ior A, respectively. More precisely, if we set H =/1 ... In, then each Im (1 < m < n) should be of the form \[Am ~ olin • Bmflm\] and for 1 < m < n we should have Am -- Bm+l. Furthermore, for a state qiH with I = \[A --* a • fl\] we demand A = B1 if n > 0. For a state qAH we demand A -- B1 if n > 0. (Strictly speaking, states qAH and qrH, with \[HI < d - 1 and I = \[A --+ a • fl\], will only be needed if AIH \] is the start symbol in the case IH\[ > 0, or if A is the start symbol in the case H = c.)

The transitions of the automaton that pertain to terminals in right-hand sides of rules are very similar to those in the case of the unparameterized method: For a state qIH with I of the form \[A ~ a • aft\], we create a transition (q~H, a, qi,H), with I' = \[A ~ aa • fl\].

Similarly, we create epsilon transitions that connect left-hand sides and right-hand sides of rules: For each state qAa there is a transition (qAH, e, qIH) for each item I = \[A --* • a\], for some a, and for each state of the form qI,u, with I' = \[A ~ a •\], there is a transition (qFa, c, q~H).

For transitions that pertain to nonterminals in the right-hand sides of rules, we need to manipulate the histories. For a state qIH with I of the form \[A ~ a • Bfl\], we create two epsilon transitions. One is (qIH, c, qBn,), where H' is defined to be IH if \[IH\[ < d, and to be the first d - 1 items of IH, otherwise. Informally, we extend the history by the item I representing the rule position that we have just come from, but the oldest information in the history is discarded if the history becomes too long. The second transition is (q'BH,, ~, q~'H), with I' = \[A --* aB • fl\].

If the start symbol is S, the initial state is qs and the final state is q~ (after the symbol S in the subscripts we find empty lists of items). Note that the parameterized method with d -- 1 concurs with the unparameterized method, since the lists of items then remain empty.

An example with parameter d-- 2 is given in Figure 6. For the unparameterized method, each I = \[A --* a • fl\] corresponded to one state (Figure 5). Since reaching A can have three different histories of length shorter than 2 (the empty history, since A is the start symbol; the history of coming from the rule position given by item \[A -~ c • A\]; and the history of coming from the rule position given by item \[B ~ d • Ae\]), in Figure 6 we now have three states of the form qI~ for each I -- \[A ~ a • fl\], as well as three states of the form qA~r and q~H"

The higher we choose d, the more precise the approximation is, since the histories allow the automaton to simulate part of the mechanism of recursion from the original grammar, and the maximum length of the histories corresponds to the number of levels of recursion that can be simulated accurately. 4.2 Refinement of RTN Superset Approximation We rephrase the method of Grimley-Evans (1997) as follows: First, we construct he approximating finite automaton according to the unparameterized RTN method above. Then an additional mechanism is introduced that ensures for each rule A --~ X1 • .. Xm separately that the list of visits to the states qo,.. • • qm satisfies some reasonable criteria: a visit to qi, with 0 < i < m, should be followed by one to qi+l or q0. The latter option amounts to a nested incarnation of the rule. There is a complementary condition for what should precede a visit to qi, with 0 < i < m.

Since only pairs of consecutive visits to states from the set {q0 ..... qm} are considered, finite-state techniques uffice to implement such conditions. This can be realized by attaching histories to the states as in the case of the parameterized RTN method above, but now each history is a set rather than a list, and can contain at most one item \[A --* a • fl\] for each rule A ---* aft. As reported by Grimley-Evans (1997) and conComputational Linguistics a Volume 26, Number 1

b c , / a b A~a B b A~c A i H = \[A----> c.Al qA~_ ..... g ~', , ,, x , a II ,, b E , . . . . " H= \[B-->d.A el qA e',, ,,'" \-qA. d " ,,, , Z e _ B---,'d A e H = \[A --~ a.B b\] qB B--->f -5.. "'L Q~___ qBH Figure 6 Application of the parameterized RTN method with d = 2. We again assume A is the start symbol. States qm have not been labeled in order to avoid cluttering the picture. firmed by our own experiments, the nondeterministic finite automata resulting from this method may be quite large, even for small grammars. The explanation is that the number of such histories is exponential in the number of rules.

We have refined the method with respect to the original publication by applying the construction separately for each nonterminal in a set Ni such that recursive(Ni) = self. 4.3 Subset Approximation by Transforming the Grammar Putting restrictions on spines is another way to obtain a regular language. Several methods can be defined. The first method we present investigates pines in a very detailed way. It eliminates from the language only those sentences for which a subderivation is required of the form B --~* aBfl, for some a ~ ¢ and fl ~ e. The motivation is that such sentences do not occur frequently in practice, since these subderivations make them difficult for people to comprehend (Resnik 1992). Their exclusion will therefore not lead to much loss of coverage of typical sentences, especially for simple application domains.

We express the method in terms of a grammar transformation in Figure 7. The effect of this transformation is that a nonterminal A is tagged with a set of pairs (B, Q), where B is a nonterminal occurring higher in the spine; for any given B, at most one such pair (B, Q) can be contained in the set. The set Q may contain the element l to indicate that something to the left of the part of the spine from B to A Nederhof Experiments with Regular Approximation We are given a grammar G = (E,N, P, S). The following is to be performed for each set Ni EAf such that recursive(Ni) = self.

. For each A E Ni and each F E 2 (Nix2~l''}), add the following nonterminal

to N. • A F . 2. For each A E Ni, add the following rule to P. • A---~A 0. . For each (A --* o~0A1o~1A2... C~m-lAmCrm) E P such that A, A1 .... ,Am E Ni

and no symbols from c~0 .... , am are members of Ni, and each F such that

(A, (l, r}) ~ F, add the following rule to P. a F o~0A F1 1 oq... A Fm m O~m, where, for 1 G j _< m, --Fj= {(B, QUC~U~F) I (B,Q) E F'}; F' = FU {(A, 0)} if -~3Q\[(A,Q) E F\], and F' = F otherwise;

= 0 if c~0AlC~l...Aj-I~j-1 = c, and ~ = {l} otherwise; -QJr = 0 if o/.jaj+lOLj+l...AmOL m = £, and QJr = {r} otherwise. 4. Remove from P the old rules of the form A --* c~, where A E Ni. 5. Reduce the grammar. Figure 7 Subset approximation by transforming the grammar. was generated. Similarly, r E Q indicates that something to the right was generated. If Q = {l, r}, then we have obtained a derivation B --** c~Afl, for some c~ ~ c and fl ~ ~, and further occurrences of B below A should be blocked in order to avoid a derivation with self-embedding.

An example is given in Figure 8. The original grammar is implicit in the depicted parse tree on the left, and contains at least the rules S --+ A a, A --, b B, B -* C, and C --* S. This grammar is self-embedding, since we have a subderivation S --~* bSa. We explain how FB is obtained from FA in the rule A ~ --* b B r'. We first construct F' = {(S, {r}), (A, 0)} from FA = {(S, (r})} by adding (A, 0), since no other pair of the form (A, Q) was already present. To the left of the occurrence of B in the original rule A --* b B we find a nonempty string b. This means that we have to add l to all second components of pairs in F', which gives us FB = {(S, (l, r}), (A, {l})}.

In the transformed grammar, the lower occurrence of S in the tree is tagged with the set {(S, {I, r}), (A, {l}), (B, 0), (C, 0)}. The meaning is that higher up in the spine, we will find the nonterminals S, A, B, and C. The pair (A, (1}) indicates that since we saw A on the spine, something to the left has been generated, namely, b. The pair (B, 0) indicates that nothing either to the left or to the right has been generated since we saw B. The pair (S, {1, r}) indicates that both to the left and to the right something has been generated (namely, b on the left and a on the right). Since this indicates that an Computational Linguistics Volume 26, Number 1 s s B C s Figure 8 A parse tree m a self-embedding grammar (a), and the corresponding parse tree in the transformed grammar (b), for the transformation from Figure 7. For the moment we ignore step 5 of Figure 7, i.e., reduction of the transformed grammar. (a) (b) s Fs /\ A b a /\ b a B 'B I ~F c X offending subderivation S --** c~Sfl has been found, further completion of the parse tree is blocked: the transformed grammar will not have any rules with left-hand side S {(S'{I'r})'(A'{I})'(B'O)'(C'O)}. In fact, after the grammar is reduced, any parse tree that is constructed can no longer even contain a node labeled by S {(s'U'r})'(a'{O)'(B'°)'(c'°)}, or any nodes with labels of the form A r such that (A, {l,r}) c F.

One could generalize this approximation in such a way that not all self-embedding is blocked, but only self-embedding occurring, say, twice in a row, in the sense of a subderivation of the form A --** alAfll --+* oqol2Afl2fll. We will not do so here, because already for the basic case above, the transformed grammar can be huge due to the high number of nonterminals of the form A F that may result; the number of such nonterminals is exponential in the size of Ni.

We therefore present, in Figure 9, an alternative approximation that has a lower complexity. By parameter d, it restricts the number of rules along a spine that may generate something to the left and to the right. We do not, however, restrict pure left recursion and pure right recursion. Between two occurrences of an arbitrary rule, we allow left recursion followed by right recursion (which leads to tag r followed by tag rl), or right recursion followed by left recursion (which leads to tag l followed by tag lr).

An example is given in Figure 10. As before, the rules of the grammar are implicit in the depicted parse tree. At the top of the derivation we find S. In the transformed grammar, we first have to apply S --* S -r'°. The derivation starts with a rule S --* A a, which generates astring (a) to the right of a nonterminal (A). Before we can apply zero or more of such rules, we first have to apply a unit rule S T,° --* S r,° in the transformed grammar. For zero or more rules that subsequently generate something on the left, such as A ~ b B, we have to obtain a superscript containing rl, and in the example this is done by applying A r,° ~ A rl,°. Now we are finished with pure left recursion and pure right recursion, and apply B rl,O ---+ B ±,0. This allows us to apply one unconstrained rule, which appears in the transformed grammar as B ±,° ---* c S T'I d. Fs = 0 FB {(S, {l, r}), (A, {/})} Fc = {(S, {l, r}), (A, {/}), (B, 0)} ' -- {(S, {l, r}), (A, {/}), (B, 0), (C, 0)} Nederhof Experiments with Regular Approximation We are given a grammar G = (G, N, P, S). The following is to be performed for each set Ni C .IV" such that recursive(Ni) = self. The value d stands for the maximum number of unconstrained rules along a spine, possibly alternated with a series of left-recursive rules followed by a series of right-recursive rules, or vice versa. 1. For each A c Ni, each Q E { T, l, r, It, rl, 3_ }, and each f such that

0 < f < d, add the following nonterminals to N. • AQ,f. 2. For each A E Ni, add the following rule to P. • A ---+ A T'0. 3. • • • • • • 4. • 5. • 6. • 7. 8. Figure 9 A simpler subset approximation by transforming the grammar.

Now the counter f has been increased from 0 at the start of the subderivation to 1 at the end. Depending on the value d that we choose, we cannot build derivations by repeating subderivation S --+* b c S d a an unlimited number of times: at some point the counter will exceed d. If we choose d = 0, then already the derivation at For each A E Ni and f such that 0 G f G d, add the following rules to P. AT,f ___+ Al,f. ATd: __+ Ar,f. Aid ---+ Alr,f. Ar,f ---, A~l,/. Atr,f __+ A±,d. Arl,f ___+ A±,f. For each (A -+ Ba) ~ P such that A, B c Ni and no symbols from ~ are members of Ni, eachf such that 0 <f G d, and each Q E {r, lr}, add the following rule to P. AQd ~ BQ/a. For each (A --+ c~B) E P such that A, B E Ni and no symbols from c~ are members of Ni, eachf such that 0 Gf < d, and each Q c {l, rl}, add the following rule to P. Aqd ~ c~BQ,f. For each (A -~ o~0AloqA2... O~m-lAmC~m) C P such that A, A1 ..... Am E Ni and no symbols from s0 ..... C~m are members of Ni, and each f such that 0 < f G d, add the following rule to P, provided m = 0 vf < d. A±/ ---4 c~0Alq-d+lc~l . . .~l AT,f+1 m ' OLm . Remove from P the old rules of the form A ~ c~, where A E Ni. Reduce the grammar. Computational Linguistics Volume 26, Number 1

S S !T,O (a) /~ (b)!r,O A a So\ ' a rl, O b B b B rl'O :o t t t t Figure 10 t t A parse tree in a self-embedding rammar (a), and the corresponding parse tree in the transformed grammar (b), for the simple subset approximation from Figure 9. t t Figure 10 (b) is no longer possible, since no nonterminal in the transformed grammar would contain 1 in its superscript.

Because of the demonstrated increase of the counter f, this transformation is guaranteed to remove self-embedding from the grammar. However, it is not as selective as the transformation we saw before, in the sense that it may also block subderivations that are not of the form A --** ~Afl. Consider for example the subderivation from Figure 10, but replacing the lower occurrence of S by any other nonterminal Cthat is mutually recursive with S, A, and B. Such a subderivation S ---** b c C d a would also be blocked by choosing d = 0. In general, increasing dallows more of such derivations that are not of the form A ~" o~Afl but also allows more derivations that are of that form.

The reason for considering this transformation rather than any other that eliminates self-embedding ispurely pragmatic: of the many variants we have tried that yield nontrivial subset approximations, this transformation has the lowest complexity in terms of the sizes of intermediate structures and of the resulting finite automata.

In the actual implementation, we have integrated the grammar transformation and the construction of the finite automaton, which avoids reanalysis of the grammar to determine the partition of mutually recursive nonterminals after transformation. This integration makes use, for example, of the fact that for fixed Ni and fixed f, the set of nonterminals of the form A,f, with A c Ni, is (potentially) mutually right-recursive. A set of such nonterminals can therefore be treated as the corresponding case from Figure 2, assuming the value right.

The full formulation of the integrated grammar transformation and construction of the finite automaton is rather long and is therefore not given here. A very similar formulation, for another grammar transformation, is given in Nederhof (1998). Nederhof Experiments with Regular Approximation

The methods of approximation described above take as input the parts of the grammar that pertain to self-embedding. It is only for those parts that the language is affected. This leads us to a way to increase the precision: before applying any of the above methods of regular approximation, we first transform the grammar.

This grammar transformation copies grammar rules containing recursive nonterminals and, in the copies, it replaces these nonterminals by new nonrecursive nonterminals. The new rules take over part of the roles of the old rules, but since the new rules do not contain recursion and therefore do not pertain to self-embedding, they remain unaffected by the approximation process.

Consider for example the palindrome grammar from Figure 1. The RTN method will yield a rather crude approximation, namely, the language {a, b}*. We transform this grammar in order to keep the approximation process away from the first three levels of recursion. We achieve this by introducing three new nonterminals S\[1\], S\[2\] and S\[3\], and by adding modified copies of the original grammar rules, so that we obtain: S\[1\] S\[2\] S\[3\]

S The new start symbol is S\[1\].

The new grammar generates the same language as before, but the approximation process leaves unaffected the nonterminals S\[1\], S\[2\], and S\[3\] and the rules defining them, since these nonterminals are not recursive. These nonterminals amount to the upper three levels of the parse trees, and therefore the effect of the approximation on the language is limited to lower levels. If we apply the RTN method then we obtain the language that consists of (grammatical) palindromes of the form ww R, where w E {¢, a, b} U {a, b} 2 U {a, b} 3, plus (possibly ungrammatical) strings of the form wvw R, where w E {a, b} 3 and v E {a, b}*. (w R indicates the mirror image of w.)

The grammar transformation in its full generality is given by the following, which is to be applied for fixed integer j> 0, which is a parameter of the transformation, and for each Ni such that recursive(Ni) = self.

For each nonterminal A E Ni we introduce jnew nonterminals All\] ..... A~\]. For each A --, X1...Xm in P such that A E Ni, and h such that 1 ~ h < j, we add A\[h\] --* X'I... X" to P, where for 1 < k < m: aS\[2\]a \] bS\[2\] b I ¢ aS\[3\]a \] bS\[3\] b I c aSa l bSb i c aSa i bSb i e X~k = Xk\[h + 1\] if X k E Ni /X h < j = Xk otherwise Further, we replace all rules A --* X1 ... Xm such that A ~ Ni by A --* X~ ... X~m, where for 1 < k < m: X~ -- Xk\[1\] ifXkENi = Xk otherwise If the start symbol S was in Ni, we let S\[1\] be the new start symbol.

A second transformation, which shares some characteristics with the one above, was presented in Nederhof (1997). One of the earliest papers suggesting such transformations as a way to increase the precision of approximation is due to ~ulik and Cohen (1973), who only discuss examples, however; no general algorithms were defined.

Computational Linguistics Volume 26, Number 1 • -5 ._N_ 300

250 E E 200 E I I I I I I Figure 11 The test material. The left-hand curve refers to the construction of the grammar from 332 sentences, the right-hand curve refers to the corpus of 1,000 sentences used as input to the finite automata.

In this section we investigate mpirically how the respective approximation methods behave on grammars of different sizes and how much the approximated languages differ from the original context-free languages. This last question is difficult to answer precisely. Both an original context-free language and an approximating regular language generally consist of an infinite number of strings, and the number of strings that are introduced in a superset approximation or that are excluded in a subset approximation may also be infinite. This makes it difficult to attach numbers to the "quality" of approximations.

We have opted for a pragmatic approach, which does not require investigation of the entire infinite languages of the grammar and the finite automata, but looks at a certain finite set of strings taken from a corpus, as discussed below. For this finite set of strings, we measure the percentage that overlaps with the investigated languages.

For the experiments, we took context-free grammars for German, generated automatically from an HPSG and a spoken-language corpus of 332 sentences. This corpus consists of sentences possessing rammatical phenomena of interest, manually selected from a larger corpus of actual dialogues. An HPSG parser was applied on these sentences, and a form of context-free backbone was selected from the first derivation that was found. (To take the first derivation is as good as any other strategy, given that we have at present no mechanisms for relative ranking of derivations.) The label occurring at a node together with the sequence of labels at the daughter nodes was then taken to be a context-free rule. The collection of such rules for the complete corpus forms a context-free grammar. Due to the incremental nature of this construction of the grammar, we can consider the subgrammars obtained after processing the first p sentences, where p = 1, 2, 3 ..... 332. See Figure 11 (left) for the relation between p and the number of rules of the grammar. The construction is such that rules have at most two members in the right-hand side.

As input, we considered aset of 1,000 sentences, obtained independently from the 332 sentences mentioned above. These 1,000 sentences were found by having a speech recognizer provide a single hypothesis for each utterance, where utterances come from actual dialogues. Figure 11 (right) shows how many sentences of different lengths the corpus contains, up to length 30. Above length 25, this number quickly declines, but still a fair quantity of longer strings can be found, e.g., 11 strings of a length between Nederhof Experiments with Regular Approximation 51 and 60 words. In most cases however such long strings are in fact composed of a number of shorter sentences.

Each of the 1,000 sentences were input in their entirety to the automata, although in practical spoken-language systems, often one is not interested in the grammaticality of complete utterances, but tries to find substrings that form certain phrases bearing information relevant o the understanding ofthe utterance. We will not be concerned here with the exact way such recognition of substrings could be realized by means of finite automata, since this is outside the scope of this paper.

For the respective methods of approximation, we measured the size of the compact representation ofthe nondeterministic automaton, the number of states and the number of transitions of the minimal deterministic automaton, and the percentage of sentences that were recognized, in comparison to the percentage of grammatical sentences. For the compact representation, we counted the number of lines, which is roughly the sum of the numbers of transitions from all subautomata, not considering about three additional lines per subautomaton for overhead.

We investigated the size of the compact representation because it is reasonably implementation i dependent, barring optimizations of the approximation algorithms themselves that affect the sizes of the subautomata. For some methods, we show that there is a sharp increase in the size of the compact representation for a small increase in the size of the grammar, which gives us a strong indication of how difficult it would be to apply the method to much larger grammars. Note that the size of the compact representation is a (very) rough indication of how much effort is involved in determinization, minimization, and substitution of the subautomata into each other. For determinization and minimization of automata, we have applied programs from the FSM library described in Mohri, Pereira, and Riley (1998). This library is considered to be competitive with respect to other tools for processing of finite-state machines. When these programs cannot determinize or minimize in reasonable time and space some subautomata constructed by a particular method of approximation, then this can be regarded as an indication of the impracticality of the method.

We were not able to compute the compact representation for all the methods and all the grammars. The refined RTN approximation from Section 4.2 proved to be quite problematic. We were not able to compute the compact representation for any of the automatically obtained grammars in our collection that were self-embedding. We therefore eliminated individual rules by hand, starting from the smallest selfembedding rammar in our collection, eventually finding grammars mall enough to be handled by this method. The results are given in Table 1. Note that the size of the compact representation i creases ignificantly for each additional grammar ule. The sizes of the finite automata, after determinization and minimization, remain relatively small.

Also problematic was the first approximation from Section 4.4, which was based on LR parsing following Pereira and Wright (1997). Even for the grammar of 50 rules, we were not able to determinize and minimize one of the subautomata ccording to step 1 of Section 3: we stopped the process after it had reached a size of over 600 megabytes. Results, as far as we could obtain them, are given in Table 2. Note the sharp increases in the size of the compact representation, resulting from small increases, from 44 to 47 and from 47 to 50, in the number of rules, and note an accompanying sharp increase in the size of the finite automaton. For this method, we see no possibility of accomplishing the complete approximation process, including determinization and minimization, for grammars in our collection that are substantially larger than 50 rules.

Since no grammars of interest could be handled by them, the above two methods will be left out of further consideration. Computational Linguistics Volume 26, Number 1 Table 1 Size of the compact representation and number of states and transitions, for the refined RTN approximation (Grimley-Evans 1997). Grammar Size Compact Representation # of States # of Transitions 10 12 13 14 15 16 17 18 19

133

427

1,139

4,895

16,297

51,493

208,350

409,348 1,326,256 11 17 17 17 17 19 19 21 21 14 26 34 36 40 52 52 59 61 Table 2 Size of the compact representation and number of states and transitions, for the superset approximation based on LR automata following Pereira and Wright (1997). Grammar Size Compact Representation # of States # of Transitions 35 44 47 50 15,921 24,651 151,226 646,419

350

499 5,112

? 2,125 4,352 35,754

?

Below, we refer to the unparameterized and parameterized approximations based on RTNs (Section 4.1) as RTN and RTNd, respectively, for d = 2,3; to the subset approximation from Figure 9 as Subd, for d = 1, 2, 3; and to the second and third methods from Section 4.4, which were based on LR parsing following Baker (1981) and Bermudez and Schimpf (1990), as LR and LRd, respectively, for d = 2, 3. We refer to the subset approximation based on left-corner parsing from Section 4.5 as LCd, for the maximal stack height of d = 2, 3, 4; and to the methods discussed in Section 4.6 as Unigram, Bigram, and Trigram.

We first discuss the compact representation of the nondeterministic automata. In Figure 12 we use two different scales to be able to represent the large variety of values. For the method Subd, the compact representation isof purely theoretical interest for grammars larger than 156 rules in the case of Sub1, for those larger than 62 rules in the case of Sub2, and for those larger than 35 rules in the case of Sub3, since the minimal deterministic automata could thereafter no longer be computed with a reasonable bound on resources; we stopped the processes after they had consumed over 400 megabytes. For LC3, LC4, RTN3, LR2, and LR3, this was also the case for grammars larger than 139, 62, 156, 217, and 156 rules, respectively. The sizes of the compact representation seem to grow moderately for LR and Bigram, in the upper panel, yet the sizes are much larger than those for RTN and Unigram, which are indicated in the lower panel.

The numbers of states for the respective methods are given in Figure 13, again using two very different scales. As in the case of the grammars, the terminals of our finite automata re parts of speech rather than words. This means that in general there will be nondeterminism during application of an automaton on an input sentence due to lexical ambiguity. This nondeterminism can be handled efficiently using tabular Nederhof Experiments with Regular Approximation i , ! i i i ! , i i :t i \] / / \[ I \[ /j / / / / \] / ; i ; ; ! / / i i / i i /' ," ., ," " LC4

LR3--x---.

RTN3 -~

LC3

LR2 ..... Trigram -~'---

LC2 -e---

RTN2

LR -+ -Bigram-E3--r'~ E O o ~: ~ ;' ~ /' ! /" // / ," ,4- ...... + .... ,' " / /' .-" /'~ _..+ ........ 4- ..... 4-"""

"" - ---E3- ............ E} ....... E}----n .... --'- ......... 1 ........ I I ' ~ - ~ I I I I "3 & ,; o i z~ ... /

"'""'' ,,,~ ::'~ ..... I I I I I I Figure 12 Size of the compact representation. techniques, provided the number of states is not too high. This consideration favors methods that produce low numbers of states, such as Trigram, LR, RTN, Bigram, and Unigram. IJ :/ /" /" /

RTN2

LC2

LR Bigram

Sub3

Sub2

Sub1

RTN Unigram / /.' ,.: / ,,: .: ,,• y~..,.' ,"

,,..,' ...." ,,,'

,,,

,'" ... ,' ......... ..,.. .... ,' . .. "~ . ......' ..-a--e--+ --' -~---x .... "a ..... -~ ..... -~,--.,0. Computational Linguistics Volume 26, Number 1 Sub2

LC4 Sub1

LC3 RTN3

LC2 ..A ..... -8---~ ..... -x--e-4000 2000 1000

0 f / j / \[ / / ~~~i °~-'~-~ ~ 100 , , , , x ,,,,,," ~--~.~¢" / 40 : ~ ~ d~3._-.D .D ---\[\]'--......... ~E\]" *...... \[\] ........... " ~" --e-- "<> Figure 13 Number of states of the deterrninized and minimized automata. ..e- ......... e ..... -e ....... ~ ...... i

200 i i i i

Note that the numbers of states for LR and RTN differ very little. In fact, for some of the smallest and for some of the largest grammars in our collection, the resulting automata were identical. Note, however, that the intermediate results for LR ,

LC3

RTN3

LR3 --x- --

LC2 -e--

RTN2 TrigramLR2 -.~--~----

LR -+ --

RTN -B~ Bigram -a-Unigram -~--Nederhof Experiments with Regular Approximation (Figure 12) are much larger. It should therefore be concluded that the "sophistication" of LR parsing is here merely an avoidable source of inefficiency.

The numbers of transitions for the respective methods are given in Figure 14. Again, note the different scales used in the two panels. The numbers of transitions roughly correspond to the storage requirements for the automata. It can be seen that, again, Trigram, LR, RTN, Bigram, and Unigram perform well.

The precision of the respective approximations is measured in terms of the percentage of sentences in the corpus that are recognized by the automata, in comparison to the percentage of sentences that are generated by the grammar, as presented by Figure 15. The lower panel represents an enlargement of a section from the upper panel. Methods that could only be applied for the smaller grammars are only presented in the lower panel; LC4 and Sub2 have been omitted entirely.

The curve labeled G represents the percentage of sentences generated by the grammar. Note that since all approximation methods compute either supersets or subsets, a particular automaton cannot both recognize some ungrammatical sentences and reject some grammatical sentences.

Unigram and Bigram recognize very high percentages of ungrammatical sentences. Much better results were obtained for RTN. The curve for LR would not be distinguishable from that for RTN in the figure, and is therefore omitted. (For only two of the investigated grammars was there any difference, the largest difference occurring for grammar size 217, where 34.1 versus 34.5 percent of sentences were recognized in the cases of LR and RTN, respectively.) Trigram remains very close to RTN (and LR); for some grammars a lower percentage is recognized, for others a higher percentage is recognized. LR2 seems to improve slightly over RTN and Trigram, but data is available only for small grammars, due to the difficulty of applying the method to larger grammars. A more substantial improvement is found for RTN2. Even smaller percentages are recognized by LR3 and RTN3, but again, data is available only for small grammars.

The subset approximations LC3 and Sub1 remain very close to G, but here again only data for small grammars is available, since these two methods could not be applied on larger grammars. Although application of LC2 on larger grammars required relatively few resources, the approximation is very crude: only a small percentage of the grammatical sentences are recognized.

We also performed experiments with the grammar transformation from Section 5, in combination with the RTN method. We found that for increasing j, the intermediate automata soon became too large to be determinized and minimized, with a bound on the memory consumption of 400 megabytes. The sizes of the automata that we were able to compute are given in Figure 16. RTN+j, for j = 1, 2, 3,4, 5, represents the (unparameterized) RTN method in combination with the grammar transformation with parameter j. This is not to be confused with the parameterized RTNd method.

Figure 17 indicates the number of sentences in the corpus that are recognized by an automaton divided by the number of sentences in the corpus that are generated by the grammar. For comparison, the figure also includes curves for RTNd, where d = 2, 3 (cf. Figure 15). We see that j = 1, 2 has little effect. For j = 3,4, 5, however, the approximating language becomes substantially smaller than that in the case of RTN, but at the expense of large automata. In particular, if we compare the sizes of the automata for RTN+j in Figure 16 with those for RTNd in Figures 13 and 14, then Figure 17 suggests the large sizes of the automata for RTN+j are not compensated adequately by a reduction of the percentage of sentences that are recognized. RTNd seems therefore preferable to RTN+j.

Computational Linguistics Volume 26, Number 1 i Sub2

LC4 Sub1

LC3

LC2 RTN2 --~ ...... --~ ..... z~ o 0000 / /! //'/' / ' /.D -''0 .

/';'"

";Y" I I .:..i~ ...13---_ D / _ ...... ~- ..... 43".... (3 .... Figure 14 Number of transitions of the determinized and minimized automata. 7. Conclusions If we apply the finite automata with the intention of filtering out incorrect sentences, for example from the output from a speech recognizer, then it is allowed that a I

LC3 -x---

RTN3

LR3 --x- --

LC2 -e~

RTN2 Trigram -~---

LR2 -~,- --

LR -÷ --

RTN Bigram -B-Unigram -~- -Nederhof Experiments with Regular Approximation

,,' e- ...... ¢, -- --e- -- -~- ..... '~ ..'

Unigram

Bigram ,e---~ Trigram

RTN(LR)

RTN2G

LC2 -4- --\[\]--x--.-~--~---~a--e-•13-*-£3 ,.O ........ ~" "'"

~3 ..... (3" 13 ...... El----\[\]'-'-o 6O & oo \[\] ....... Ey • / , - _. lq___ + /' ,' ,~' 0 0 ._.---e--~O 0 • C\] // \[\] i / • ,D / "E // /' ,\[:: I I I Figure 15 Percentage of sentences that are recognized. certain percentage of ungrammatical input is recognized. Recognizing ungrammatical input merely makes filtering less effective; it does not affect the functionality of the system as a whole, provided we assume that the grammar specifies exactly the set of sentences that can be successfully handled by a subsequent phase of pro-

I / / ,'i ,j:/ / i- , . .I I I i

Bigram -D-RTN(LR) Trigram -x-.-

LR2 -,~- --

RTN2 -~--

LR3 --x- --

RTN3 -+---

G --t--.

LC3 -x--

Sub1 --~ .....

LC2 -e-I Computational Linguistics Volume 26, Number 1 i i i i i i i , i i i i i i i RTN+5 -~-RTN+4 -0-RTN+3 -+-RTN+2 -o-RTN+I ..... :!! RTN+5 -z~-RTN+4 -o-RTN+3 -+-RTN+2 -ra-RTN+I "~'" :i o a %0 , / ~,, ,, ;/ .4- -" a ii j } ! o ,+" ~-E\],\[3 _~ ^ ,, / :"~'~"" ............. 50 100 150 200 250 300 350 400 450 500 550

grammar size Figure 16 Number of states and number of transitions of the determinized and minimized automata. i i

RTN RTN2 -a--RTN3 RTN+I -~<-RTN+2 -D-RTN+3 -+-RTN+4 -o-RTN+5 -a-s Figure 17 Number of recognized sentences divided by number of grammatical sentences. cessing. Also allowed is that "pathological" grammatical sentences are rejected that seldom occur in practice; an example are sentences requiring multiple levels of selfembedding.

Of the methods we considered that may lead to rejection of grammatical sentences, i.e., the subset approximations, none seems of much practical value. The most serious problem is the complexity of the construction of automata from the compact representation for large grammars. Since the tools we used for obtaining the minimal Nederhof Experiments with Regular Approximation deterministic automata are considered to be of high quality, it seems unlikely that alternative implementations could succeed on much larger grammars, especially considering the sharp increases in the sizes of the automata for small increases in the size of the grammar. Only LC2 could be applied with relatively few resources, but this is a very crude approximation, which leads to rejection of many more sentences than just those requiring self-embedding.

Similarly, some of the superset approximations are not applicable to large grammars because of the high costs of obtaining the minimal deterministic automata. Some others provide rather large languages, and therefore do not allow very effective illtering of ungrammatical input. One method, however, seems to be excellently suited for large grammars, namely, the RTN method, considering both the unparameterized version and the parameterized version with d = 2. In both cases, the size of the automaton grows moderately in the grammar size. For the unparameterized version, the compact representation also grows moderately. Furthermore, the percentage of recognized sentences remains close to the percentage of grammatical sentences. It seems therefore that, under the conditions of our experiments, this method is the most suitable regular approximation that is presently available. Acknowledgments This paper could not have been written without the wonderful help of Hans-Ulrich Krieger, who created the series of grammars that are used in the experiments. I also owe to him many thanks for countless discussions and for allowing me to pursue this work. I am very grateful to the anonymous referees for their inspiring suggestions.

This work was funded by the German Federal Ministry of Education, Science, Research and Technology (BMBF) in the framework of the VERBMOBIL Project under Grant 01 IV 701 V0. References Baker, Theodore P. 1981. Extending

lookahead for LR parsers. Journal of

Computer and System Sciences, 22:243-259. Bermudez, Manuel E. and Karl M. Schimpf.

1990. Practical arbitrary lookahead LR

parsing. Journal of Computer and System

Sciences, 41:230-250. Berstel, Jean. 1979. Transductions and

Context-Free Languages. B. G. Teubner,

Stuttgart. Black, Alan W. 1989. Finite state machines

from feature grammars. In International

Workshop on Parsing Technologies, pages

277-285, Pittsburgh, PA. Chomsky, Noam. 1959a. A note on phrase

structure grammars. Information and

Control, 2:393-395. Chomsky, Noam. 1959b. On certain formal

properties of grammars. Information and

Control, 2:137-167. Culik, Karel II and Rina Cohen. 1973.

LR-regular grammars--An extension of

LR(k) grammars. Journal of Computer and

System Sciences, 7:66-96. Earley, Jay. 1970. An efficient context-free

parsing algorithm. Communications of the

ACM, 13(2):94-102, February. Grimley-Evans, Edmund. 1997.

Approximating context-free grammars

with a finite-state calculus. In Proceedings

of the 35th Annual Meeting of the Association

for Computational Linguistics an 8th

Conference of the European Chapter of the

Association for Computational Linguistics,

pages 452-459, Madrid, Spain. Harrison, Michael A. 1978. Introduction to

Formal Language Theory. Addison-Wesley. Heckert, Erik. 1994. Behandlung von

Syntaxfehlern fiir LR-Sprachen ohne

Korrekturversuche. Ph.D. thesis,

Ruhr-Universit/it Bochum. Johnson, Mark. 1998. Finite-state Krauwer, Steven and Louis des Tombe. 1981.

Transducers and grammars as theories of

language. Theoretical Linguistics, 8:173-202. Langendoen, D. Terence and Yedidyah

Langsam. 1987. On the design of finite

transducers for parsing phrase-structure

languages. In Alexis Manaster-Ramer,

editor, Mathematics of Language. John

Benjamins, Amsterdam, pages 191-235. Mohri, Mehryar and Fernando C. N. approximation of constraint-based grammars using left-comer grammar transforms. In COLING-ACL "98: 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, volume 1, pages 619-623, Montreal, Quebec, Canada. Computational Linguistics

Pereira. 1998. Dynamic compilation of

weighted context-free grammars. In

COLING-ACL "98: 36th Annual Meeting of

the Association for Computational Linguistics

and 17th International Conference on

Computational Linguistics, volume 2, pages

891-897, Montreal, Quebec, Canada. Mohri, Mehryar, Femando C. N. Pereira,

and Michael Riley. 1998. A rational design

for a weighted finite-state transducer

library. In Derick Wood and Sheng Yu,

editors, Automata Implementation. Lecture

Notes in Computer Science, Number 1436.

Springer Verlag, pages 144-158. Nederhof, Mark-Jan. 1994. Linguistic Parsing

and Program Transformations. Ph.D. thesis,

University of Nijmegen. Nederhof, Mark-Jan. 1997. Regular

approximations of CFLs: A grammatical

view. In Proceedings of the International

Workshop on Parsing Technologies,

pages 159-170, Massachusetts Institute of

Technology. Nederhof, Mark-Jan. 1998. Context-free

parsing through regular approximation.

In Proceedings of the International Workshop

on Finite State Methods in Natural Language

Processing, pages 13-24, Ankara, Turkey. Nederhof, Mark-Jan and Giorgio Satta. 1996.

Efficient tabular LR parsing. In Proceedings

of the 34th Annual Meeting, pages 239-246,

Santa Cruz, CA. Association for

Computational Linguistics. Pereira, Fernando C. N. and Rebecca N.

Wright. 1997. Finite-state approximation

of phrase-structure grammars. In Volume 26, Number 1

Emmanuel Roche and Yves Schabes,

editors, Finite-State Language Processing.

MIT Press, pages 149-173. Pulman, S. G. 1986. Grammars, parsers, and

memory limitations. Language and

Cognitive Processes, 1(3):197-225. Purdom, Paul Walton, Jr. and Cynthia A.

Brown. 1981. Parsing extended LR(k)

grammars. Acta Informatica, 15:115-127. Resnik, Philip. 1992. Left-corner parsing and

psychological plausibility. In COLING '92:

Papers presented to the Fifteenth \[sic\]

International Conference on Computational

Linguistics, pages 191-197, Nantes, France. Rosenkrantz, D. J. and P. M. Lewis, II. 1970.

Deterministic left comer parsing. In IEEE

Conference Record of the 11th Annual

Symposium on Switching and Automata

Theory, pages 139-152. Seyfarth, Benjamin R. and Manuel E.

Bermudez. 1995. Suffix languages in LR

parsing. International Journal of Computer

Mathematics, 55:135-153. Stolcke, Andreas and Jonathan Segal. 1994.

Precise N-gram probabilities from

stochastic ontext-free grammars. In

Proceedings of the 32nd Annual Meeting,

pages 74-79, Las Cruces, NM. Association

for Computational Linguistics. Woods, W. A. 1970. Transition network

grammars for natural anguage analysis.

Communications of the ACM,

13(10):591-606.