What constitutes a good model of tile learning behavior? Below we list tlve basic elements that any formal model of learning must con<,. (c.f. \[13\]) 1. Objects to be learned: l,ct us call them ~knacks' for full

generality. The question of learnability is asked of a col-

lection of knacks. 2. Environment: The way in whidl 'data' are available to tile

learner. 3. I\[ypotheses: I)escriptious t))r 'knacks', usually CXl)ressed

in a certain language. 4. /,earners: Ill general functions from data to hypotheses. 5. Criterion of l,earning: \])efines precisely what is meant by

the question; When is a learner said to 'learn' a giwm

collection of 'knacks' on the basis of data obtained through

the enviromnent ?

In most cases 'knacks' can be thought of as subsets of some universe (set) of objects, from which examples are drawn. 1 (Such a set is often called the 'domain' of the learning problem.) The obvions example is the definition of what a language is in the theory of natural language syntax. Syntactically, the English language is nothing but the set of all grammatical sentences, although this is subject to much philosophical controversy. The corresponding mathematical notion of a formal language is one that is fi'ee of such a controversy. A formal language is a subset of the set of all strings in .E* for some alphabet E. Clearly E* is tile domMn. The characterization of a kna& as a subset of a universe is in fact a very general one. For example, a boolean concept of n variables is a subset of the set of all assignments to those n variables, often written 2 '~. Positive examples in this case are assignments to the n variables which 'satisfy' the concept in question.

When the 'knacks' under consideration can in fact be thought of as subsets of some domain, the overall picture of a learning model looks like the one given in Figure 1. 2.2 Polynomial Learnability Polynomial learnability departs from the classic paradigm of language learning, 'idenitification in the limit', ~ in at least two important aspects, lilt enforces a higher demand oil tile time

1First order structures are an example in which langtlages arc more than just subsets of some set \[14\].

2Identification in the limit w¢~s originally proposed and studied by Gold \[8\], and has subsequently been generalized in many diflbrent ways. See for example \[13\] for a comprehensive treatment of this and related paradigms. The Environment o The Knacks The Hypotheses The Domain The Learner The Crileriony Figure 1: A Learning Model complexity by requiring that the learner converge in time polynomial, but on the other hand relaxes the criterion of what constitutes a 'correct' grammar by employing an approximate, and probabilistic notion of correctness, or aecraey to be'precise. Furthermore, this notion of correctness is intricately tied to both the time complexity requirement and the way in which the environment presents examples to the learner, Specifically, the environment is assumed to present to the learner examples from the domain with respect to an unknown (to the learner) but fixed probability distribution, and the accuracy of a hypothesis is measured with respect to that same probability distribution. This way, the learner is, so to speak, protected from 'bad' presentations of a knack. We now make these ideas precise by specifying the five essential parameters of this learning paradigm. 1. Objects to be learned are languages or subsets of ?2" for

some fixed alphabet E. Although we do not specify apri-

ori the language in which to express these grammars a, for

each collection of languages Z; of which we ask the learn-

ability, we fix a class of grammars G (such that L(~) = £

where we write L(~) to mean {L(G) I G E ~}) with re-

spect to which we will define the notion of 'complexity' or

'size' of a language. We take the number of bits it takes to

write down a grammar under a reasonable 4,fixed encod-

ing scheme to be the size of the grammar. The size of a

language is then defined as the size of a minimal grammar

for it. (For a language L, we write size(L) for its size.) 2. The environment produces a string in E* with a time-

invariant probability distribution unknown to the learner

and pairs it with either 0 or 1 depending on whether the

string is in the language in question or not, gives it to the

learner. It repeats this process indefinitely. 3. The hypotheses axe expressed as grammars. The class of

grammars allowed as hypotheses, say "H, is not necessarily

required to generate exactly the class Z; of languages to be

learned. In general, when a collection £ can be learned by

a learner which only outputs hypotheses from a class 7"/,

we say that £ is learnable by Tl, and in particular, when

Z; = L(~)) is learnable by ~, the class of representations G

is said to be properly learnable. (See \[6\].) 4. Learners passively receive an infinite sequence of positive

and negative examples in the manner described above, and

aPotentAally any 'l?urning program could be a hypothesis

~By a reasonblc encoding, we mean one which can represent n ditrerent. grannnars using O(log*~) bits. at each initial (finite) segment of such a sequence, output a hypothesis. In other words, they are functions from finite sequences of positive and negative examples 5to grammars. 5. A learning function is said to polynomially learn a col2.3 Blumer et al. \[5\] have shown an extremely interesting result revealing a connection between reliable data compression and polynomial learnability. Occam's l~azor is a principle in the philosophy of science which stipulates that a shorter theory is tobe preferred as long as it remains adequate. B\]umel" el; al. define a precise version of such a notion in the present context of learning which they call Occam Algorithm, and establishes a relation between the existence of such an algorithm and polynomiM learnability: If there exists a polynomial time algorithm which reliably "compresses" any sample of any language in a given collection to a provably small consistent grammar for it, then such an Mogorithm polynomially learns that collection in the limit. We state this theorem in a slightly weaker form. Definition 2.1 Let £ be a language collection with associated represenation ~ with size function "size". (Define a sequence of subclasses of ~ by 7~n = {G e 7-\[ \] size(G) _< n}.) Then A is an Occar(~ algorithm for £ with range size f(m, ~z) if and only if! VLE£

VS C graph(L)

if size(L) = n and \] S I= m then

A(S) is consistent with S

and A(S)) e 7~I(,~,m )

and .A runs in time polynomial in the length of S. Theorem 2.1 (Blumer et al.) If A is an Occam algorithm for f~ with range size f(n,m) = O(nk~ ~) for some k >_ ; 0 < c~ < 1 then .4 polynomially learns £ in the limit. We give below an intuitive explication of why an 0cesta Algorithm polynomiMly learns in the limit. Suppose A is an Occam Algorithm for £, and let L ~ l: be the language to be learned, and n its size. Then for an arbitrary sample for L of an arbitrary size, a minimal consistent language for it will never have size larger than size(L) itself. Hence A's output on a sample of size m will always be one of the hypotheses in H\](m,~), whose cardinality is at most 2\](~,n). As the sample size m grows, its effect on the probability that any consistent hypothesis in 7~i(,~,, 0 is accurate will (polynomially) soon dominate that of the growth of the eardinality of the hypothesis class, which is less than linear in the sample size. Sin the sequel, we shall call them 'labeled samples' SThe symmetric difference between two sets A and B is (A-B)U(B-A). rFor any langugage L, ~jraph(L) = {(x, O} I x C-: L} U {{a:, I) \] a: ~ L}. lection of languages just in case it is computable in time polynomial ill the length of the input sample, and for an arbitrary degrees of accuracy e and confidence 5, its output on a sample produced by the environment by the manner described above for any language L in that collection, will be an e-approximation of the unknown language L with confidence probability at least 1 -- a, no matter what the unknown distribution is, as long as the number of strings in the sample exceeds p(e -~, 5 -~, size (L)) for some fixed plynomial p. Here, grammar G is an e-approximation of language L, if the probability distribution over the symmetric difference 6of L and I,(G) is at most e. Occam Algorithm

In this section, we define l, hc class of nrihlly context sensitive grammars under consideration, or Ranked Node Rewriting (\]ram.mars (RNR(~'s). \[{NR(\]'s are based on the underlying ideas of Tree Adjoining Grammars (TArt's) s and are also a specical case of context fi'ee tree grammars \[15\] in which unres~,ricted use of w~rial)les for moving, copying and deleting, is not permitted, in other words each rewriting in this system replaces a "ranked" noclterminal node of say rank j with an "incomplete" tree containing exactly j edges that have no descendants. If we define a hierarchy of languages generated by subclasses of RNRG's having nodes and rules with hounded rank j (RNRLj), then RNRL0 = CFL, and RNRLa :: TAL. 9 We formally define these grammars below. Definition 'LI (Preliminaries) 77ze following definitions are necessar!l Jb'," the ,~equel. (i) The set of labeled directed trees over an alphabet E is denoted 7;> (ii) r\['ll.e Ta.'ll.'. of an "incomplete" tree is the number of outgoing edges with no descendents. (iii) The rarth oj'a node is the. number of outgoing edges. (iv) The ~u& 4'a symbol is defined if the rank of any node labeled by it is always the same, and equal~ that rank. (v) A ranked alphabet is one in which every symbol has a rank. (vi) I,l)r writ,': rank(x) for the rank of a~ything x, if it is defined. Definition 3.2 (Ranked Node Rewriting Grammars) A ronl;ed nodt; re'writing grammar C is a q'uinl,ph' {>',,v, E'e, ~, It,., Re;) where: (i) EN is a ranked nonterminal alphabet. (ii) );'r is a germinal alphabet di4oint fi'om F~N. We let ~; = }-;N U 2T. (iii) ~ is a distinguished symbol distinct from any member of E, indicating "a'a outgoing edge with no descendent", m (iv) It; is a finite set of labeled trees over E. We refer ~o I(; as ~he "initial trees" of the grammar. (v) Ra is a finite set of rewriting rules: R<~ C {(A,a} I A e Y,'N & a C T~u{.} & rank(A) = rank(re)}. (In the sequel, we write A --. o for rewriting rule {A, ce).) (vO ,'a,,V(c) = ,ha, {,-~,4.(A) I A e EN}. We emphasize that the nonterminM vs. terminal distinction above does not coiadde with the internal node vs. frontier node distinction. (See examples 2.1 - 2.3.) tiaving defined the notions of 'rewriting' and 'derivation' in the obvious manner, the tree language of a grammar is then defiimd as the set of trees over the terminal alphabet, whid~ can be derived fi'om the grammar. 11 This is analogous to the way the string language of a rewriting grammar in the Chomsky hierarchy is defined. Definition 3.:"1 ('IYee Languages and String Languages) The tree language and string Iang~tagc of a RNRG G, denoted

s'\]?ree adjoitdng grammars were introduced a.s a formalism for linguistic description by aoshi et al. \[10\], \[9\]. Various formal and computational properties of TAG's were studied in \[17\]. Its linguistic relevance was demons~rated in \[12\].

9This hierar,:hy is different fi'om the hierarchy of "meta-TAL's" invented and studied exl.ensively by Weir in \[20\].

l°ln context free t.ree grammars iu \[15\], variables are used in place of ~J. 'l'hese variables can then be used in rewriting rules to move, copy, or erase subtrees.. \[t is i;his restriction of avoiding such use of variables Hint keeps RNR,G's within the class of etlicient, ly recognizable rewriting systems called "Linear context fi'ee rewriting systems" (\[18\]).

II'Phis is how an "obligatory adjunction constraint" in the tree adjoining

nunar formalism can be sintulated. S

S IV.. b a 8 f S derived : s a s a s f S $ f b # c d # c S d s s I a a j\[--. b s c d b # c

b )v c d Figurc 2: a, fl, 7 and deriving 'aabbccddeeff' by G:~ T((;) and Leo repectively, are defined as follows; s e )~ e /~(c') = {.,ji~ld(~) t ~ ~ T(O)}. If we now define a hierarchy of languages generated by subclasses of RNRG's with bounded ranks, context fi'ee languages ((',FL) and tree adjoining languages (TAt) constitute the first two members of the hierarchy. Definition 3.4 l;br each j ~ N RNI~Gj = {GIG C RNRG & rank(G) < J}. l;br each j ~ N, I{NIU, j = {L(C) I O e: antiC;;} Theorem 3.1 I{NI~Lo - CFL ~tn.d l~N I~\[.1 : !I'AL. We now giw; some examples of grammars in this laierarchy, J2 which also illustrate the way in which the weak generative capacity of different levels of this hierarchy increases progressively. Example 3.1. 1), = {3% ~ \[ n. C N} C Gl' , is generated by the following l?~Nl~(_7o 9rammar~ where o' is shown in Figure 2. 6', = ({s}, {,,a,b},L {s'}, {,5'--~ ~,,~ + s(~)}) Example 3.2 I)2 -- {a'W~c'~d '~ \] n G N} C- TAL is ocher, ted by the following \]~N I~G1 grammar, where/~ is shown in Figure 2. C;~=({S},{s,a,b,e,d},~,{(S(,~))},{S'-, ,'<S'+s(~)}> Example 3.3 L3 = {a'%'*c'~d'~e'~f '~ I n C N} ¢ TAL is generated by the following RNI?,G2 grnmn~ar, where 7 is shown 5* t,'igure 2. C;':~ = ({S'}, {s, a, b, ,.', d, c, f}, ~, {(,5'(A, A))}, {5'--, 7, ,5'-~ ,~(~, I1)})

q'he notion of qocality' of a grammar we define in this paper is a measure of how much global dependency there is within the grammar. By global dependency within a gramnlar, we. mean the interactions that exist between different rules and nonterminals in the grammar. As it is intuitively clear, allowing unbounded amont of global interaction is a major, though not only, cause of a combinatorial explosion in a search for a right grammar. K-locality limits the amount of such interaction, by

tSSimpler trees are represented as term struct.ures, whereas lnore involved trees are shown in the figure. Also note that we rise uppercase letters for nonterminals and lowercase for terminals.

IaSome linguistic motiwltions of this extension of'lDkG's are argagned for by the author in \[1\]. bounding the number of different rules that can participate in any slngle derivation.

Pormally, the notion of "k-locality" of a grammar is defined with respect to a formulation of derivations due originally to Vijay-Shankar, Weir, and 3oshi (\[\[9\]), which is a generalization of the notion of parse trees for CFO's. In their formulation, a derivation is a tree recording the tfistory of rewritings. The root of a derivation tree is labeled with an initial tree, and the rest of the nodes with rewriting rules. Each edge corresponds to a rewriting; the edge from a rule (host rule) to auother rule (applied rule) is labeled with the address of the node in the host l, ree at which the rewriting takes place.

The degree of locality of a derivation is the number of distinct kinds of rewritings that appear in it. In terms of a derivation tree, the degree of locality is the number of different kinds of edges in it, where two edges are equivalent just in ease the two end nodes are labeled by the same rules, and the edges themselves are labeled by the same node address. Definition 4.1 Let 7)(G) denote the set of all derivation trees of G, and let r 6 D(G). Then, the degree of locality oft, written locality(r), is d4ned as follows, locality(r) = card{(p,q,,t) I there is an edge in r from a node labeled with p to another labeled with q, and is itself labeled with 77} The degree of locality of a gramm,~r is the maximum of those of all its derivations. Definition 4.2 a RNRG G is called k-local if max{locality(r) \] r e ~(C)} _< k. We write k-Local-I~NRO - {(7 I G (5 RNRG and G is k-Local} and k-Local-t2Nl~L = { L(G) I G C k-Local-i~NR(: }, etc.. Example 4.1 L1 = {a"bna"b '' I n,m C N} ~ /t-Local-RNRLo since all the derivations of G, -({S}, {s,a,b}, ~, {s(S,S)}, {S -+ sea, S,b), S --~ A}) generating Lt have deflree of locality at most 4. l,br example, the derivation for the string a3b3ab has degree of locality 4 as shown in Figure 8.

Because locality of a derivation is the number of distinct kinds of rewritings, inclusive of the positions at which they takc place, k-locality also puts a bound on the number of nonterminal occurrences in any rule. In fact, had we defined the notion of klocality by the two conditins: (i) at most k rules take part in any derivation, (if) each rule is k-bounded, t4, the analogous learnability result would follow essentially by the same argument. So, k-locality in effect forces a grammar to be an unbounded union of boundedly simple grammar, with bounded number of rules each boundedly small, with a bounded number of nonterminals. This fact is captured formally by the existence of the following normal form with only a polynomial expansion factor. Lelnma 4.1 (K-Local Normal Form) For every k-Local-RNRGj G, if we let n = size(G), then there is a RNRGj G' such that ~. L( C') = r,,( a). 2. c' is in k-local normal form, i.c. O' = U{1\]~ I i C -rG,}

such that: (a) each lIi has a nonterminal set that is: disjoint from

any other IIj. (b) each tI~ is k-sire, pie, that is i. each Ili contains exactly i initial tree. a Sb s 2 s 2 s ---../1Xm a S b a Sb locality(~-) = 4 s s s s--*A A m s.. A a Sb S S a Sb s s s -'/1",, a S b a Sb s.o ,/-: A 2..£- s-*A.--S S I s s-./l',, aS b s s s s 2 A S S s aS b Figure 3: Degree of locality of a derivation of a3b3ab by G1 if. each Hi contains at most k rules. iii. each IIi contains at most k nonterminal occurrences. s. ~i~e(c~") = o(~+'). Crucially, the constraint of k-locality on RNRG's is an interesting one because not only each k-local subclass is an exponential class containing infinitely many infinite languages, but also klocal subclasses of the RNRG hierarchy become progressively more complex as we go higher in the hierarchy. In particular, for each j, IlNP~Gj can "count up to" 2(j + 1) and for each k > 2, k-local-RN\[4Gj can also count up to 2(j + 1)) 5 We summarize these properties of k-loeal-RNRL's below. Theorem 4.1 Pbr every k E N, 1. Vj E N UkeN k-local-RNRLj = RNRLj. ~. Vj C N Vk > 3 k-local-RNRLj+l is incomparable with

RNRLp 3. Vj, k ~ N k-local:RNRLj is a p~oper subset of (k+I)-

loeal-t~NRLj. 4. Vj Vk > 2 E N k-local-RNRLj contains infinitely many

infinite languages. hfformal t'roof:

1 is obvious because for each grammar in RNRLj, the degree of locality o~" the grannnar is finite.

As for 2, we note that the sequence of the languages (for the first three of which we gave example grammars) L~ = {a~*a~...a~ I u ~ N} are each in 3-1ocal-RNRLI_I but not in RNRLi_2.

To verii} 3, we give the following sequence of languages Lj,k such that for each j and k, Lj, k is in k-local-RNRLj but not in (k-1)-local-RNRL/. Intuitively this is because k-local-languages can have at most O(k) mutually independent dependencies ina single sentence. Example 4.2 For each j, k ~ N, let Lj,k = { al ~ ...a20+1 '~ ) al 2,~2 ...a 2~, (j+l) ... a knk 1 ...a2(j~t) kn~ \]nl,n2,...,nk e N}.

is obvious because Zoo = Uwe~.Lw where Lt~ = {w" \] n e N} are a subset of 2-1ocal-I~NRL0, and hence is a subset of k:localRNl~Lj for every j and k >_ 2. £¢¢ clearly contains inifinitely many infinite languages. \[\]