A fundamental debate in the machine learning of language has been the role of prior knowledge in the learning process. Purely nativist approaches, such as the Principles and Parameters model, build parameterized linguistic generalizations directly into the learning system. Purely empirical approaches use a general, domain-independent learning rule (Error Back-Propagation, Instance-based Generalization, Minimum Description Length) to learn linguistic generalizations directly from the data.

In this paper we suggest that an alternative to the purely nativist or purely empiricist learning paradigms is to represent the prior knowledge of language as a set of abstract learning biases, which guide an empirical inductive learning algorithm. We test our idea by examining the machine learning of simple Sound Pattern of English (SPE)-style phonological rules. We represent phonological rules as finite-state transducers that accept underlying forms as input and generate surface forms as output. We show that OSTIA, a general-purpose transducer induction algorithm, was incapable of learning simple phonological rules like flapping. We then augmented OSTIA with three kinds of learning biases that are specific to natural language phonology, and that are assumed explicitly or implicitly by every theory of phonology: faithfulness (underlying segments tend to be realized similarly on the surface), community (similar segments behave similarly), and context (phonological rules need access to variables in their context). These biases are so fundamental to generative phonology that they are left implicit in many theories. But explicitly modifying the OSTIA algorithm with these biases allowed it to learn more compact, accurate, and general transducers, and our implementation successfully learns a number of rules from English and German. Furthermore, we show that some of the remaining errors in our augmented model are due to implicit biases in the traditional SPE-style rewrite system that are not similarly represented in the transducer formalism, suggesting that while transducers may be formally equivalent to SPE-style rules, they may not have identical evaluation procedures.

Because our biases were applied to the learning of very simple SPE-style rules, and to a non-psychologically-motivated and nonprobabilistic theory of purely deterministic transducers, we do not expect that our model as implemented has any practical use as a phonological learning device, nor is it intended as a cognitive model of human learning. Indeed, because of the noise and nondeterminism inherent to linguistic data, we feel strongly that stochastic algorithms for language induction are much more likely to be a fruitful research direction. Our model is rather intended to suggest the kind of biases that may be added to other empiricist induction models, and the way in which they may be added, in order to build a cognitively and computationally plausible learning model for phonological rules.

* 1947 Center Street, Berkeley, CA 94704. E-mail: gildea@cs.berkeley.edu t Department of Linguistics, Boulder, CO 80302

A fundamental debate in the machine learning of language has been the role of prior knowledge in the learning process. Nativist models suggest that learning in a com­plex domain like natural language requires that the learning mechanism either have some previous knowledge about language, or some learning bias that helps direct the formation of correct generalizations. In linguistics, theories of such prior knowledge are referred to as Universal Grammar (UG); nativist linguistic models of learning as­sume, implicitly or explicitly, that some kind of prior knowledge that contributes to language learning is innate, a product of evolution. Despite sharing this assumption, nativist researchers disagree strongly about the exact constitution of this Universal Grammar. Many models, for example, assume that much of the prior knowledge that children bring to bear in learning language is not linguistic at all, but derives from constraints imposed by our general cognitive architecture. Others, such the influen­tial Principles and Parameters model (Chomsky 1981), assert that what is innate is linguistic knowledge itself, and that the learning process consists mainly of search­ing for the values of a relatively small number of parameters. Such nativist models of phonological learning include, for example, Dresher and Kaye's (1990) model of the acquisition of stress-assignment rules, and Tesar and Smolensky's (1993) model of learning in Optimality Theory.

Other scholars have argued that a purely nativist, parameterized learning algo­rithm is incapable of dealing with the noise, irregularity, and great variation of human language data, and that a more empiricist learning paradigm is possible. Such data-driven models include the stress acquisition models of Daelemans, Gillis, and Durieux (1994) (an application of Instance-based Learning [Aha, Kibler, and Albert 1991]) and Gupta and Touretzky (1994) (an application of Error Back-Propagation), as well as Elli­son's (1992) Minimum-Description-Length-based model of the acquisition of the basic concepts of syllabicity and the sonority hierarchy. In each of these cases a general, domain-independent learning rule (BP, IBL, MDL) is used to learn directly from the data.

In this paper we suggest that an alternative to the purely nativist or purely em­piricist learning paradigms is to represent the prior knowledge of language as a set of abstract learning biases, which guide an empirical inductive learning algorithm. Such biases are implicit, for example, in the work of Riley (1991) and Withgott and Chen (1993), who induced decision trees to predict the realization of a phone in its context. By initializing the decision-tree inducer with a set of phonological features, they es­sentially gave it a priori knowledge about the kind of phonological generalizations that the system might be expected to learn.

Our idea is that abstract biases from the domain of phonology, whether innate (i.e., part of UG) or merely learned prior to the learning of rules, can be used to guide a domain-independent empirical induction algorithm. We test this idea by examining the machine learning of simple Sound Pattern of English (SPE)-style phonological rules (Chomsky and Halle 1968), beginning by representing phonological rules as finite-state transducers that accept underlying forms as input and generate surface forms as output. Johnson (1972) first observed that traditional phonological rewrite rules can be expressed as regular (finite-state) relations if one accepts the constraint that no rule may reapply directly to its own output. This means that finite-state transducers (FSTs) can be used to represent phonological rules, greatly simplifying the problem of parsing the output of phonological rules in order to obtain the underlying, lexical forms (Koskenniemi 1983; Karttunen 1993; Pulman and Hepple 1993; Bird 1995; Bird and Ellison 1994). The fact that the weaker generative capacity of FSTs makes them easier to learn than arbitrary context-sensitive rules has allowed the development of a number of learning algorithms including those for deterministic finite-state automata (FSAs) (Freund et al. 1993), deterministic transducers (Oncina, Garcia, and Vidal 1993), as well as nondeterministic (stochastic) FSAs (Stolcke and Omohundro 1993; Stolcke and Omohundro 1994; Ron, Singer, and Tishby 1994). Like the empiricist models discussed above, these algorithms are all general-purpose; none include any domain knowledge about phonology, or indeed natural language; at most they include a bias toward simpler models (like the MDL-inspired algorithms of Ellison [1992]).

Our experiments were based on the OSTIA (Oncina, Garcia, and Vidal 1993) al­gorithm, which learns general subsequential finite-state transducers (SFSTs; formally defined in Section 2). We presented pairs of underlying and surface forms to OSTIA, and examined the resulting transducers. Although OSTIA is capable of learning ar­bitrary SFSTs in the limit, large dictionaries of actual English pronunciations did not give enough samples to correctly induce phonological rules.

We then augmented OSTIA with three kinds of learning biases, which are specific to natural language phonology, and are assumed explicitly or implicitly by every the­ory of phonology: faithfulness (underlying segments tend to be realized similarly on the surface), community (similar segments behave similarly), and context (phonolog­ical rules need access to variables in their context). These biases are so fundamental to generative phonology that they are left implicit in many theories. But explicitly modifying the OSTIA algorithm with these biases allowed it to learn more compact, accurate, and general transducers, and our implementation successfully learns a num­ber of rules from English and German. The algorithm is also successful in learning the composition of multiple rules applied in series. The more difficult problem of decom­posing the learned underlying/surface correspondences into simple, individual rules remains unsolved.

Our transducer induction algorithm is not intended as a cognitive model of hu­man phonological learning. First, for reasons of simplicity, we base our model on simple segmental SPE-style rules; it is not clear what the formal correspondence is of these rules to the more recent theoretical machinery of phonology (e.g., optimality constraints). Second, we assume that a cognitive model of automaton induction would be more stochastic and hence more robust than the OSTIA algorithm underlying our work.

Rather, our model is intended to suggest the kind of biases that may be added to empiricist induction models to build a learning model for phonological rules that is cognitively and computationally plausible. Furthermore, our model is not necessarily nativist; these biases may be innate, but they may also be the product of some other earlier learning algorithm, as the results of Ellison (1992) and Brown et al. (1992) suggest (see Section 5.2). So our results suggest that assuming in the system some very general and fundamental properties of phonological knowledge (whether innate or previously learned) and learning others empirically may provide a basis for future learning models.

Ellison (1994), for example, has shown how to map the optimality constraints of Prince and Smolensky (1993) to finite-state automata; given this result, models of

1 Although our assumption of the simultaneous presentation of surface and underlying forms to the learner may seem at first glance to be unnatural as well, it is quite compatible with certain theories of word-based morphology. For example, in the word-based morphology of Aronoff (1976), word-formation rules apply only to already existing words. Thus the underlying form for any morphological rule must be a word of the language. Even if this word-based morphology assumption holds only for a subset of the language (see e.g., Orgun [1995]) it is not unreasonable to assume that a part of the learning process will involve previously-identified underlying/surface pairs.

Rule-based variation in phonology has traditionally been represented with context-sensitive rewrite rules. For example, in American English an underlying t is realized as a flap (a tap of the tongue on the alveolar ridge) after a stressed vowel and zero or more r's, and before an unstressed vowel. In the rewrite-rule formalism of Chomsky and Halle (1968), this rule would be represented as in (1).

Since Johnson's (1972) work, researchers have proposed a number of different ways to represent such phonological rules by transducers. The most popular method is the two-level formalism of Koskenniemi (1983), based on Johnson (1972) and the (belatedly published) work of Kaplan and Kay (1994), and various implementations and extensions (summarized and contrasted in Karttunen [1993]). The basic intuition of two-level phonology is that a rule that rewrites an underlying string as a surface string can be implemented as a transducer that reads from an underlying tape and writes to a surface tape. Figure 1 shows an example of a transducer that implements the flapping rule in (1). Each arc has an input symbol and an output symbol, separated by a colon. A single symbol (such as t or V) is a shorthand for a symbol that is the same in the input and output (i.e., t:t or V:V). Either the input or the output symbols can be null; a null input symbol is used for an insertion of a phone; a null output symbol for a deletion. A transduction of an input string to an output string corresponds to a path through the transducer, where the input string is formed by concatenating the input symbols of the arcs taken, and the output string by concatenating the output symbols of the arcs. The transducer's input string is the phonologically underlying form, while the transducer's output is the surface form. A transduction is valid if there is a corresponding path beginning in state 0 and ending in an accepting state (indicated by double circles in the figure). Table 1 shows our phone set—an ASCII symbol set based on the ARPA-sponsored ARPAbet alphabet—with the IPA equivalents.

More recently, Bird and Ellison (1994) show that a one-level finite-state automa­ton can model richer phonological structure, such as the multitier representations of autosegmental phonology. In their model, each tier is represented by a finite-state au­tomaton, and autosegmental association by the synchronization of two automata. This synchronized-automata-based rather than transducer-based model generalizes over the two-level models of Koskenniemi (1983) and Karttunen (1993) but also the three-level models of Lakoff (1993), Goldsmith (1993), and Touretzky and Wheeler (1990).

Our phonological-rule induction algorithm is based on augmenting the Onward Subse­quential Transducer Inference Algorithm (OSTIA) of Oncina, Garcia, and Vidal (1993). This section outlines the OSTIA algorithm to provide background for the modifications that follow; see their original paper for further details.

OSTIA takes as input a training set of valid input-output pairs for the transduction to be learned. The algorithm begins by constructing a tree transducer that covers all the training samples according to the following procedure: for each input pair, the algorithm walks from the initial state taking one transition on each input symbol, as if doing a transduction. When there is no move on the next input symbol from the present state, a new branch is grown on the tree. The entire output string of each transduction is initially stored as the output on the last arc of the transduction, that is, the arc corresponding to the end-of-string symbol. An example of an initial tree transducer constructed by this process is shown in Figure 3.

As the next step, the output symbols are "pushed forward" as far as possible towards the root of the tree. This process begins at the leaves of the tree and works its way to the root. At each step, the longest common prefix of the outputs on all the arcs leaving one state is removed from the output strings of all the arcs leaving the state and suffixed to the (single) arc entering the state. This process continues until the longest common prefix of the outputs of all arcs leaving each state is the null string—the definition of an onward transducer. The result of making the transducer of Figure 3 onward is shown in Figure 4.

At this point, the transducer covers all and only the strings of the training set. OSTIA now attempts to generalize the transducer, by merging some of its states to­gether. For each pair of states (s,t) in the transducer, the algorithm will attempt to merge s with t, building a new state with all of the incoming and outgoing transitions of s and t. The result of the first merging operation on the transducer of Figure 4 is shown in Figure 5.

A conflict arises whenever two states are merged that have outgoing arcs with the same input symbol. When this occurs, an attempt is made to merge the destination Figure 4 Onward tree transducer for bat, batter, and band with flapping applied.

Figure 5 Result of merging states 0 and 1 of Figure 4.

Figure 6 Example push-back operation and state merger. Input words and and amp.

states of the two conflicting arcs. First, all output symbols beyond the longest common prefix of the outputs of the two arcs are "pushed back" to arcs further down the tree. This operation is only allowed under certain conditions that guarantee that the trans­ductions accepted by the machine are preserved. The push-back operation allows the two arcs to be combined into one and their destination states to be merged. An exam­ple of a push-back operation and subsequent merger on a transducer for the words and and amp is shown in Figure 6. This method of resolving conflicts repeats until no conflicts remain, or until resolution is impossible. In the latter case, the transducer is restored to its configuration before the merger causing the original conflict, and the algorithm proceeds by attempting to merge the next pair of states.

The OSTIA algorithm can be proven to learn any subsequential relation in the limit. That is, given an infinite sequence of valid input/output pairs, it will at some point derive the target transducer from the samples seen so far. When trying to learn phono­logical rules from finite linguistic data, however, we found that the algorithm was unable to learn a correct, minimal transducer.

We tested the algorithm using a synthetic corpus of 99,279 input/output pairs. Each pair consisted of an underlying pronunciation of an individual word of English and a machine generated "surface pronunciation." The underlying string of each pair was taken from the phoneme-based CMU pronunciation dictionary (CMU 1993). The surface string was generated from each underlying form by mechanically applying the one or more rules we were attempting to induce in each experiment.

In our first experiment, we applied the flapping rule (repeated again in (3)) to training corpora of between 6,250 and 50,000 words. Figure 7 shows the transducer induced from 25,000 training samples, and Table 2 shows some performance results. For obvious reasons we have left off the labels on the arcs in Figure 7. The only differ­ence between underlying and surface forms in both the training and test sets in this experiment is the substitution of dx for a t in words where flapping applies. Therefore, inaccuracies in predicting output strings represent real errors in the transducer, rather than manifestations of other phonological phenomena.

Figure 7 and Table 2 show OSTIA's failure to learn the simple flapping rule. Recall that the optimal transducer, shown in Figure 2, has only 3 states, and would have no error on the test set of synthetic data. OSTIA's induced transducer not only is much more complex (between 19 and 257 states) but has a high percentage of error. In addition, giving the model more training data does not seem to help it induce a smaller or better model; the best transducer was the one with the smallest number of training samples.

Since OSTIA can learn any subsequential relation in the limit, why these difficul­ties with the phonological-rule induction task? The key provision here, of course, is "the limit"; we are clearly not giving OSTIA sufficient training data. There are two reasons this data may not be present in any reasonable training set. First, the neces­sary number of sample transductions may be several times the size of any natural language's vocabulary. Thus even the entire vocabulary of a language may be insuffiFigure 7 First attempt of OSTIA to learn flapping. Transducer induced on 25,000 samples.

Inputs: bat batter band

Figure 8 Final result of merging process on transducer from Figure 4.

cient in size to learn an efficient or correct transducer. Second, even if the vocabulary were larger, the necessary sample may require types of strings that are not found in the language for phonotactic or other reasons. Systematic phonological constraints such as syllable structure may make it impossible to obtain the set of examples that would be necessary for OSTIA to learn the target rule. For example, given one training set of examples of English flapping, the algorithm induced a transducer that realizes an underlying t as dx either in the environment Vr*_V or after a sequence of six consonants. This is possible since such a transducer will accurately cover the training set, as no English words contain six consonants followed by a t. The lack of natural language bias causes the transducer to miss correct generalizations and learn incorrect transductions.

In order to give OSTIA the prior knowledge about phonology to deal with the prob­lems in Section 4, we augmented it with three biases, each of which is assumed explic­itly or implicitly by most if not all theories of phonology. These biases are intended to express universal constraints about the domain of natural language phonology.

Faithfulness: Underlying segments tend to be realized similarly on the surface. Community: Phonologically similar segments behave similarly. Context: Phonological rules need access to variables in their context.

As discussed above, our algorithm is not intended as a direct model of human learning of phonology. Rather, since only by adding these biases was a general-purpose algorithm able to learn phonological rules, and since most theories of phonology as­sume these biases as part of their model, we suggest that these biases may be part of the prior knowledge or state of the learner.

The OSTIA algorithm as described by Oncina, Garcia, and Vidal (1993) had a worst-case complexity of 0(n(m + k) + nmk), where n is the sum of all the input strings' lengths, m is the length of the longest output string, and k is the size of the input alphabet; Oncina, Garcia, and Vidal's (1993) experiments showed the average case time to grow more slowly. We will discuss the complexity implication of each of our enhancements to the algorithm.

The calculation of alignment information adds a preprocessing step to the al­gorithm that requires 0(nm) time for the dynamic programming string-alignment algorithm. After the initial tree is constructed using the alignment information, the above-mentioned worst-case bound still applies for the process of merging states; it does not require that the initial tree be onward. Since this modification only alters the initial tree transducer, the behavior of the main state-merging loop of the OSTIA algo­rithm is essentially unchanged. In practice, we found the use of alignment information significantly sped up the algorithm by allowing states to collapse more quickly. In any case, the 0(nm) complexity of the preprocessing step is subsumed by the 0(nmk) term of OSTIA's complexity.

5 The behavior of the input-based ordering depends on the ordering of the training set. We used a random ordering of our training set, but a corpus-based ordering would not be significantly different. While more frequent words tend to be seen earlier in a corpus, there is no reason to think that more frequent words provide better chances of successful state mergers.

An examination of the final few errors (three samples) in the induced flapping and three-rule transducers in Section 5.2.2 turned out to demonstrate a significant problem in the assumption that an SPE-style rule is isomorphic to a regular relation.

While the learned transducer correctly makes the generalization that flapping oc­curs after any stressed vowel, it does not flap after two stressed vowels in a row:

Recent work in the machine learning of phonology includes algorithms for learning both segmental and nonsegmental information. Nonsegmental approaches include those of Daelemans, Gillis, and Durieux (1994) for learning stress systems, as well as approaches to learning morphology such as Gasser 's (1993) system for inducing Semitic morphology, and Ellison's (1992) extensive work on syllabicity, sonority, and harmony. Since our approach learns only segmental structure, a more relevant com­parison is with other algorithms for inducing segmental structure.

Johnson (1984) gives one of the first computational algorithms for phonological rule induction. His algorithm works for rules of the form where C is the feature matrix of the segments around a. Johnson's algorithm sets up a system of constraint equations that C must satisfy, by considering both the positive contexts, i.e., all the contexts C, in which a b occurs on the surface, as well as all the negative contexts Cj in which an a occurs on the surface. The set of all positive and negative contexts will not generally determine a unique rule, but will determine a set of possible rules. Johnson then proposes that principles from Universal Grammar might be used to choose between candidate rules, although he does not suggest any particular principles.

Johnson's system, while embodying an important insight about the use of positive and negative contexts for learning, did not generalize to insertion and deletion rules, and it is not clear how to extend his system to modern autosegmental phonological systems. Touretzky, Elvgren, and Wheeler (1990) extended Johnson's insight by using the version spaces algorithm of Mitchell (1981) to induce phonological rules in their Many Maps architecture. Like Johnson's, their system looks at the underlying and surface realizations of single segments. For each segment, the system uses the version space algorithm to search for the proper statement of the context. The model also has a separate algorithm that handles harmonic effects by looking for multiple segmental changes in the same word, and has separate processes to deal with epenthesis and deletion rules. Touretzky, Elvgren, and Wheeler's approach seems quite promising; our use of decision trees to generalize each state is a similar use of phonological feature information to form generalizations.

Riley (1991) and Withgott and Chen (1993) first proposed a decision-tree approach to segmental mapping. A decision tree is induced for each segment, classifying pos­sible realizations of the segment in terms of contextual factors such as stress and the surrounding segments. One problem with these particular approaches is that since the decision tree for each segment is learned separately, the technique has difficulty forming generalizations about the behavior of similar segments. In addition, no gener­alizations are made about segments in similar contexts, or about long-distance depen­dencies. In a transducer-based formalism, generalizations about segments in similar contexts follow naturally from generalizations about the behavior of individual seg­ments. The context is represented by the current state of the machine, which in turn depends on the behavior of the machine on the previous segments. A possible adjust­ment to the decision-tree approach to capture some of these generalizations would be to augment the decision tree with information about the features of the output segment, or about features of more distant phones, perhaps about nearby syllables.

Our goal in this paper has been to explore the role of prior knowledge in phonologi­cal learning. We showed that a domain-independent, empiricist induction algorithm, OSTIA, failed to induce minimal transducers even for very simple rules like flapping. But adding three domain-specific learning biases to OSTIA allowed it to successfully learn transducers implementing simple phonological rules of English and German: faithfulness (underlying segments tend to be realized similarly on the surface), commu­nity (similar segments behave similarly), and context (phonological rules need access to variables in their context). These biases are so fundamental to generative phonology that, although they are present in some respect in every phonological theory, they are left implicit in most. Furthermore, we have shown that some of the remaining errors in our augmented model are due to implicit biases in the traditional SPE-style rewrite system that are not similarly represented in the transducer formalism, suggesting that while transducers may be formally equivalent to rewrite rules, they may not have identical evaluation procedures.

Because our biases were applied to the learning of very simple SPE-style rules, and to a nonprobabilistic theory of purely deterministic transducers, we do not expect that our model as implemented has any practical use as a phonological learning device. Indeed, because of the noise and nondeterminism inherent to linguistic data, we feel strongly that stochastic algorithms for language induction are much more likely to be a fruitful research direction (e.g., Kupiec 1992; Lucke 1993; Stolcke and Omohundro 1993, 1994; Ron, Singer, and Tishby 1994). But we believe that the biases we have relied on to improve the OSTIA algorithm may also prove useful when applied to such stochastic linguistic-rule induction algorithms. For example Wooters and Stolcke (1994) used the Stolcke and Omohundro model-merging algorithm to induce word-pronunciation HMMs for a speech recognition system. This algorithm has no domain knowledge about phonology, and so is unable to classify together similar phones, or generalize across phones that were missing in the input data. Adding phonological feature biases to such a model could improve its generalization performance just as it improved OSTIA.

In summary, we believe that augmenting an empirical learning element with rela­tively abstract learning biases is a very fruitful ground for research between the often restated strict nativist and strict empiricist language learning paradigms.

Many thanks to Jerry Feldman for advice and encouragement, to Isabel Galiano-Ronda for her help with the OSTIA algorithm, and to Eric Fosler, Sharon Inkelas, Lauri Karttunen, José Oncina, Orhan Orgun, Ronitt Rubinfeld, Stuart Russell, Andreas Stolcke, Gary Tajchman, four anonymous COLI reviewers, and an anonymous reviewer for ACL-95. This work was partially funded by ICSI.