Language modeling is an important aspect to consider in large-vocabulary speech recognition systems (Bahl et al., 1983; Jelinek, 1998). The n-gram models are the most widely-used for a wide range of domains (Bahl et al., 1983). The n-grams are simple and robust models and ad­equately capture the local restrictions between words. Moreover, it is well-known how to es­timate the parameters of the model and how to integrate them in a speech recognition sys­tem. However, the n-gram models cannot ad­equately characterize the long-term constraints of the sentences of the tasks.

On the other hand, Stochastic Context-Free Grammars (SCFGs) allow us a better model-

* This work has been partially supported by the Spanish CICYT under contract (TIC98/0423-C06).

ing of long-term relations and work well on limited-domain tasks of low perplexity. How­ever, SCFGs work poorly for large-vocabulary, general-purpose tasks because learning SCFGs and the computation of word transition proba­bilities present serious problems for complex real tasks.

In the literature, a number of works have pro­posed ways to generalize the n-gram models (Je­linek, 1998; Siu and Ostendorf, 2000) or com­bining with other structural models (Bellegarda, 1998; Gilet and Ward, 1998; Chelba and Jelinek, 1998) .

In this paper, we present a combined lan­guage model defined ctS et linear combination of n-grams, which are used to capture the local re­lations between words, and a stochastic gram­matical model which is used to represent the global relation between syntactic structures. In order to capture these long-term relations and to solve the main problems derived from the large-vocabulary complex tasks, we propose here to define: a category-based SCFG and a probabilis­tic model of word distribution in the categories. Taking into account this proposal, we also de­scribe here how to solve the learning of these stochastic models and their integration prob­lems.

With regard to the learning problem, several algorithms that learn SCFGs by means of es­timation algorithms have been proposed (Lari and Young, 1990; Pereira and Schabes, 1992; Sanchez and Benedf, 1998), and promising re­sults have been achieved with category-based SCFGs on real tasks (Sanchez and Benedf, 1999) .

In relation to the integration problem, we present two algorithms that compute the word transition probability: the first algorithm is based on the Left-to-Right Inside algorithm (LRI) (Jelinek and Lafferty, 1991), and the sec­ond is based on an application of a Viterbi scheme to the LRI algorithm (the VLRI algo­rithm) (Sanchez and Benedf, 1997).

Finally, in order to evaluate the behavior of this proposal, experiments with a part of the Wall Street Journal processed in the Penn Tree-bank project were carried out and significant im­provements with regard to the classical n-gram models were achieved.

An important problem related to language mod­eling is the evaluation of Pr(wk | w\.. .Wk-i). In order to compute this probability, we pro­pose a hybrid language model defined as a sim­ple linear combination of n-gram models and a stochastic grammatical model Gs:

where 0 < a < 1 is a weight factor which de­pends on the task.

The expression PT(wk\wk-n ■ ■ ■ Wk-i) '1S the word probability of occurrence of Wk given by the n-gram model. The parameters of this model can be easily estimated, and the ex­pression PT(wk\wk-n ■ ■ ■ Wk-i) can be efficiently computed (Bahl et al., 1983; Jelinek, 1998).

In order to define the stochastic gram­matical model Gs of the expression Pr(wk\wi... Wk-i, Gs) for large-vocabulary complex tasks, we propose a combination of two different stochastic models: a category-based SCFG (Gc), that allows us to represent the long-term relations between these syntacti­cal structures and a probabilistic model of word distribution into categories (Cw).

This proposal introduces two important as­pects, which are the estimation of the parame­ters of the stochastic models, Gc and Cw, and the computation of the following expression:

Pi{wk\wi.. .wk-i,Gc,Cw) _ Pt(wi .. .wk .. .\GC,CW) . .

The parameters of the described model are es­timated from a training sample, that is, from a set of sentences. Each word of the sentence has a part-of-speech tag (POStag) associated to it. These POStags are considered as word categories and are the terminal symbols of the SCFG. From this training sample, the parame­ters of Gc and Cw can be estimated as follows.

First, the parameters of Cw, represented by Pr(M?|c), are computed as:

where N(m?,c) is the number of times that the word w has been labeled with the POStag c. It is important to note that a word w can belong to different categories. In addition, it may happen that a word in a test set does not appear in the training set, and therefore some smoothing technique has to be carried out.

With regard to the estimation of the category-based SCFGs, one of the most widely-known methods is the Inside-Outside (10) algo­rithm (Lari and Young, 1990). The application of this algorithm presents important problems which are accentuated in real tasks: the time complexity per iteration and the large number of iterations that are necessary to converge. An alternative to the 10 algorithm is an algorithm based on the Viterbi score (VS algorithm) (Ney, 1992). The convergence of the VS algorithm is faster than the 10 algorithm. However, the SCFGs obtained are, in general, not as well learned (Sanchez et al., 1996).

Another possibility for estimating SCFGs, which is somewhere between the 10 and VS al­gorithms, has recently been proposed. This ap­proach considers only a certain subset of deriva­tions in the estimation process. In order to select this subset of derivations, two alterna­tives have been considered: from structural in­formation content in a bracketed corpus (Pereira and Schabes, 1992; Amaya et al., 1999), and from statistical information content in the le­best derivations (Sanchez and Benedf, 1998). In the first alternative, the IOb and VSb algo­rithms which learn SCFGs from partially brack­eted corpora were defined (Pereira and Schabes, 1992; Amaya et al., 1999). In the second alter­native, the fcVS algorithm for the estimation of the probability distributions of a SCFG from the fc-best derivations was proposed (Sanchez and Benedf, 1998).

From expression (2), it can bee seen that in or­der to integrate the model, it is necessary to efficiently compute the expression:

In order to describe how this computation can be made, we first introduce some notation.

A Context-Free Grammar G is a four-tuple (N, S, P, S), where N is the finite set of nonter­minals, S is the finite set of terminals (N D S = 0), S G N is the axiom or initial symbol and P is the finite set of productions or rules of the form A -> a, where A G N and a €E (N U £)+ (only grammars with non empty rules are con­sidered). For simplicity (but without loss of gen­erality) only context-free grammars in Chomsky Normal Form are considered, that is, grammars with rules of the form A BC or A ^ v where A,B,C <E N and »eE.

A Stochastic Context-Free Grammar Gs is a pair (G,p), where G is a context-free gram­mar and p : P —»]0,1] is a probability func­tion of rule application such that VA €E N:

Eae(jvus)+P(^ -> a) = 1.

Now, we present two algorithms in order to compute the word transition probability. The first algorithm is based on the LRI algorithm, and the second is based on an application of a Viterbi scheme to the LRI algorithm (the VLRI algorithm).

Probability of generating an initial substring

The computation of (4) is based on an algo­rithm which is a modification of the LRI algo­rithm (Jelinek and Lafferty, 1991). This new algorithm is based on the definition of Pt(A << i,j)) = Pt(A w{... Wj ... |GC, Cw) as the probability that A generates the initial substring Wi... Wj ... given Gc and Cw. This can be com­puted with the following dynamic programming

Pr(A«iJ)= Y^A^BC) B,C£N l=i Pt{B < i,l >)Pr(G «1 + 1, j) .

In this way, Pt(w\ ... wk ■ ■ ■ |GC, Cw) = Pr(5« l,fcj.

In this expression, Q(A => D) is the proba­bility that D is the leftmost nonterminal in all sentential forms which are derived from A. The value Q(A => BC) is the probability that BC is the initial substring of all sentential forms de­rived from A. Pt(B < i,l >) is the probability that the substring W{ ...wi is generated from B given Gc and Cw. Its computation will be de­fined later.

It should be noted that the combination of the models Gc and Cw is carried out in the value Pr(A « i,i). This is the main difference with respect the LRI algorithm.

Probability of the best derivation generating an initial substring

An algorithm which is similar to the previous one can be defined based on the Viterbi scheme. In this way, it is possible to obtain the best pars­ing of an initial substring. This new algorithm is also related to the VLRI algorithm (Sanchez and Benedf, 1997) and is based on the definition of P~i(A « i,j)) = P~i(A 4> Wi...Wj ...\GC,CW) as the probability of the most probable parsing which generates W{...Wj ... from A given Gcand Cw. This can be computed as follows:

P~i(B < i,l >)fr(C « I + 1, j)) .

Therefore Pr(«?i ... wk ... |GC, Cw) = Pr(S « l,fc).

In this expression, Q(A => D) is the prob­ability that D is the leftmost nonterminal in the most probable sentential form which is de­rived from A. The value Q(A => BC) is the probability that BC is the initial substring of most the probable sentential form derived from A. Pt(B < i,l >) is the probability of the most probable parse which generates W{...wi from B.

Probability of generating a string

The value Pi(A < i,j >) = Pr(A 4> W{... Wj\Gc, Cw) is defined as the probability that the substring W{... Wj is generated from A given Gc and Cw. To calculate this proba­bility a modification of the well-known Inside algorithm (Lari and Young, 1990) is proposed. This computation is carried out by using the following dynamic programming scheme:

In this way, Pt(w\ ... wn\Gc, Cw) = Pr(S < l,n >).

As we have commented above, the combina­tion of the two parts of the grammatical model is carried out in the value Pt(A <i,i >).

Probability of the best derivation generating a string

The probability of the best derivation that generates a string, Pt(w\ ... wn\Gc, Cw), can be evaluated using a Viterbi-like scheme (Ney, 1992). As in the previous case, the computation of this probability is based on the definition of Pt(A < i,j >) = Pt(A =4> wi... Wj\Gc, Cw) as the probability of the best derivation that gen­erates the substring Wi... Wj from A given Gcand Cw. Similarly:

P~i(B < i,l >)P~i(C < I + 1, j >) .

Therefore, Pr(«?i... wn\Gc, Cw) = Pt(S < l,n >).

Finally, the time complexity of these algo­rithms is the same as the algorithms they are related to, therefore the time complexity is 0(fc|P|), where k is the length of the input string and \P\ is the size of the SCFG.

The corpus used in the experiments was the part of the Wall Street Journal which had been pro­cessed in the Penn Treebank project (Marcus et al., 1993). This corpus consists of English texts collected from the Wall Street Journal from edi­tions of the late eighties. It contains approx­imately one million words. This corpus was automatically labelled, analyzed and manually checked as described in (Marcus et al., 1993). There are two kinds of labelling: a POStag la­belling and a syntactic labelling. The size of the vocabulary is greater than 25,000 different words, the POStag vocabulary is composed of 45 labels and the syntactic vocabulary is com­posed of 14 labels.

The corpus was divided into sentences accord­ing to the bracketing. In this way, we obtained a corpus whose main characteristics are shown in Table 1.

We took advantage of the category-based SCFGs estimated in a previous work (Sanchez and Benedf, 1998). These SCFGs were esti­mated with sentences which had less than 15 words. Therefore, in this work, we assumed such restriction. The vocabulary size of the new cor­pus was 6,333 different words. For the exper­iments, the corpus was divided into a training corpus (directories 00 to 19) and a test corpus (directories 20 to 24). The characteristics of these sets can be seen in Table 2. The part of the corpus labeled with POStags was used to esti­mate the parameters of the grammatical model, while the non-labeled part was used to estimate the parameters of the n-gram model. We now describe the estimation process in detail.

Table 2: Characteristics of the data sets defined for the experiments when the sentences with more than 15 POStags were removed.

The parameters of a 3-gram model were es­timated with the software tool described in (Rosenfeld, 1995) . We used the linear interpo­lation smooth technique supported by this tool. The out-of-vocabulary words were grouped in the same class and were used in the computa­tion of the perplexity. The test set perplexity with this model was 180.4.

The values of expression (3) were computed from the tagged and non-tagged part of the training corpus. In order to avoid null val­ues, the unseen events were labeled with a spe­cial symbol w' which did not appear in the vocabulary, in such a way that Pt(w'\c) ^ 0, Vc G C, where C was the set of categories. That is, all the categories could generate the un­seen event. This probability took a very small value (several orders of magnitude less than minwev;cec Pr(tu|c), where V was the vocabu­lary of the training corpus), and different values of this probability did not change the results.

The parameters of an initial ergodic SCFG were estimated with each one of the estimation methods mentioned in Section 3. This SCFG had 3,374 rules, composed from 45 terminal symbols (the number of POStags) and 14 non­terminal symbols (the number of syntactic la­bels). The probabilities were randomly gener­ated and three different seeds were tested, but only one of them is reported given that the re­sults were very similar. The training corpus was the labeled part of the described corpus. The perplexity of the labeled part of the test set for different estimation algorithms can be seen in Table 3.

Table 3: Perplexity of the labeled part of the test set with the SCFG estimated with the methods mentioned in Section 3.

Once we had estimated the parameters of the defined model, we applied expression (1) by us­ing the LRI algorithm and the VLRI algorithm in expression (4). The test set perplexity that was obtained in function of a for different esti­mation algorithms (VS, fcVS, IOb and VSb) can be seen in Fig. 1.

In the best case, the proposed language model obtained more than a 30% improvement over results obtained by the 3-gram language model (see Table 4). This result was obtained when the SCFG estimated with the IOb algorithm was used. The SCFGs estimated with other al­gorithms also obtained important improvements compared to the 3-gram. In addition, it can be observed that both the LRI algorithm and the VLRI algorithm obtained good results.

Table 4: Best test perplexity for different SCFG estimation algorithms, and the percentage of im­provement with respect to the 3-gram model.

An important aspect to note is that the weight of the grammatical part was approxi­mately 50%, which means that this part pro­vided important information to the language model.

A new language model has been introduced. This new language model is defined ctS et lin­ear combination of an n-gram which repre­sents relations between words, and a stochastic

Figure 1: Test set perplexity obtained with the proposed language models in function of gamma. Different curves correspond to SCFGs estimated with different algorithms. The up­per graphic corresponds to the results obtained when the LRI algorithm was used in the lan­guage models, and the lower graphic corre­sponds to the results obtained with the VLRI algorithm.

grammatical model which is used to represent the global relation between syntactic structures. The stochastic grammatical model is composed of a category-based SCFG and a probabilistic model of word distribution in the categories. Several algorithms have been described to esti­mate the parameters of the model from a the sample. In addition, efficient algorithms for solving the problem of the interpretation with this model have been presented.

The proposed model has been tested on the part of Wall Street Journal processed in the Penn Treebank project, and the results obtained improved by more than 30% the test set perplexity over results obtained by a simple 3-gram model.