The syntactic description of a word can be in many forms like part-of-speech tag, phrase type or supertags. In the description hereafter we call any such syntactic information as tag of the word. Now, consider a tagged training corpus of sufficient size in the domain of interest. The first step is to construct a matrix whose rows correspond to word-tag pairs and columns cor­respond to documents in the corpus. A docu­ment can be a sentence, a paragraph or a larger unit of text. If the vocabulary V consists of I words, tagset T consists of J tags and the number of documents in corpus is K, then the matrix will be IJ x K. Let c%_jk denote the frequency of word wi with tag tj in the docu­ment dk. The notation ij (i underscore j) in subscript is used for convenience and indicates word wi with tag tj i.e., (i — 1) J + jth row of the matrix. Then we find entropy e^j of each word-tag pair and scale the corresponding row of the matrix by (1—e^j). The document length normalization to each column of the matrix is also applied by dividing the entries of kth doc­ument by nk, the number of words in document dk. Let cijj be the frequency of ijth word-tag pair in the whole corpus i.e. cijj = K=1 cijjkk. Then, '

Once the matrix X is obtained, we perform its singular value decomposition (SVD) and ap­proximate it by keeping the largest R singular values and setting the rest to zero. Thus, where, U(IJ x R) and V(K x R) are orthonor­mal matrices and S(Rx R) is a diagonal matrix. It is this dimensionality reduction step through SVD that captures major structural associa­tions between words-tags and documents, re­moves 'noisy' observations and allows the same dimensional representation of words-tags and documents (albeit, in different bases).This same dimensional representation is used (eq. 12) to find syntactic-semantic correlation between the present word and the history of words and then to derive the language model probabilities. This R-dimensional space can be called either syntac­tically enhanced latent semantic space or latent syntactic-semantic space.

After the knowledge is represented in the latent syntactic-semantic space, we can project any new document as a R dimensional vector vseisain this space. Let the new document consist of a word sequence wi1, wi2,..., win and let the corresponding tag sequence be tj1 ,tj2,..., tjn, where ip and jp are the indices of the pth word and its tag in the vocabulary V and the tagset T respectively. Let d be the IJ x 1 vector rep­resenting this document whose elements dijj are the frequency counts i.e. number of times word wi occurs with tag pj, weighted by its corre­sponding entropy measure (1 — eijj). It can be thought of as an additional column in the ma­trix X, and therefore can be thought of as hav­ing its corresponding vector v in the matrix V. Then, d = USvT and which is a 1 x R dimensional vector representa­tion of the document in the latent space. Here represents the row vector of the SELSA U matrix corresponding to the word wip and tag tjp in the current document.

We can also define a syntactic-semantic simi­larity measure between any two text documents as the cosine of the angle between their pro­jection vectors in the latent syntactic-semantic space. With this measure we can address the problems that LSA has been applied to, namely natural language understanding, cognitive mod­eling, statistical language modeling etc.

We follow the framework in (Bangalore, 1996) to define a class-based language model where classes are defined by the tags. Here probability of a sequence Wn of n words is given by where ti is a tag variable for the word wi. To compute this probability in realtime based on local information, we make certain assumptions:

where probability of a word is calculated by renormalizing the tri-gram probability to those words which are compatible with the tag in con­text. Similarly, tag probability is modeled using a bi-gram model. Other models like tag based likelihood probability of a word or tag tri-grams can also be used. Similarly there is a motiva­tion for using the syntactically enhanced latent semantic analysis method to derive the word probability given the syntax of tag and seman­tics of word-history.

The calculation of perplexity is based on con­ditional probability of a word given the word history, which can be derived in the following manner using recursive computation.

A further reduction in computation is achieved by restricting the summation over only those tags which the target word can anchor. A similar expression using the tag tri-gram model can be derived which includes double summa­tion. The efficiency of this model depends upon the prediction of tag tq using the word his­tory Wq_l. When the target tag is correctly known, we can derive a performance bench­mark in terms of lower bound on the perplexity achievable. Furthermore, if we assume tagged corpus, then tq s and Tq s become deterministic variables and (5) and (7) can be written as, respectively in which case the next described SELSA language model can be easily applied to calculate the benchmarks.

SELSA model using tag information for each word can also be developed and used along the line of LSA based language model. We can observe in the above framework the need for the probability of the form P(wq^q, Wq_l, Tq_l) which can be evaluated using the SELSA rep­resentation of the word-tag pair corresponding to wq and tq and the history Wq_lTq_l. The former is given by the row uiq_jq of SELSA U matrix and the later can be projected onto the SELSA space as a vector Vq_l using (4). The length of history can be tapered to reduce the effect of far distant words using the exponential forgetting factor 0 < X < 1 as below:

The next step is to calculate the cosine mea­sure reflecting the syntactic-semantic 'closeness' between the word wq and the history Wq l as below:

uiq_jq V Tq_l

II uiq-jqS IUI Vq_lS II

Then SELSA based probability P(sel"l(wq\Wq-\) is calculated by allocating total probability mass in proportion to this closeness measure such that least likely word has a probability of 0 and all probabilities sum to 1:

But this results in a very limited dynamic range for SELSA probabilities which leads to poor performance. This is alleviated by raising the above derived probability to a power y > 1 and then normalizing as follows(Coccaro and Juraf-sky, 199S):

This probability gives more importance to the large span syntactic-semantic dependencies and thus would be higher for those words which are syntactic-semantically regular in the recent his­tory as compared to others. But it will not predict very well certain locally regular words like of, the etc whose main role is to support the syntactic structure in a sentence. On the other hand, n-gram language models are able to model them well because of maximum likeli­hood estimation from training corpus and var­ious smoothing techniques. So the best perfor­mance can be achieved by integrating the two.

One way to derive the 'SELSA + N-gram' joint probability P(sel+na\wq \Wq-l) is to use the ge­ometric mean based integration formula given for LSA in (Coccaro and Jurafsky, 1998) as fol­lows:

where, ^iq =

Supertags are the elementary structures of Lex-icalized Tree Adjoining Grammars (LTAGs) (Bangalore and Joshi, 1999). They are com­bined by the operations of substitution and ad­junction to yield a parse for the sentence. Each supertag is lexicalized i.e. associated with at least one lexical item - the anchor. Further, all the arguments of the anchor of a supertag are localized within the same supertag which al­lows the anchor to impose syntactic and seman­tic (predicate-argument) constraints directly on its arguments. As a result, a word is typically associated with one supertag for each syntac­tic configuration the word may appear in. Su­pertags can be seen as providing a much more refined set of classes than do part-of-speech tags and hence we expect supertag-based language models to be better than part-of-speech based language models.

Words in a sentence are not just strung to­gether as a sequence of parts of speech, but rather they are organized into phrases, group­ing of words that are clumped as a unit. A sentence normally rewrites as a subject noun phrase (NP) and a verb phrase (VP) which are ometric mean weights for SELSA probabilities for the current word wq and any word wi £ V respectively.

If a text corpus is analyzed by counting word frequencies, it is observed that there are cer­tain words which occur with very high frequen­cies e.g. the, and, a, to etc. These words have a very important grammatical behaviour, but they do not convey much of the semantics. Thse words are called function or stop words. Sim­ilarly in a text corpus, there are certain words with frequencies in moderate to low range e.g. car, wheel, road etc. They each play an impor­tant role in deciding the semantics associated with the whole sentence or document. Thus they are known as content words. Generally a list of vocabulary consists of a few hundred function words and a few tens of thousands of content words. However, they span more or less the same frequency space of a corpora. So it is also essential to give them equal importance by treating them separately in a language model­ing framework as they both convey some sort of orthogonal information - syntactic vs seman­tic. LSA is better at predicting topic bearing content words while parsing based models are better for function words. Even n-gram mod­els are quite better at modeling function words, but they lack the large-span semantic that can be achieved by LSA. On the other hand, SELSA model is suitable for both types of words as it captures semantics of a word in a syntactic con­text.

We performed experiments with LSA and SELSA with various levels of syntactic informa­tion in both the situations - content words only vs content and function words together. In the former case, the function words are treated by n-gram model only.

In the first experiment, we performed SELSA using supertag information for each word. The word-supertag vocabulary was about 60000. This resulted in a matrix of about 60000X87000 for which we performed SVD at various dimen­sions. Similarly we trained LSA matrix and per­formed its SVD. Then we used this knowledge to calculate language model probability and then

l Available from the Linguistic Data Consor-tium(LDC) www.ldc.upenn.edu

integrated with tri-gram probability using geo­metric interpolation method (Coccaro and Ju-rafsky, 1998). In the process, we had assumed the knowledge of the content/function word type for the next word being predicted. Fur­thermore, in this experiment, we had used only content words for LSA as well as SELSA repre­sentation, while the function words were treated by tri-gram model only. We also used the su-pertagged test corpus, thus we knew the su­pertag of the next word being predicted. These results thus sets benchmarks for content word based SELSA model. With these assumptions, we obtained the perplexity values as shown in

Table 1.

Table 1: Perplexity at different SVD dimensions with content/function word type knowledge as­sumed. For SELSA, these are benchmarks with correct supertag knowledge.

These benchmark results show that given the knowledge of the content or function word as well as the supertag of the word being predicted, SELSA model performs far better than the LSA model. This improvement in the performance is attributed to the finer level of syntactic infor­mation available now in the form of supertag. Thus given the supertag, the choice of the word becomes very limited and thus perplexity de­creases. The decrease in perplexity across the SVD dimension shows that the SVD also plays an important role and thus for SELSA it is tru-ely a latent syntactic-semantic analysis. Thus if we devise an algorithm to predict the supertag of the next word with a very high accuracy, then there is a gurantee of performance improvement by this model compared to LSA.

Our next experiment, was based on no knowl­edge of content or function word type of the next word. Thus the LSA and SELSA matrices had all the words in the vocabulary. We also kept the SVD dimensions for both SELSA and

LSA to 125. The results are shown in Table 2. In this case, we observe that LSA achieves the perplexity of 88.20 compared to the base­line tri-gram 103.12. However this is more than LSA perplexity of 68.42 when the knowledge of content/function words was assumed. This rel­ative increase is mainly due to poor modeling of function words in the LSA-space. However for SELSA, we can observe that its perplexity of 36.37 is less than 50.39 value in the case of knowledge about content/function words. This is again attributed to better modeling of syntac­tically regular function words in SELSA. This can be better understood from the observation that there were 305 function words compared to 19801 content words in the vocabulary span­ning 19.8 and 20.3 million words respectively in the training corpus. Apart from this, there were 152, 145 and 147 supertags anchoring function word only, content word only and both types of words respectively. Thus given a supertag belonging to function word specific supertags, the 'vocabulary for the target word is reduced by orders of magnitude compared to the case for content word specific supertags. It is also worth observing that the 125-dimensional SVD case of SELSA is better than the 0-dimensional SVD or uniform SELSA case. Thus the SVD plays a role in deciphering the syntactic-semantically important dimensions of the information space.

Table 2: Perplexity without content/function word knowledge. For SELSA, these are bench­marks with correct supertag knowledge.

We also performed experiments using the phrase-type (NP, VP, others) knowledge and incorporated them within SELSA framework. The resultant model was also used to calcu­late perplexity values and the results on con­tent/function type assumption set compares favourably with LSA by improving the perfor­mance. In another experiment we used the part-of-speech tag of the previous word (prevtag) within SELSA, but it couldn t improve against the plain LSA. These results shows that phrase level information is somewhat useful if it can be predicted correctly, but previous POS tags are not useful.

Table 3: Perplexity of phrase/prevtag based SELSA with the knowledge of content/function word type and the correct phrase/prevtag

Finally the utility of this language model can be tested in a speech recognition experiment. Here it can be most suitably applied in a second-pass rescoring framework where the output of first-pass could be the N-best list of either joint word-tag sequences (Wang and Harper, 2002) or word sequences which are then passed through a syntax tagger. Both these approaches allow a direct application of the results shown in above experiments, however there is a possibility of error propagation if some word is incorrectly tagged. The other approach is to predict the tag left-to-right from the word-tag partial prefix followed by word prediction and then repeating the procedure for the next word.