Supertagging involves assigning words lexical en­tries based on a lexicalized grammatical theory, such as Combinatory Categorial Grammar (CCG) (Steedman, 2000) Tree-adjoining Grammar (Joshi, 1988), or Head-driven Phrase Structure Grammar (Pollard and Sag, 1994). Supertag sets are larger than part-of-speech (POS) tag sets and their ele­ments are generally far more articulated. For ex­ample, the English verb join has the POS VB and the CCG category ((Sb\NP)/PP)/NP in CCG-bank (Hockenmaier and Steedman, 2007). This category indicates that join requires a noun phrase

© 2008. Licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported li­cense (http://creativecommons.Org/licenses/by-nc-sa/3.0/). Some rights reserved.

to its left, another to its right, and a prepositional phrase to the right of that.

Supertags convey such detailed syntactic sub-categorization information that supertag disam­biguation is referred to as almost parsing (Banga­lore and Joshi, 1999). Standard sequence predic­tion models are highly effective for supertagging, including Hidden Markov Models (Bangalore and Joshi, 1999; Nielsen, 2002), Maximum Entropy Markov Models (Clark, 2002; Hockenmaier et al., 2004; Clark and Curran, 2007), and Conditional Random Fields (Blunsom and Baldwin, 2006). The original motivation for supertags-parse pre-filtering for lexicalized grammars-of Bangalore and Joshi (1999) has been realized to good effect: the supertagger of Clark and Curran (2007) pro­vides staged n-best lists of multi-tags that dramat­ically improve parsing speed and coverage without much loss in accuracy. Espinosa et al. (2008) have shown that hypertagging (predicting the supertag associated with a logical form) can improve both speed and accuracy of wide-coverage sentence re­alization with CCG. Supertags have gained fur­ther relevance as they are increasingly used as fea­tures for other tasks, including machine translation (Birch et al., 2007; Hassan et al., 2007).

Supertaggers typically rely on a significant amount of carefully annotated sentences. As with many problems, there is pressing need to find strategies for reducing the amount of supervision required for producing accurate supertaggers, but as yet, no one has explored the use of weak super­vision for the task. In particular, there are many di­alog systems which rely on hand-crafted lexicons that both provide a starting point for bootstrapping a supertagger and which could benefit greatly from supertag pre-parse filter. For example, the dialog system used by Kruijff et al. (2007) uses a handcrafted CCG grammar for OpenCCG (White and Baldridge, 2003). It is important to stress that there are many such uses of CCG and related frame­works which do not rely on first annotating (even a small number of) sentences in a corpus: these de­fine a lexicon that maps from words to categories (supertags) for a particular domain/application.

This scenario is a natural fit for learning taggers from tag dictionaries using hidden Markov mod­els with Expectation-Maximization (EM). Here, 1 investigate such weakly supervised learning for supertagging and demonstrate the importance of proper initialization of the tag transition distribu­tions of the HMM. In particular, such initializa­tion can be done using inherent properties of the CCG formalism itself regarding how categoriesmay combine. Informed initialization should help with supertagging for two reasons. First, cate­gories have structure-lacking in POS tags-waiting to be exploited. For example, it is far more likely a priori to see the category sequence (S\NP)/NP NP/N than the sequence S/S NP\NP. Given the categories for a word, this information can be used to influence our expectations about categories for adjacent words. Second, this kind of information truly matters for the task: a key aspect of supertag­ging that differentiates it from POS tagging is that the contextual information is much more important for the former. Lexical probabilities handle most of the ambiguity for POS tagging, but supertags are inherently about context and, furthermore, lex­ical ambiguity is much greater for supertagging, making lexical probabilities less effective.

I start by defining a distribution over lexical categories and then use this distribution as part of creating a CCG-informed transition distribution that appropriately breaks the symmetry of uniform HMM initialization. After describing how these components are included in the HMM, I describe experiments with CCGbank varying the ambiguity of the lexicon provided. I show that using knowl­edge about the formalism consistently improves performance, and is especially important as cate-gorial ambiguity increases.

The categories of CCG are an inductively defined set containing elements that are either atomic ele­ments or (curried) functions specifying the canon-

!For the rest of the paper, I will refer to categories rather supertags, but will still refer to the task as supertagging.

ical linear direction in which they seek their argu­ments. Some example entries from CCGbank are:

the := NPnb/N

buy := ((((sb\np)/pp)/pp)/(sadj\np))/np Words can be associated with multiple categories; the distribution over these categories is typically quite skewed. For example, the first entry for buy occurs 33 times in CCGbank, compared with just once for the second. That the simpler category is more prevalent is unsurprising: a general strategy when creating CCG lexicons is to use simpler cate­gories whenever possible. This points to the possi­bility of defining distributions over CCG lexicons based on measures of the complexity of categories. I use a simple distribution here: given a lexicon C, the probability of a category i is inversely propor­tional to its complexity:

. complexity(ci) ^—lj(zC complexity(cj)

Here, a very simple complexity measure is assumed: the number of subcategories (to­kens) contained in a category. For example, ((S\NP)\(S\NP)/NP contains 9: S (twice), NP (thrice), S\NP (twice), (S\NP)\(S\NP), and ((S\NP)\(S\NP)/NP.

The tag transition distribution defined in the next section uses A to bias transitions toward simpler categories, e.g., preferring the first category for buy over the second. Performance when using A is compared to using a uniform distribution.

Other distributions could be given, e.g., one which gives more mass to adjunct categories such as (S\NP)\(S\NP) than to ones which are oth­erwise similar but do not display such symmetry, like (S/NP)\(NP\S). However, the most impor­tant thing for present purposes is that simpler cate­gories are more likely than more complex ones.

This distribution imposes no internal struc­ture on the likelihood of a lexicon. As far as A is concerned, lexicons can as well have the category (S\NP)\NP for transitive verbs and ((S/NP)/NP)/NP for ditransitive verbs, even though this is a highly unlikely pattern since we would expect both types of verbs to seek their ar­guments in the same direction. Languages also tend to prefer lexicons with one or the other slash direction predominating (Villavicencio, 2002). In the future, it would be interesting to consider Bayesian approaches that could encode more com­plex structure and assign priors over distributions over lexicons, building on these observations.

An aspect of CCGbank that relevant for Aj is that some categories actually are not true cate­gories. For example, many punctuation "cate­gories" are given as LRB, ., :, etc. In most grammars, the category of '.' is usually assumed to be S\S. The grammatical behavior of such pseudo-categories is handled via special rules in the parsers of Hockenmaier and Steedman (2007) and Clark and Curran (2007). I relabeled three of these: , to NP\NP, . to S\S and ; to (S\S)/S. A single best change was not clear for others such as LRB and :, so they were left as is.

CCG analyses of sentences are built up from lex­ical categories combining to form derived cate­gories, until an entire sentence is reduced to a sin­gle derived category with corresponding depen­dencies. One of CCG's most interesting linguis­tic properties is it allows alternative constituents. Consider the derivations in Figures 1 and 2, which show a normal form derivation (Eisner, 1996) and fully incremental derivation, respectively. Both produce the same dependencies, guaranteed by the semantic consistency of CCG's rules (Steedman, 2000). This property of CCG of supporting mul­tiple derivations of the same analysis has been termed spurious ambiguity. However, the extra constituents are anything but spurious: they are implicated in a range of CCG (along with other forms of categorial grammar) linguistic analyses, including coordination, long-distance extraction, intonation, and incremental processing.

This all boils down to associativity: just as (1 + (4 + 2)) = ((1 + 4) + 2) = 7, CCG ensures that (Ed-(saw-Ted)) = {{Ed-saw)-Ted) = S Such multiple derivations arise when adjacent categories can combine through either application or compo­sition. Thus, we would expect that the lexical cat­egories needed to analyze an entire sentence will more often than not be able to combine with their immediate neighbors. For example, six of seven pairs of adjacent lexical categories in the sentence in Figure 1 can combine. Only N PP/NP of board as cannot.

This observation can be used in different ways by different models for CCG supertagging. For example, discriminative tagging models could in­clude features that capture whether or not the cur­rent supertag can combine with the previous one and possibly via which CCG rule. Here, I show how it can be used to provide a non-uniform start­ing point for the transition distributions 9-^ in a first-order Hidden Markov Model. This is similar to how Grenager et al. (2005) use diagonal initial­ization in an HMM for field segmentation to en­courage the model to remain in the same state (and thus predict the same label for adjacent words). For CCG supertagging, the initialization should discourage diagonalization and establish a prefer­ence for some transitions over others.

There are many ways to define such a starting point. The simplest would be to reserve a small part of the mass spread uniformly over category pairs which cannot combine and then spread the rest of the mass uniformly over those which can. However, we can provide a more refined distri­bution, by incorporating the lexical category distribution Aj defined in the previous section to weight these transitions according to this further information. In a similar manner to Grenager et al. (2005), I define * as follows:

A first-order Hidden Markov Model (bitag HMM) is used for bootstrapping a supertagger from a lex­icon. See Rabiner (1989) for an extensive intro­duction to and discussion of HMMs. There are several reasons why this is an attractive tagging model here. First, though extra context in the form of tritag transition distributions or other tech­niques can improve supervised POS tagging accu­racy, the accuracy of bitag HMMs is not far behind. The goal here is to investigate the relative gains of using CCG-based information in weakly super­vised HMM learning. Second, the expectation-maximization algorithm for bitag HMMs is effi­cient and has been shown to be quite effective for acquiring accurate POS taggers given only a lex­icon (tag dictionary) and certain favorable condi­tions (Banko and Moore, 2004). Third, the model's simplicity makes it straightforward to test the idea of CCG-initialization on tag transitions.

Dirichlet priors can be used to bias HMMs to­ward more skewed distributions (Goldwater and Griffiths, 2007; Johnson, 2007), which is espe­cially useful in the weakly supervised setting con­sidered here. Following Johnson (2007), I use vari­ational Bayes EM (Beal, 2003) during the M-step for the transition distribution:

The idea of using knowledge from the formalism for constraining supertagging originates with Bangalore and Joshi (1999). They used constraints based on how elementary trees of Tree-Adjoining Grammar could or could not combine as filters to block out tags that do not fit in certain locations in the string. My approach is different is sev­eral ways. First, they dealt with fully supervised supertagging; here I show that using this knowl­edge is important for weakly supervised supertag­ging where we are given only a tag dictionary (lex­icon). Second, my approach encodes grammar-based cues only as an initial bias, so categories are never explicitly filtered. Finally, I use CCG rather than TAG, which makes it possible to exploit a much higher degree of associativity in derivations. This in turn makes it easier to utilize prior knowl­edge about adjacent contexts - precisely what is needed for using the grammar to influence the tran­sition probabilities of a bigram HMM.

On the other hand, Bangalore and Joshi (1999) use constraints that act at greater distances than I have considered here. For example, if one wishes to provide a word with the category ((S\NP)/PP)/NP, then there should be a word with a category which results in a PP two or more words to its right - this is something which the bigram transitions considered here cannot capture. An interesting way to extend the present approach would be to enforce such patterns as posterior con­straints during EM (Graca et al., 2007).

Recent work considers a damaged tag dictionary by assuming that tags are known only for words that occur more than once or twice (Toutanova and Johnson, 2007). A very interesting aspect of this work is that they explicitly model ambiguity classes to exploit commonality in the lexicon be­tween different word forms, which could be even more useful for supertagging.

In a grammar development context, it is often the case that only some of the categories for a word have been assigned. This is the scenario consid­ered by Haghighi and Klein (2006) for POS tag­ging: how to construct an accurate tagger given a set of tags and a few example words for each of those tags. They use distributional similarity of words to define features for tagging that effectively allow such prototype words to stand in for others. This idea could be used with my approach as well; the most obvious way would be to use prototype words to suggest extra categories (beyond the tag dictionary) for known words and a reduced set of categories for unknown words.

Other work aims to do truly unsupervised learn­ing of taggers, such as Goldwater and Griffiths (2007) and Johnson (2007). No tag dictionaries are assumed, and the models are parametrized with Dirichlet priors. The states of these models implic­itly represent tags; however, it actually is not clear what the states in such models truly represent: they are (probably interesting) clusters that may or may not correspond to what we normally think of as parts-of-speech. POS tags are relatively inert, pas­sive elements in a grammar, whereas CCG cate­gories are the very drivers of grammatical analy­sis. That is, syntax is projected, quite locally, by lexical categories. It would thus be interesting to consider the induction of categories with grammar-based priors with such models.

I have shown that weakly supervised learning can indeed be used to induce supertaggers from a lex­icon mapping words to their possible categories, but that the extra ambiguity in the supertagging task over that of POS tagging makes performance much more sensitive to rare categories that occur in larger, more ambiguous lexicons. However, I have also shown that the CCG formalism itself can provide the basis for useful distributions over lexi­cal categories and tag transitions in a bitag HMM. By using these distributions to initialize the HMM, it is possible to improve performance regardless of the underlying ambiguity. This is especially im­portant for reducing error when the lexicon used for bootstrapping is highly ambiguous and con­tains very rare categories.

Thanks to the UT Austin Natural Language Learn­ing group and three anonymous reviewers for use­ful comments and feedback. This work was sup­ported by NSF grant BCS-0651988 and a Faculty Research Assignment from UT Austin.