The capacity to rapidly connect language to referential meaning is an essential aspect of communication between humans. Eye-tracking studies show that humans listening to spoken directives are able to actively attend to the entities that the words in these directives might refer to, even while the words are still being pronounced (Tanenhaus et al. 1995; Brown-Schmidt, Campana, and Tanenhaus 2002). This timely access to referential information about input utterances may allow listeners to adjust their pref­erences among likely interpretations of noisy or ambiguous utterances to favor those that make sense in the current environment or discourse context, before any lower-level disambiguation decisions have been made. This same capability in a spoken language interface system could allow reliable human-machine interaction in the idiosyncratic language of day-to-day life, populated with proper names of co-workers, objects, and events not found in broad training corpora. When domain-specific training corpora are

* Department of Computer Science and Engineering, 200 Union St. SE, Minneapolis, MN 55455. E-mail: schuler@cs.umn.edu; swu@cs.umn.edu; lane@cs.umn.edu.

Submission received: 25 April 2007; revised submission received: 4 March 2008; accepted for publication: 2 June 2008.

Early approaches to incremental interpretation (Mellish 1985; Haddock 1989) apply semantic constraints associated with each word in a sentence to progressively winnow the set of individuals that could serve as referents in that sentence. These incrementally constrained referents are then used to guide the syntactic analysis of the sentence, dis-preferring analyses with empty interpretations in the current environment or discourse context. Similar approaches were applied to broad-coverage text processing, querying a large commonsense knowledge base as a world model (Martin and Riesbeck 1986). But this winnowing is done deterministically, invoking default assumptions and potentially exponential backtracking when default assumptions fail.

The idea of basing analysis decisions on constrained sets of referent individuals was later extended to pursue multiple interpretations at once by exploiting polynomial structure-sharing in a dynamic programming parser (Schuler 2001; DeVault and Stone 2003; Gorniakand Roy 2004; Aist et al. 2007). The resulting shared interpretation is similar to underspecified semantic representations (Bos 1996), except that the rep­resentation mainly preserves syntactic ambiguity rather than semantic (e.g., quantifier scoping) ambiguity, and the size complexity of the parser chart representation is polynomially bounded. This approach was further extended to support hypothetical referents (DeVault and Stone 2003), domains with continuous relations (Gorniakand Roy 2004), and updates to the shared parser chart by components handling other levels of linguistic analysis in parallel, during real-time recognition (Aist et al. 2007).

The advantage of this use of the parser chart is that it allows a straightforward mapping between syntax and semantics using familiar compositional semantic rep­resentations. But the standard dynamic programming algorithm for parsing derives its complexity bounds from the fact that each recognized constituent can be analyzed independently of every other constituent. These independence assumptions must be relaxed if dynamic context dependencies are to be applied across sibling constituents (e.g., in the package data directory, open where the files to be opened should be restricted to the contents of the package data directory). More importantly, from an engineering perspective, the dynamic programming algorithm for parsing runs in cubic time, not linear, which means this interpretation frameworkcannot be directly applied to continuous audio streams. Interface systems therefore typically perform utterance or sentence segmentation as a stand-alone pre-process, without integrating syntactic or referential semantic dependencies into this decision.

Finally, some speech recognition systems employ inter-utterance context-dependent language models that are pre-compiled into word n-grams for particular discourse or environment states, and swapped out between utterances (Young et al. 1989; Lemon and Gruenstein 2004; Seneff et al. 2004). But in some cases accurate interpretation will require spoken language interfaces to exploit context continuously during utterance recognition, not just between utterances. For example, the probability distribution over the next word in the utterance go to the package data directory and get the . . . (or in the package data directory get the . . . ) will depend crucially on the linguistic and environment context leading up to this point: the meaning of package data directory in the first part of this directive, as well as the objects that will be available once this part of the directive has been carried out. Moreover, in rich environments pre-compilation to word n-grams can be expensive, since all referents in the world model must be considered to build accurate n-grams. This will not be practical if environments change frequently.

In contrast to the approaches described in Section 2, this article proposes an incremental interpretation frameworkwhich is entirely contained within a single-pass probabilistic decoding search. Essentially, this approach directly integrates model theoretic seman­tics, summarized in Section 3.1, with conventional probabilistic time-series models used in speech recognition, summarized in Section 3.2.

Semantic interpretation requires a frameworkwithin which a speaker's intended mean­ings can be formalized. Sections 3.1.1 and 3.1.2 describe a model theoretic approach to semantic interpretation that will later be extended in Section 4.1. The referential states defined here will then be incorporated into a representation of nested syntactic constituents in a hierarchic time-series model in Section 4.2. Some of the notation introduced here is summarized later in Table 1 (Section 4).

A subsumption lattice (laid on its side) over the power set of a domain containing three individuals: i1, L2,and l3. Subsumption relations are represented as gray arrows from supersets (or super-concepts) to subsets (or sub-concepts).

3.1.1 Model Theory. The language model described in this article defines semantic ref­erents in terms of a world model M. In model theory (Tarski 1933; Church 1940), a world model is defined as a tuple M = (E, 0) containing a domain of individuals E = {ii,l2,...} and an interpretation function [•] to interpret expressions in terms of those individuals. This interpretation function accepts expressions tfy of various types: logical statements, of simple type T (for example, the demo file is writable) which may be true or false; references to individuals, of simple type E (for example, the demo file) which may refer to any individual in the world model; or functors of complex type (a, p), which take an argument of type a and produce output of type p. Functor expressions § of type (a, p) can be applied to other expressions il> of type a as arguments to yield expressions 43(il>)oftype p (for example, writable may take the demo file as an argument and return true). By nesting functors, complex expressions can be defined, denoting sets or properties of individuals: (E, T) (for example, writable), relations over individual pairs: (E, (E,T)) (for example, contains), or first-order functors over sets: ((E,T), (E,T)) (for example, a comparative adjective like larger).

3.1.2 Ontological Promiscuity. First-order or higher models (in which functors can take sets as arguments) can be mapped to equivalent zero-order models (with functors defined only on entities). This is generally motivated by a desire to allow sets of individuals to be described in much the same way as individuals themselves (Hobbs 1985). Entities in a zero-order model M can be defined from individuals in a higher-order model M* by mapping or reifying each set S = i2, ... } in V(Em*) (or each set of sets in V(V(Em*)), etc.) as an entity eS in a new domain EM. Relations l inter­preted as zero-order functors in M can be defined directly from relations l* interpreted as higher-order functors (over sets) in M* by mapping each instance of (S1,S2) in ¥*]m* : V(Em* ) x V(Em* ) to a corresponding instance of (eS1, eS2) in 11]m : Em x EM.Set subsumption in M* can then be defined on entities made from reified sets in M, similar to 'ISA' relations over concepts in knowledge representation systems (Brachman and Schmolze 1985).

These subset or subsumption relations can be represented in a subsumption lattice, as shown in Figure 1, with supersets to the left connecting to subsets to the right. This representation will be used in Section 4 to define weighted transitions over first-order referents in a statistical time-series model of interpretation.

1 Here, V (X)isthe powerset of X, containing the set of all subsets.

The referential semantic language model described in this article is based on Hierar­chic Hidden Markov Models (HHMMs), an existing extension of the standard Hidden Markov Model (HMM) language modeling framework used in speech recognition, which has been factored to represent hierarchic information about language structure over time. This section will review HMMs (Section 3.2.1) and Hierarchic HMMs (Sec­tions 3.2.2 and 3.2.3). This underlying frameworkwill then be extended to include random variables over semantic referents in Section 4.2.

3.2.1 HMMs and Language Models. The model described in this article is a specialization of the HMM framework commonly used in speech recognition (Baker 1975; Jelinek, Bahl, and Mercer 1975). HMMs characterize speech as a sequence of hidden states ht(which may consist of speech sounds, words, or other hypothesized syntactic or se­mantic information), and observed states ot (typically finite, overlapping frames of an audio signal) at corresponding time steps t. A most-probable sequence of hidden states hi..r can then be hypothesized given any sequence of observed states oi..r, using Bayes' Law (Equation 2) and Markov independence assumptions (Equation 3) to define the full probability P(h1..T | o1..T)asthe productofa Language Model (LM) prior proba­bility P(h1..T) ==111 PeLM(ht |ht-1)and an Acoustic Model (AM) likelihood probability P(o1..T I h1..T) = Ut PeAM(ot I ht):

The initial hidden state h0 may be defined as a constant. HMM transitions can be modeled using Weighted Finite State Automata (WFSAs), corresponding to regular expressions. An HMM state ht may then be defined as a WFSA state, or a symbol position in a corresponding regular expression.

3.2.2 Hierarchic HMMs. Language model transitions PeLM(<rt | at-1) over internally structured hidden states at can be modeled using synchronized levels of stacked-up component HMMs in an HHMM (Murphy and Paskin 2001), generalized here as an abstract topology over unspecified random variables p and ct. In this topol­ogy, HHMM transition probabilities are calculated in two phases: a "reduce" phase (resulting in an intermediate, marginalized state pt at time step t), in which compo­nent HMMs may terminate; and a "shift" phase (resulting in a modeled state <rt), in which unterminated HMMs transition, and terminated HMMs are re-initialized from their parent HMMs. Variables over intermediate and modeled states are factored 2 It is also common to define a prior distribution over initial states at ho, but this is not necessary here.

A referential semantic language model can now be defined as an instantiation of an HHMM (as described in Section 3.2), interpreting directives in a reified world model (as described in Section 3.i). This interpretation frameworkis novel in that it is defined dynamically in terms of transitions over referential states—evocations of entity referents from a (e.g., first-order) world model—stacked up in a Hierarchic HMM. This allows (i) a straightforward fast implementation of semantic interpretation (as transition) that is compatible with conventional time-series models used in speech recognition; and (2) a broader notion of semantic composition that exploits referential context in time order (from previous constituents to later constituents) as well as bottom-up (from component constituents to composed constituents).

First, Section 4.i will describe a definition of semantic constraints as transitions in a time-series model. Then Section 4.2 will apply these transitions to nested referents in a Hierarchic HMM. Section 4.3 will introduce a state-based syntactic representa­tion to linkthis semantic representation with recognized words. Finally, Section 4.4 will demonstrate the expressive power of this model on some common linguistic constructions.

Because this section combines notation from different theoretical frameworks (in particular, from formal semantics and statistical time-series modeling), a notation summary is provided in Table i.

Semantic interpretation maybe easily integrated into a probabilistic time-series model if it is formulated as a type of transition, from source to destination referents of equivalent type at adjacent time steps. In other words, while relations in an ordinary Montagovian interpretation framework(Montague i973) may be functions from entity referents to truth value referents, all relations in the world model defined here must be transition functions from entity referents to entity referents.

One-place properties l may be modeled in this system by defining transitions from preceding, unconstrained referents to referents constrained by l. The unconstrained referents can be thought of as context arguments: For example, in the context of the set of user-writable files, a property like EXECUTABLE evokes the subset of writable executables. In the subsumption lattice shown in Figure i, this will define a rightward transition from each set referent to some subset referent, labeled with the traversed relation (see Figure 4 in Section 4.4).

Summary of notation used in Section 4.

Model theory (see Section 3.i) M :a world model EM : the domain of individuals in world model M i : an individual

0M : an interpretation function from logical symbols (e.g., relation labels)

to logical functions over individuals, sets of individuals, etc. variables with asterisks : refer to an initial world model prior to reification the type of an individual the type of a truth value the type of a function from type a to type p (variables over types)

Set theory (see Section 4.i)

R : a relation over tuples of individuals

Random variables (see Sections 3.2 and 4.2) h : a hidden variable in a time-series model o : an observed variable in a time-series model,

(in this case, a frame of the acoustical signal) p : a complex variable occurring in the reduce phase of processing; for example, composed of (ep ,fp}

0 : a complex variable occurring in the shift phase of processing;

for example, composed of (ed, qd} f : a random variable over final state status; for example, with value 1 or 0 q : a random variable over FSA (syntax) states,

in this case compiled from regular expressions; for example, with value qi or q2 e : a random variable over referent entities; for example, with value e{L1L2L3j 1 : a random variable over relation labels; for example, with value EXECUTABLE t : a time step, from i to the end of the utterance T d : a depth level, from i to the maximum depth level D

Q : a probability model mapping variable values to probabilities

(real numbers form 0.0 to i.0) L : functions from FSA (syntax) states to relation labels instances or values of a random variable are specific to a time step; for example, ptare specific to a reduce or shift phase within a time step; for example, ep t, qdtnon-bold variables with superscripts : are specific to a depth level; for example, pdt, ep t

General w-ary semantic relations l in this frameworkare therefore formulated as a type of multi-source transition, distinguishing one argument of an original, ordinary relation l* as an output (destination) and leaving the rest as input (source); then intro­ducing a context referent as an additional input. Instead of defining simple transition arcs on a subsumption lattice, w-ary relations more accurately define hyperarcs,with multiple source referents: zero or more conventional arguments and one additional con­text referent, leading to a destination referent intersectively constrained to this context.

Like the simple HHMM described in Section 3.2.3, the referential semantic language model described in this article (henceforth RSLM), is defined by instantiating the gen­eral HHMM "template" defined in Section 3.2.2. This RSLM instantiation incorporates both the switching variables f e{0,1} and FSA state variables q of the simple HHMM, and adds variables over semantic referents e to the "reduce" and "shift" phases at each level. Thus, the RSLM decomposes each HHMM reduce variable pd into a joint variable subsuming an intermediate referent eptt and a final-state switching variable fp t;and decomposes each HHMM shift variable of into a joint variable subsuming a modeled referent editt and an ordinary FSA state :

A graphical representation of this referential semantic language model is shown in Figure 3.

The intermediate referents edPtt in this frameworkcorrespond to the traditional notion of compositional semantics (Frege 1892), in which meanings of composed constituents (at higher levels in the HHMM hierarchy) are derived from meanings of component constituents (at lower levels in the hierarchy). However, in addition to the referents

A graphical representation of the dependencies in the referential semantic language model described in this article (compare with Figure 2). Again, circles denote random variables and edges denote conditional dependencies. Shaded circles denote random variables with observed values.

The language model described herein defines referential semantics purely in terms of HHMM shift and reduce operations over referent entities, made from reified sets of individuals in some original world model. This section will show that this basic model is sufficiently expressive to represent many commonly occurring linguistic phenomena,

5 When expressed as a function of the size of the grammar, HHMM recognition is asymptotically exponential on D, whereas CFG recognition is cubic regardless of depth. In practice, however, exact inference using either formalism is impractical, so approximate inference is used instead (e.g., maintaining a beam at each time step or at each constituent span in CFG parsing).

A subsumption lattice (laid on its side, in gray) over the power set of a domain containing three files:/1 (a writable executable),f2 (a read-only executable), andf3 (a read-only data file). "Reference paths" made up of conjunctions of relations l (directed arcs, in black) traverse the lattice from left to right toward the empty set, as referents (e{...}, corresponding to sets of files) are incrementally constrained by intersection with each [ljx. (Some arcs are omitted for clarity.) including intersective modifiers (e.g., adjectives like executable), multi-argument rela­tions (e.g., prepositional phrases or relative clauses, involving trajector and landmark referents), negation (as in the adverb not), and comparatives over continuous properties (e.g., larger).

4.4.1 Properties. Properties (traditionally unary relations like EXECUTABLE or WRITABLE) can be represented in the world model as labeled edges lt from supersets et-1 to subsets etdefined by intersecting the set et-1 with the set lltjM satisfying the property lt. Recall that a reified world model can be cast as a subsumption lattice as described in Section 3.1.2. The result of conjoining a property l with a context set e can therefore be found by downward traversal of an edge in this lattice labeled l and departing from e.

Thus, in Figure 4, the set of executables that are read-only would be reachable by traversing a READ-ONLY relation from the set of executables, or by traversing an EX­ECUTABLE relation from the set of read-only objects, or by a composed path READ-ONLYoEXECUTABLE or EXECUTABLEoREAD-ONLY from eT. The resulting set may then serve as context for subsequent traversals. Property relations may also result in self-traversals (e.g., DATAFlLEoREAD-ONLY in Figure 4) or traversals to the empty set e_|_ (e.g., DATAFlLEoWRITABLE). Property relations like EXECUTABLE can be defined using the dynamic relations in the first case of Equation (12) in Section 4.1, which simply ignore the non-context argument.

A general template for intersective nouns and modifiers can be expressed as a noun phrase (NP) expansion using the following regular expression (where ld and ld indicate relation labels constraining referents at the beginning and end of the NP):

6 Although properties (and later, n-ary relations) are defined in terms of an exponentially large subsumption lattice, this lattice need not be an actual data structure. If the world model is queried from a decoder trellis with a beam filter rather than from a complete search, only those lattice relations that are phonologically, syntactically, and semantically most likely (in other words, those that are on this beam) will be explored.

Reference paths for a relation containing in the directory containing the executable file. A reference path forks to specify referents using a two-place relation CONTAIN in a domain of directories di,d2,d3 and filesfi,f2,f3. Here, d2 containsf2 and d3 containsf3,and fi and f2 are executable. The ellipsis in the referent set indicates the presence of additional individuals that are not directories. Again, subsumption is represented in gray and relations are represented in black. (Portions of the complete subsumption lattice and relation graph are omitted for clarity.) functioning as a stack). A sample template for branching reduced relative clauses (or prepositional phrases) that exhibit this forking behavior can be expressed as below:

RC — containing NP; with la = CONTAIN and lp = CONTAIN' (22) where the inverse relation CONTAIN' is applied when the NP expansion concludes or reduces (when the forked paths are re-joined). Relations like CONTAIN are covered in the second case of Equation (12) in Section 4.1, which define transitions from sets of individuals associated with one argument of an original relation CONTAIN* to sets of individuals associated with the other argument of this relation, in the presence of a context set, which is a superset of the destination. The calculation of semantic tran­sition probabilities for n-ary relations thus resembles that for properties, except that the probability term associated with the relation la and the inverse relation lp would depend on both context and argument referents (to its left and below it, in the HHMM hierarchy).

Note that there is ultimately a singleton referent {f2} of the executable file in Figure 5, even though there are two executable files in the world model used in these examples. This illustrates an important advantage of a dynamic context-dependent (three referent) model of semantic composition over the strict compositional (two referent) model. In a dynamic context model, the executable file is interpreted in the context of the files that are contained in a directory. In a strict compositional model, the executable file is interpreted only in the context of fixed constraints covering the entire utterance, and the constraints related to the relation containing are applied only to the directories. This means that a generative model based on strict composition will assign some probability to an infi­nitely recursive description the directories containing executables contained by directories ... In generation systems, this problem has been addressed by adding machinery to keep trackof redundancy (Dale and Haddock1991). But in this framework, a description model (©Ref-Init) which is sensitive to the sizes of its source referent and destination ref­erent at the end of each departing labeled transition will be able to disprefer referential transitions that attempt to constrain already singleton referents, or that provide only trivial or vacuous (redundant) constraints in general. This solution is therefore more in line with graph-based models of generation (Krahmer, van Erk, and Verleg 2003), except that the graphs proposed here are over reified sets rather than individuals, and the goal is a generative probability model of language rather than generation per se.

4.4.3 Negation. Negation can be modeled in this frameworkas a relation between sets. Although it does not require any syntactic memory, negation does require referential semantic memory, in that the complement of a specified set must be intersected with some initial context set. Files that are not writable must still be files after all; only the writable portion of this description should be negated.

A regular expression for negation of adjectives is and is applied to a world model in Figure 6. Relations like NOT are covered in the third case of Equation (12) in Section 4.1, which define transitions between sets in an original relation NOT*.

4.4.4 Comparatives, Superlatives, and Subsective Modifiers. Comparatives (e.g., larger), superlatives (e.g., largest), and subsective modifiers (e.g., large, relative to some context

Reference paths for negation in files that are not writable, using a world model with files f1, f2, andf3 of which onlyf1 is writable. The recognizer first forks a copy of the set of files {1,f2,f3} using the relation IDENTITY, then applies the adjective relation WRITABLE to yield {f1}.The complement of this set {f2,f3,... } is then intersected with the stored top-level referent set {f1,f2,f3} to produce the set of files that are not writable: {f2,f3}. Ellipses in referent sets indicate the presence of additional individuals that are not files.

set) define relations from sets to sets, or from sets to individuals (singleton sets). They can be handled in much the same way as negation. Here the context is provided from previous words and from sub-structure, in contrast to DeVault and Stone (2003), which define the context of a comparative either from fixed inter-utterance constraints or as the referent of the portion of the noun phrase dominated by the comparative (in addition to inter-utterance constraints). One advantage of dynamic (time-order) constraints is that implicit comparatives (in the Clark directory, select the file that is larger, with no complement) can be modeled with no additional machinery. If substructure context is not needed, then no additional HHMM storage is necessary. A regular expression for superlative adjectives is

Noun — largest Noun; with la = IDENTITY and lp = LARGEST (24) and is applied to a world model in Figure 7. Relations like LARGEST are also covered in the third case of Equation (12), which defines transitions between sets in an original relation LARGEST*.

Much of the motivation for this approach has been to develop a human-like model of language processing. But there are practical advantages to this approach as well. One of the main practical advantages of the referential semantic language model described

Reference paths for a comparative in the largest executable; this forks a copy of the referent set {f\ffiff3} using the relation IDENTITY, applies EXECUTABLE to the forked set to obtain {f1,f2}, and returns the referent {f2} with the largest file size using LARGEST.

To evaluate the contribution to recognition accuracy of referential semantics over that of syntax and phonology alone, a baseline (syntax only) and test (baseline plus referential semantics) recognizer were run on sample ontology manipulation directives in a "stu­dent activities" domain. This domain has the form of a simple tree-like taxonomy, with some cross-listings (for example, students may be listed in homerooms and in activities).

Taxonomic ontologies (e.g., for organizing biological classifications or computer file directories) can be mapped to reified world models of the sort described in Section 3.1.2. Concepts C in such an ontology define sets of individuals described by that concept: {l|C(l)}. Subconcepts C of a concept C then define subsets of individuals: [i\C(l)}C (l|C(l)}. These sets and subsets can be reified as referent entities and arranged on a subsumption lattice as described in Section 3.1.2. A sample taxonomic ontology is shown in Figure 8a (tilted on its side to match the subsumption lattices shown elsewhere in this article). Thus defined, such ontologies can be navigated using referent transitions described in Section 4.1 by entering concept referents via "downward" (rightward in the figure) transitions, and leaving concept referents via "upward" (leftward) transitions. For example, this ontology can be manipulated using directives such as:

(1) set Crookston campus homeroom two Clark to sports football captain which are incrementally interpreted by transitioning down the subsumption lattice (e.g., from sports to football to captain) or forking to another part of the lattice (e.g., from Clark to sports).

As an ontology like this is navigated in spoken language, there is a sense in which other referents e' at the same level of the ontology as the most recently described refer­ent e, or at higher levels of the ontology than the most recently described entity, should be semantically accessible without restating the ontological context (the path from the root concept ey)sharedby e' and e. Thus, in the context of having recently referred to someone in Homeroom 2 at a particular campus in a school activities database, other students in the same homeroom or other activities at the same campus should be accessible without giving an explicit back up directive at each branch in the ontology. To see the value of implicit upward transitions, compare Example (1) to a directive that makes upward transitions explicit using the keyword back (similar to in the syntax of Unix paths) to exit the homeroom two and Clark folders:

(2) set Crookston campus homeroom two Clarkto backbacksports football captain or if starting from the Duluth campus sports football directory:

(3) set backbackbackCrookston campus homeroom two Clarkto backbacksports football captain

Instead of requiring explicit back keywords, these upward transitions can be implic­itly composed with downward transitions, resulting in transitions from source eS1 to destination eS via some ancestor eS0:

The composed transition function finds a referent eS0 which subsumes both eS1 and eS, then finds an ordinary (downward) transition l connecting eS0 to eS. The result is a Up-l transition to every immediate child of an ancestor a referent (or in genealogical terms,

Upward and downward transitions in a sample student activities world model. Downward transitions (a) define basic sub-type relations. Upward transitions (b) relate sibling, ancestor, and (great-great-...-)aunt/uncle concepts. The entire model is reachable from any given referent via these two kinds of transitions.

to every sibling, ancestor, and sibling of ancestor), making these contextually salient concepts immediately accessible without explicit back-stepping (see Figure 8b).

Downward transitions are ordinary properties, as defined in the first case of Equation (12) in Section 4.1.

Although navigation in this domain is constrained to tree-like graphs, this domain tests all of the features of a referential semantic language model that would be required in richer domains. As described in Section 4, rich domains (in particular, first-order domains, in which users can describe sets of individuals as referents) are mapped to transition edges on a simple graph, similar to the tree-like graphs used in this ontology. In first-order domains, the size of this graph may be exponential on the number of individuals in the world model. But once the number of referents exceeds the size of the decoder beam, the time performance of the recognizer is constrained not by the number of entities in the world model, but by the beam width and the number of outgoing relations (labels) that can be traversed from each hypothesis. In a first-order system, just as in the simple ontology navigation system evaluated here, this number of relations is constrained to the set of words defined by the user up to that point. In both cases, although the interface may be used to describe any one of an arbitrarily large set of referents, the number of referents that can be evoked at the next time step is bounded by a constant.

When this model is extended to first-order or continuous domains, the time re­quired to calculate sets of individuals or hypothetical planner states that result from a transition may be nontrivial, because it may not be possible in such domains to retain the entire referent transition model in memory. In first-order domains, for example, this may require evaluating certain binary relations over all pairs of individuals in the world model, with time complexity proportional to the square of the size of the world model domain. Fortunately the model described herein, like most generative language models, hypothesizes words before recognizing them. This means a recognizer based on this model will be able to compute transitions that might follow a hypothesized word during the time that word is being recognized. If just the current set of possible transitions is known (say, these have already been pre-fetched into a cache), the set of outgoing transitions that will be required at some time following one of these current transitions can be requested as soon as the beginning of this transition is hypothesized—as soon as any word associated with this transition makes its way onto the decoder beam. From this point, the recognizer will have the entire duration of the word to compute (in parallel, in a separate thread, or on a separate server) the set of outgoing transitions that may follow this word. In other words, the model described herein may be scaled to richer domains because it is amenable to parallelization.

The student activities ontology used in this evaluation is a taxonomic world model defined with upward and downward transitions as described in Section 5.1. It organizes extracurricular activities under subcategories (e.g., offense C football C sports), and organizes students into homerooms, in which context they can be identified by a single (first or last) name. Every student or activity is an entity e in the set of entities E,and relations l are subcategory labels or student names.

5.3.1 World Model M240. In the original student activities world model M240, a total of 240 entities were created in E: 158 concepts (groups or positions) and 82 instances (students), each connected via a labeled arc from a parent concept.

Because a world model in this frameworkis a weighted set of labeled arcs, it is possible to calculate a meaningful perplexity statistic for transitions in this model, assuming all referents are equally likely to be a source. The perplexity of this world model (the average number of departing arcs) is 16.79, after inserting "UP" arcs as described in Section 5.1.

Baseline and test versions of this system were run using a Recurrent Neural Network (RNN) acoustic model (Robinson 1994). This acoustic model performs competitively with multi-state triphone models based on multivariate Gaussian mixtures, but has the advantage of using only uniphones with single subphone states. As a result, less of the HMM trellis beam is occupied with subphone variations, so that a larger number of semantically distinct hypotheses may be considered at each frame.

Each model was evaluated using parameters trained from the TIMIT corpus of read speech (Fisher et al. 1987). This corpus yields several thousand examples for each of the relatively small set of single-state uniphones used in the RNN model. Read speech is also appropriate training data for this evaluation, because the test subjects are constrained to perform fixed edit tasks given written prompts, and the number of reasonable ways to perform these tasks is limited by the ontology, so hesitations and disfluencies are relatively rare.

The language model used in these experiments is decomposed into five hierarchic levels, each with referent e and ordinary FSA state q components, as described in Section 4.2. The top three levels of this model represent syntactic states as q (derived from regular expressions defined in Section 4.3) and associated semantic referents as e. The bottom two levels represent pronunciation and subphone states as q, and ignore e.

Transitions across pronunciation states are defined in terms of sequences of phones associated with a word via a pronunciation model. The pronunciation model used in these experiments is taken from the CMU ARPABET dictionary (Weide 1998). Transi­tions across subphone states are defined in terms of sequences of subphones associated with a phone. Because this evaluation used an acoustic model trained on the TIMIT corpus (Fisher et al. 1987), the TIMIT phone set was used as subphones. In most cases, these subphones map directly to ARPABET phones, so each subphone HMM consists of a single, final state; but in cases of plosive phones (B, D, G, K, P,and T), the subphone HMM consists of a stop subphone (e.g., bcl) followed by a burst subphone (e.g., b).

The three upper levels of the HHMM comprise the syntactic and referential portion of the language model. Concept error rate tests were performed on three baseline and test versions of this portion of the language model, using the same acoustic, phone, and subphone models, as described in Sections 5.5 and 5.6.

5.7.1 Language Model @LMsem. First, the syntactic and referential portion of the language model was implemented as described in Section 4.2. A subset of the regular expres­sion grammar appears in Figure 9. Any nondeterminism resulting from disjunction or Kleene-star repetition in the regular expressions was handled in ©Syn-Trans using uniform distributions over all available following states. Distributions over regular expression expansions in ©Syn-Init were uniform over all available expansions. Distributions over labels in @Ref-Init were also uniform over all labels departing the entity referent condition that were compatible with the FSA state category generated by ©Syn-Init.

Sample grammar for student activities domain. Relations lt,lp = IDENTITY unless otherwise specified.

Per-subject results for Language Model 0LM-Sem with M240.

Per-subject results for Language Model 0LM-Sem with M4i75.

Here the size of the vocabulary was roughly equal to the number of referents in the world model. The sentence error rate for this experiment was 59.44%.

5.8.2 Language Model &LM-Sem and World Model M4i75. With the number of entities (and words) increased to 4,i75 (M4i75), the CER increases slightly to i9.9% (Table 3). Here again, the size of the vocabulary was roughly equal to the number of referents in the world model. The sentence error rate for this experiment was 62.24%. Here, the use of a world model (Language Model ©LM-Sem) with no linguistic training data is comparable to that reported for other large-vocabulary systems (Seneff et al. 2004; Lemon and Gruenstein 2004), which were trained on sample sentences.

5.8.3 Language Model @LM-NoSem with no World Model. In comparison, a baseline using only the grammar and vocabulary from the students domain M240 without any world model information and no linguistic training data (Language Model ©LM-NoSem) scores 43.5% (Table 4). The sentence error rate for this experiment was 93.0i%.

Ignoring the world model significantly raises error rates compared to Model ©LM-Sem (p < 0.0i using pairwise t-test against Language model ©LM-Sem with M240, grouping scores by subject), suggesting that syntactic constraints are poor predictors of

9 Ordinarily a syntactic model would be interpolated with word n-gram probabilities derived from corpus training, but in the absence of training sentences these statistics cannot be included.

Per-subject results for Language Model 0LM-NoSem.

concepts without considering reference. But this is not surprising: because the grammar by itself does not constrain the set of ontology labels that can be used to construct a path, the perplexity of this model is 240 (reflecting a uniform distribution over nearly the entire lexicon), whereas the perplexity of M240 is only i6.79.

5.8.4 Language Model &LM-rrigram and World Model M240. In order to test how well the model would function as a pre-process to a conventional trigram-based speech recog­nizer, the referential semantic language model (Language Model ©LM-Sem) was compiled into a word trigram model. This word trigram language model (Language Model ©LM-Trigram), compiled from the referential semantic model (in the 240-entity domain), shows a concept error rate of 26.6% on the students experiment (Table 5). The sentence error rate for this experiment was 66.43%.

Using trigram context (Language Model ©LM-Trigram) similarly shows statistically significant increases in error over Language Model ©LM-Sem with M240 (p = 0.0i using pairwise t-test, grouping scores by subject), showing that referential context is also more predictive than word n-grams derived from referential context. Moreover, the compilation to trigrams required to build Language Model ©LM-Trigram is expensive (requiring several hours of pre-processing) because it must consider all combinations of entities in the world model. This would make the pre-compiled model impractical in mutable domains.

Per-subject results for Language Model 0LM-Trigram with M240.

Experimental results with four model configurations.

5.8.5 Summary of Results. Results in Table 6 summarize the results of the four experiments.

Some of the erroneously hypothesized directives in this domain described im­plausible edits: for example, making one student a subset of another student. Domain information or meta-data could eliminate some of these kinds of errors, but in content-creation applications it is not always possible to provide this information in advance; and given the subtle nature of the effect of this information on recognition, it is not clear that users would want to manage it themselves, or allow it to be automatically induced without supervision.ici In any case, the comparison described in this section to a non-semantic model ©LM-NoSem suggests that the world model by itself is able to apply useful constraints in the absence of domain knowledge. This suggests that, in an interpolated approach, direct world model information may relieve some of the burden on authored or induced domain knowledge to perform robustly, so that this domain knowledge may be authored more sparsely or induced more conservatively than it otherwise might.

All evaluations ran in real time on a 4-processor dual-core 2.6GHz server, with a beam width of i,000 hypotheses per frame. Differences in runtime performance were minimal, even between the simple trigram model and HHMM-based referential seman­tic language models. This was due to two factors:

1. All recognizers were run with the same beam width. Although it might be possible to narrow the beam width to produce faster than real-time performance for some models, widening the beam beyond i,000 did not return significant reductions in CER in the experiments described herein.

2. The implementation of the Viterbi decoder used in these experiments was optimized to skip combinations of joint variable values that would result in zero probability transitions (which is a reasonable optimization for any factored time-series model), significantly decreasing runtime for HHMM recognition.

5.8.6 Statistical Significance vs. Magnitude of Gain. The experiments described in this article show a statistically significant increase in accuracy due to the incorporation of referential semantic information into speech decoding. But these results should not be interpreted to demonstrate any particular magnitude of error reduction (as might be claimed for the introduction of head words into parsing models, for example).

10 Ehlen et al. (2008) provide an example of a user interface for managing imperfect automatically-induced information about taskassignments from meeting transcripts, which is much more concrete than the kind of domain knowledge inference considered here.

This article has described a referential semantic language model that achieves recogni­tion accuracy favorably comparable to a pre-compiled trigram baseline in user-defined domains with no available domain-specific training corpora, through the use of ex­plicit hypothesized semantic referents. This architecture requires that the interfaced application make available a queryable world model, but the combined phonological, syntactic, and referential semantic decoding process ensures the world model is only queried when necessary, allowing accurate real time performance even in large domains containing several thousand entities.

The frameworkdescribed in this article is defined over first-order sets (of individu­als), making transition functions over referents equivalent to expressions in first-order logic. This frameworkcan be extended to model other kinds of references (e.g., to time intervals or events) by casting them as individuals (Hobbs i985).

The system as defined herein also has some ability to recognize referents con­strained by quantifiers: for example, the directory containing two files. Because its referents are reified sets, the system can naturally model relations that are sensitive to cardinality (self-transitioning if the set has N or greater individuals, transitioning to e_|_ otherwise). But a dynamic view of the referential semantics of nested quantifiers requires referents to be indexed to particular iterations of quantifiers at higher levels of nesting in the HHMM hierarchy (corresponding to higher-scoping quantifiers). Extending the system to dynamically interpret nested quantifiers therefore requires that all semantic opera­tions preserve an "iteration context" of nested outer-quantified individuals for each inner-quantified individual. This is left for future work.

Some analyses of phenomena like intensional or non-inherent adjectives—for ex­ample, toy in toy guns, which are not actually guns; or old in old friends, who are not necessarily elderly (Peters and Peters 2000)—involve referents corresponding to second-order sets (this allows these adjectives to be composed before being applied to anoun: old but casual friend). Unfortunately, extending the frameworkdescribed in this article to use a similarly explicit representation of second- or higher-order sets would be impractical. Not only would the number of possible second- or higher-order sets be exponentially larger than the number of possible first-order sets (which is already exponential on the number of individuals), but the length of the description of each referent itself would be exponential on the number of individuals (whereas the list of individuals describing a first-order referent is merely linear).

The definition of semantic interpretation as a transition function does support interesting extensions to hypothetical reasoning and planning beyond the standard closed-world model-theoretic framework, however. Recall the sentence go to the package data directory and hide the executable files, or equivalently, in the package data directory, hide the executable files, exemplifying the continuous context-sensitivity of the referential semantic language model. Here, the system focuses on the contents of this directory because a sequence of transitions resulting from the combined phonological, syntactic, and referential semantic context of the sentence led it to this state. One may characterize the referential semantic transitions leading to this state as a hypothetical sequence of change directory actions moving the active directory of the interface to this directory (for the purpose of understanding the consequences of the first part of this directive). The hypothesized context of this directory is then a world state or planning state resulting from these actions. Thus characterized, the referential semantic decoder is performing a kind of statistical plan recognition (Blaylock and Allen 2005). By viewing referents as world states, or as having world-state components, it would then be possible to use logical conclusions of other types of actions as implicit constraints—e.g., unpack the tar file and hide the executable [which will result from this unpacking]—without adding extra functionality to the recognizer implementation. Similarly, referents for hypothetical objects like the noun phrase a tar file in the directive create a tar file, are not part of the world model when the user describes them.

Recognizing references to these hypothetical states and objects requires a capacity to dynamically generate referents not in the current world model. The domain of referents in this extended system is therefore unbounded. Fortunately, as mentioned in Section 5.2, the number of referents that can be generated at each time step is still bounded by a constant, equal to the recognizer's beam width multiplied by the num­ber of traversable relation labels. This means that distributions over outgoing relation labels are still well-defined for each referential state. The only difference is that, when modeling hypothetical referents, these distributions must be calculated dynamically.

Finally, this article has primarily focused on connecting an explicit representation of referential semantics to speech recognition decisions. Ordinarily this is thought of as being mediated by syntax, which is covered in this article only through a rela­tively simple frameworkof bounded recursive HHMM state transitions. However, the bounded HHMM representation used in this paper has been applied (without seman­tics) to rich syntactic parsing as well, using a transformed grammar to minimize stack usage to cases of center-expansion (Schuler et al. 2008). Coverage experiments with this transformed grammar demonstrated that over 97% of the large syntactically annotated Penn Treebank(Marcus, Santorini, and Marcinkiewicz i994) could be parsed using only three elements of stackmemory, with four elements giving over 99% coverage. This suggests that the relatively tight bounds on recursion described in this paper might be expressively adequate if syntactic states are defined using this kind of transform.

This transform model (again, without semantics) was then further applied to pars­ing speech repairs, in which speakers repeat or edit mistakes in their directives: for example, select the red, uh, the blue folder (Miller and Schuler 2008). The resulting system models incomplete disfluent constituents using transitions associated with ordinary fluent speech until the repair point (the uh in the example), then processes the speech repair using only a small number of learned repair reductions. Coverage results for the same transform model on the Penn TreebankSwitchboard Corpus of transcribed spontaneous speech showed a similar three- to four-element memory requirement. If this HHMM speech repair model were combined with the HHMM model of referen­tial semantics described in this article, referents associated with ultimately disfluent constituents could similarly be recognized using referential transitions associated with ordinary fluent speech until the repair point, then reduced using a repair rule that discards the referent. These results suggest that an HHMM-based semantic framework such as the one described in this article may be psycholinguistically plausible.

The authors would like to thank the anonymous reviewers for their input. This research was supported by National Science Foundation CAREER/PECASE award 0447685. The views expressed are not necessarily endorsed by the sponsors.