JANUS is a multi-lingual speech-to-speech trans­lation system designed to facilitate communica­tion between two parties engaged in a sponta­neous conversation in a limited domain. In this paper we describe the current design and perfor­mance of the machine translation module of our system. The analysis of spontaneous speech re­quires dealing with problems such as speech dis-fluencies, looser notions of grammatically and the lack of clearly marked sentence boundaries. These problems are further exacerbated by errors of the speech recognizer. We describe how our machine translation system is designed to effectively han­dle these and other problems. In an attempt to achieve both robustness and translation accu­racy we use two different translation components: the GLR module, designed to be more accurate, and the Phoenix module, designed to be more ro­bust. Both modules follow an interlingua-based approach. The translation modules are designed to be language-independent in the sense that they each consist of a general processor that applies in­dependently specified knowledge about different languages. This facilitates the easy adaptation of the system to new languages and domains. We an­alyze the strengths and weaknesses of each of the translation approaches and describe our work on combining them. Our current system is designed to translate spontaneous dialogues in the Schedul­ing domain, with English, Spanish and German as both source and target languages. A recent focus has been on developing a detailed end-to-end eval­uation procedure to measure the performance and effectiveness of the system. We describe this pro­cedure in the latter part of the paper, and present our most recent Spanish-to-English performance evaluation results.

The JANUS System is a large scale multi-lingual speech-to-speech translation system designed to facilitate communication between two parties en­gaged in a spontaneous conversation in a limited domain. A diagram of the architecture of the sys­tem is shown in Figure 1. The system is com­posed of three main components: a speech recog­nizer, a machine translation (MT) module and a speech synthesis module. The speech recognition component of the system is described elsewhere (Woszczyna et al. 1994). For speech synthesis, we use a commercially available speech synthesizer.

The MT module is composed of two separate translation sub-modules which operate indepen­dently. The first is the GLR module, designed to be more accurate. The second is the Phoenix module, designed to be more robust. Both mod­ules follow an interlingua-based approach. The source language input string is first analyzed by a parser, which produces a language-independent interlingua content representation. The interlin-gua is then passed to a generation component, which produces an output string in the target lan­guage.

The discourse processor is a component of the GLR translation module. It disambiguates the speech act of each sentence, normalizes temporal

Speech in Source Language

GLR* Parser Phoenix Parser Discourse Processor

ILT Feature Structure

Discourse Memory & , Calendar

Frame-Slot Structure

Text in Target Language

Speech in Target Language

Figure 1: The JANUS System expressions, and incorporates the sentence into a discourse plan tree. The discourse processor also updates a calendar which keeps track of what the speakers have said about their schedules. The dis­course processor is described in greater detail else­where (Rosé et al. 1995).

The GLR* parser (Lavie and Tomita 1993; Lavie 1994) is a parsing system based on Tomita's Gen­eralized LR parsing algorithm (Tomita 1987). The parser skips parts of the utterance that it cannot incorporate into a well-formed sentence structure. Thus it is well-suited to domains in which non-grammaticality is common. The parser conducts a search for the maximal subset of the original input that is covered by the grammar. This is done using a beam search heuristic that limits the combinations of skipped words considered by the parser, and ensures that it operates within feasible time and space bounds.

The GLR* parser was implemented as an ex­tension to the GLR parsing system, a unification-based practical natural language system (Tomita 1990). The grammars we develop for the JANUS system are designed to produce feature struc­tures that correspond to a frame-based language-independent representation of the meaning of the input utterance. For a given input utterance, the parser produces a set of interlingua texts, or ILTs.

The main components of an ILT are the speech act (e.g., suggest, accept, reject), the sentence type (e.g., state, query-ii, fragment), and the main semantic frame (e.g., free, busy). An ex­ample of an ILT is shown in Figure 2. A detailed ILT Specification was designed as a formal de­scription of the allowable ILTs. All parser output must conform to this ILT Specification. The GLR unification based formalism allows the grammars to construct precise and very detailed ILTs. This in turn allows the GLR translation module to pro­duce highly accurate translations for well-formed input.

The GLR* parser also includes several tools de­signed to address the difficulties of parsing spon­taneous speech. To cope with high levels of am­biguity, the parser includes a statistical disam­biguation module, in which probabilities arc at­tached directly to the actions in the LR parsing table. The parser can identify sentence bound­aries within each hypothesis with the help of a statistical method that determines the probabil­ity of a boundary at each point in the utterance. The parser must also determine the "best" parse from among the different parsable subsets of an input. This is done using a collection of parse evaluation measures which are combined into an integrated heuristic for evaluating and ranking the parses produced by the parser. Additionally, a parse quality heuristic allows the parser to self-

(day-of-week Wednesday)

(time-of-day morning))) ( a-speech-act (*tnultiple* *suggest *accept)) (sentence-type +state))) Sentence: I could do it Wednesday morning too.

judge the quality of the parse chosen as best, and to detect cases in which important information is likely to have been skipped.

Target language generation in the GLR module is done using GenKit (Tomita and Nyberg 1988), a unification-based generation system. With well-developed generation grammars, GenKit results in very accurate translation for well-specified ILTs.

The JANUS Phoenix translation module (May-field et al. 1995) is an extension of the Phoenix Spoken Language System (Ward 1991; Ward 1994). The translation component consists of a parsing module and a generation module. Trans­lation between any of the four source languages (English, German, Spanish, Korean) and five tar­get languages (English, German, Spanish, Korean, Japanese) is possible, although we currently focus only on a few of these language pairs.

Unlike the GLR method which attempts to con­struct a detailed ILT for a given input utterance, the Phoenix approach attempts to only identify the key semantic concepts represented in the ut­terance and their underlying structure. Whereas GLR* is general enough to support both seman­tic and syntactic grammars (or some combination of both types), the Phoenix approach was specifi­cally designed for semantic grammars. Grammat­ical constraints are introduced at the phrase level (as opposed to the sentence level) and regulate semantic categories. This allows the ungrammat-icalities that often occur between phrases to be ignored and reflects the fact that syntactically in­correct spontaneous speech is often semantically well-formed.

The parsing grammar specifies patterns which represent concepts in the domain. The patterns are composed of words of the input string as well as other tokens for constituent concepts. Elements (words or tokens) in a pattern may be specified as optional or repeating (as in a Klcene star mecha­nism). Each concept, irrespective of its level in the hierarchy, is represented by a separate grammar file. These grammars are compiled into Recursive Transition Networks (RTNs).

The interlingua meaning representation of an input utterance is derived directly from the parse tree constructed by the parser, by extracting the represented structure of concepts. This represen­tation is usually less detailed than the correspond­ing GLR ILT representation, and thus often re­sults in a somewhat less accurate translation. The set of semantic concept tokens for the Scheduling domain was initially developed from a set of 45 example English dialogues. Top-level tokens, also called slots, represent speech acts, such as sugges­tion or agreement. Intermediate-level tokens dis­tinguish between points and intervals in time, for example; lower-level tokens capture the specifics of the utterance, such as days of the week, and represent the only words that are translated di­rectly via lookup tables.

The parser matches as much of the input ut­terance as it can to the patterns specified by the RTNs. Out-of-lexicon words are ignored, unless they occur in specific locations where open con­cepts are permitted. A word that is already known to the system, however, can cause a concept pat­tern not to match if it occurs in a position un­specified in the grammar. A failed concept docs not cause the entire parse to fail. The parser can ignore any number of words in between top-level concepts, handling out-of-domain or otherwise un­expected input. The parser has no restrictions on the order in which slots can occur. This can cause added ambiguity in the segmentation of the utterance into concepts. The parser uses a dis­ambiguation algorithm that attempts to cover the largest number of words using the smallest num­ber of concepts.

Figure 3 shows an example of a speaker ut­terance and the parse that was produced using the Phoenix parser. The parsed speech recog­nizer output is shown with unknown (-) and un­expected (*) words marked. These segments of the input were ignored by the parser. The rele­vant concepts, however, are extracted, and strung together they provide a general meaning represen­tation of what the speaker actually said.

Generation in the Phoenix module is accom­plished using a simple strategy that sequentially generates target language text for each of the top level concepts in the parse analysis. Each con­cept has one or more fixed phrasings in the target language. Variables such as times and dates arc extracted from the parse analysis and translated directly. The result is a meaningful translation, but can have a telegraphic feel.

Phoenix Translation Modules

In this paper we described the design of the two translation modules used in the JANUS system, outlined their strengths and weaknesses and de­scribed our efforts to combine the two approaches. A newly developed end-to-end evaluation proce­dure allows us to assess the overall performance of the system using each of the translations meth­ods separately or both combined.

Our evaluations have confirmed that the GLR approach provides more accurate translations, while the Phoenix approach is more robust. Com­bining the two approaches using the parse qual­ity judgement of the GLR* parser results in im­proved performance. We are currently investigat­ing other methods for combining the two transla­tion approaches. Since GLR* performs much bet­ter when long utterances are broken into sentences or sub-utterances which are parsed separately, we are looking into the possibility of using Phoenix to detect such boundaries. We are also develop­ing a parse quality heuristic for the Phoenix parser using statistical and other methods.

Another active research topic is the automatic detection of out-of-domain segments and utter­ances. Our experience has indicated that a large proportion of bad translations arise from the translation of small parsable fragments within out-of-domain phrases. Several approaches are under consideration. For the Phoenix parser, we have implemented a simple method that looks for small islands of parsed words among non-parsed words and rejects them. On a recent test set, we achieved a 33% detection rate of out-of-domain parses with no false alarms. Another approach we are pursuing is to use word salience measures to identify and reject out-of-domain segments.

We are also working on tightening the coupling of the speech recognition and translation modules of our system. We are developing lattice parsing versions of both the GLR* and Phoenix parsers, so that multiple speech hypotheses can be efficiently analyzed in parallel, in search of an interpretation that is most likely to be correct.

The work reported in this paper was funded in part by grants from ATR - Interpreting Telecom­munications Research Laboratories of Japan, the US Department of Defense, and the Verbmobil Project of the Federal Republic of Germany.

We would like to thank all members of the JANUS teams at the University of Karlsruhe and Carnegie Mellon University for their dedicated work on our many evaluations.

pected, this result strengthens our belief in the poten­tial of this approach.