Natural language understanding systems have im­proved markedly in recent years, and natural language interfaces have even begun to enter the commercial marketplace, for example, the INTELLECT system of Artificial Intelligence Corporation (Harris 1978). These systems promise to make major improvements in the ease of use of data base management and other computer systems. However, they have only begun to consider the problems of truly natural input. The emphasis has been, and continues to be, on the under­standing of well-formed inputs. True natural language input is often ill-formed in the absolute sense of being filled with misspellings, mistypings, mispunctuations, tense and number errors, word order problems, run-on sentences, extraneous forms, meaningless sentences, and impossible requests. In addition, natural input is ill-formed in the relative sense of containing requests that are beyond the limits of either the computer sys­tem or the natural language interface.

Evidence indicates that absolutely ill-formed input regularly occurs in a data base query environment. For instance, in an extensive study (Thompson 1980) including 1615 inputs, only 1093 were parsable, and an overall total of 446 contained various kinds of er­rors: 161 with vocabulary problems, 72 with punctua­tion errors, 62 with ungrammaticality, and 61 with spelling errors. Furthermore, 211 inputs were frag­mentary, including 91 parsed terse replies and 67 parsed terse questions.

In another experiment (Eastman and McLean 1981), 693 English queries to a data base system were analyzed. Co-occurrence violations, including subject-verb disagreement, tense errors, apostrophe problems, and possessive/plural errors occurred in 12.3% of the queries. Omitted words, extraneous words/phrases, telegraphic ellipsis, and incomplete sentences arose in 14% of the queries.

Our conjecture is that wherever natural language interfaces are employed, ill-formed input will occur. Any natural language interface, when faced with ill-formed input, must either intelligently guess at a user's intent, request direct clarification, or at the very least accurately identify the ill-formedness. As Wilks (1976) states, "Understanding requires, at the very least, ... some attempt to interpret, rather than merely reject, what seems to be ill-formed utterances."

Though experience has shown that users can adapt to the limitations of the system's well-formed antici­pated input (Harris 1977b, Hendrix et al. 1978), we feel that relying on such user adaptation ignores one of the most powerful motivations for English input: enabling infrequent users to access data without an intermediary and without extensive practice. Even the person who frequently uses such a system will be exas­perated if it cannot explain why it misunderstands an input.

In some circumstances, ill-formed input may be less frequent. For instance, Fineman (1983) reports that in an experiment where users were constrained to dis­crete speech, ill-formedness occurred in as little as 2% of the input. Another unusual environment can be created by informing the users that the system cannot really understand natural language, thus biasing lan­guage use.

In addition to natural language interfaces, process­ing ill-formed input is critical to correcting language use. Prototype systems have already been investigated in the language-learning environment (Weischedel et al. 1978) and in the office automation environment of document preparation (Miller et al. 1981). Even in published text unintentional ungrammaticalities occur.

Most natural language understanding systems deal with a few types of ill-formedness. Out of our own work, and that of others, we have produced a frame­work for processing ill-formed input. This approach treats ill-formedness as rule-based. First, natural lan­guage interfaces should process all input as presumably well-formed until the rules of normal processing are violated. At that point, error handling procedures based on meta-rules relating ill-formed input to wellformed structures through the modification of the vio­lated normal rules should be employed. These meta­rules correspond to types of errors.

The rest of the paper argues for this rule-based approach. Section 2 characterizes both the types of ill-formed input, and the types of possible approaches to them, including our proposal. Section 3 gives ex­amples of meta-rules for processing ill-formed input. Section 4 describes how some heuristics developed by others fit within our paradigm. An implementation is sketched in Section 5. Section 6 discusses limitations of the proposal. Sections 7 and 8 present directions for future work and conclusions.

This section introduces the problem of interpreting ill-formed input. First, we discuss the types of ill-formed input briefly. Then we consider the range of approaches that have been tried for allowing for such input.

Ill-formedness phenomena can be divided into two sets. The first defines what we call absolute ill-formedness. An utterance is absolutely ill-formed if the typical listener considers it ill-formed. The defini­tion unfortunately appeals to subjective evaluations; these are known to differ widely (Ross 1979). But it seems to include the majority of typical cases and exclude the majority of types of good English sen­tences.

The second set defines relative ill-formedness. This is ill-formedness with respect to the normal processing rules of the formal computing system including the natural language interface and the underlying applica­tion system. The set of ill-formed inputs for an inter­face can be defined as the union of these two sets for that interface.

The set of ill-formed input captured by these defi­nitions can also be seen through the four typical phas­es of interpretation in natural language interfaces: lexical, syntactic, semantic, and pragmatic processing. In lexical processing, absolute ill-formedness can come from misspelling, mistyping, and mispronunciation; relative ill-formedness can arise from unknown words. In syntactic processing, absolute ill-formedness is seen in faulty subject-verb agreement, word order errors, omitted words, run-on sentences, etc; relative ill-formedness is seen in grammatical combinations of words that exceed the interface's grammar.

Semantic processing can be defined as the interpre­tation of the input in isolation. Knowledge of the task domain can be applied, but the context of input with respect to previous interactions and the state of the underlying computing system are only considered in pragmatic processing.

Absolute ill-formedness in semantics includes omit­ting needed information and violating of selectional restrictions. Absolute ill-formedness in pragmatics includes breaking the rules of conversation, as when answering a question with a question, having presup­positions of the speaker fail, and failing to make clear an anaphoric reference. Relative ill-formedness in both cases includes "overshoot", requesting capabili­ties or information not covered by the system in its current state, and parenthetical expressions incompre­hensible to the system.

To further argue for meta-rules as a uniform frame­work for processing ill-formed input, we describe a wide variety of meta-rules in this section and the next. We adopt the following notation for meta-rules in this paper:

Approach 1

Characterization: Do not encode well-formedness constraints.

Flaw: Well-formedness rules convey meanings by constraining interpretations.

Approach 2

Characterization: Design systems for well-formed input and ill-formed input together.

Flaw: This gives no preference of well-formed interpretations over ill-formed ones.

Approach 3

Characterization: Search for well-formed interpretations prior to considering any ill-formed ones.

Flaw: This does not explicitly relate handling ill-formedness to processing well-formedness.

Approach 4

Characterization: Use a metric to rank ill-formedness interpretations, and select the one that comes closest to satisfying all constraints.

Flaw: This does not state what the deviation is so that one may draw inferences from the ill-formedness, nor does it capture actual patterns of error.

Table 1. Four Rejected Approaches.

Many natural language interfaces have some heuristics for processing one or more classes of ill-formed input. Since an exhaustive analysis would be impossible here, we will review only a handful of techniques that have inspired us to develop the meta-rule framework. We describe each technique by showing how it could be phrased as a meta-rule within our paradigm.

The LADDER system (Hendrix et al. 1978) imple­ments three major techniques for processing ill-formed input. All fit within the framework we suggest. One deals with recovery from lexical processing. In this system, the developer of a question-answering system prepares only a dictionary of well-formed words. If a sentence contains a word that is not in the dictionary, the parser will fail. The system localizes the area of failure to the ATN state associated with the partial interpretation that has proceeded rightmost in the input and that is shallowest (in terms of incompleted ATN PUSH arcs). Candidates for the correct spelling are limited to the words that would permit the parser to proceed and that are close to the spelling that ap­pears. An equivalent meta-rule would check in the LHS that the parser failed. The RH S would compute a list of words expected next for each type of arc leav­ing that state, for example, the category members and literal words expected next. The next action would apply the Interlisp spelling correction algorithm to postulate a known word that was expected next. This word would replace the unrecognized one in the input and parsing would resume. A similar heuristic is run­ning in our current implementation, with the addition that, if the unrecognized word appears to have an inflected ending, spelling correction is performed on the possible root.

A second technique in LADDER deals with under­standing contextual ellipsis, if no parse for the input is found. This heuristic interprets "the fastest submarine" as "To what country does the fastest sub­marine belong", if it occurs after a query such as "To what country does each merchant ship in the North Atlantic belong". In Weischedel and Sondheimer (1982), we extended that heuristic to allow for turn-taking in dialogues and to allow expansions as well as substitutions, such as the elliptical form "Last month" following "Did you go to Chicago?"

A third technique in LADDER is the printing of error messages, in the same sense that meta-rule (v) above prints a message when all attempts have failed. We could phrase this heuristic as a meta-rule whose LHS would check that the parser has blocked. This meta-rule would be ordered strictly after the ones for spelling correction and contextual ellipsis. A state would be postulated as the locale of the problem by the same heuristic as for spelling correction. The RHS would print for each arc that leaves that state the cate­gory, constituent, or word that was expected by that arc.

Hayes and Mouradian (1980) emphasize recovery techniques for blocking during left-corner parsing (Aho and Ullman 1972). Their strategies are invoked only if the parser blocks. Two of them can be refor­mulated as meta-rules as follows. One meta-rule would check in its LHS that the parser was blocked and that a special parser variable (call it BLOCKED-PARSE) was empty. The RHS would save the blocked configuration in BLOCKED-PARSE, and start parsing as if the current word were the first word of the input. This would enable the system to ignore initial strings that could not be understood. A useful example of this is restarted inputs, such as "Copy all print all headers of messages". A second meta-rule is related. The LHS would check whether the parser was blocked and BLOCKED-PARSE had a configuration in it. Furthermore, the LHS would check to see that another parser variable (call it DONE-ONCE) was NIL. If so, the RHS would set DONE-ONCE to T. The RHS would then swap the current configuration with BLOCKED-PARSE and would try resuming the parse from the current word with that configuration. This heuristic is designed to ignore incomprehensible mate­rial in the middle of an input. For instance, it would enable skipping the parenthetical material in "List all messages, assuming there are any, from Brown".

In the area of pragmatics, solutions that could fit within our paradigm have been suggested for two classes of problems. One problem is the failure of presuppositions of an input. In the environment of an intelligent tutor for computer-assisted language in­struction, a technique suggested in Weischedel et al. (1978) could be formulated as a meta-rule as follows. The LHS would check whether processing was blocked due to a presupposition being false. Since that system would have a more complete knowledge of language than a beginning student of a foreign language, the system could treat the input as absolutely ill-formed. A sophisticated RHS could paraphrase the false pre­supposition for the student and indicate which word or syntactic construction was used inappropriately. Thus, the tutor could point out mistakes such as "Das Fraeu-lein ist Student", indicating that the student should look up the meaning of "Student" (which applies only to males).

Kaplan (1978) suggests an alternative heuristic for false extensional presuppositions in a data base envi­ronment. One can reformulate it as a meta-rule whose LHS would check that the query had requested a set as a response and that the set was empty. The RHS would compute queries corresponding to subsets that the original query presupposed would have a non­empty extension. The RHS would paraphrase the most general such query with an empty response set, report­ing to the user that the system knew of no such enti­ties.

We implemented a grammatical meta-rule processor first for an ATN interpreter and more recently for an ATN compiler (Burton and Brown 1977). Our experi­ments have used RUS (Bobrow 1978), a broad-coverage grammar of English with calls to a semantic component to block anomalous interpretations pro­posed by the grammar.

Design and implementation of a meta-rule proc­essor for violation of semantic constraints is currently underway in two different semantic interpreters. In one, case constraints are expressed as sets of logical formulas; in the other, KL-ONE is used to encode case frames (Sondheimer et al. 1984).

Four design issues are considered in the following sections.

There are a number of points of caution. It should be clear that relaxation does not necessarily guarantee understanding. After all, relaxing any arc to (TST X T ...) will accept any word syntactically; yet that is no guarantee that the word will be understood. Relaxing constraints introduces additional potential for confu­sion.

What one classifies as "absolutely ill-formed" is clearly open to dispute, as Ross (1979) points out. Therefore, the system may classify something as ill-formed, ranking it behind other interpretations, even though the user views it as well-formed. We suspect that categorizing almost any particular constraint as normative could be the basis for argument. The crite­ria for deciding whether a constraint should be includ­ed in the normative system should include at least the following:

a) whether a native speaker would edit inputs that violate it,

b) whether violating the constraint can yield useful inferences,

c) whether examples exist in which the constraint carries meaning,

d) whether the constraint, if classified as normative, trims the search space, and

e) whether a processing strategy for the constraint can be stated more easily as a modification of norma­tive processing, as in the case of conjunction (Woods 1973) or the case of contextual ellipsis in the data base environment ( and Sond­heimer, 1982).

Thus far we have considered only constraints that are associated with a single point in the processing, such as relaxing a single case frame or relaxing a single ATN arc. Obviously, this need not be the case if, for instance, word or phrase order is permuted. At pres­ent, we have no general way of dealing with such problems.

The problems of processing ill-formed input require several substantial research efforts. One is collecting additional corpora to determine patterns of errors and their frequency of occurrence. This is particularly important for two reasons. First, the more detail un­covered on patterns of error, the tighter the meta-rules for relaxing constraints. In our experience, the effort to make relaxation procedures as constrained and ac­curate as warranted by the patterns of occurrence is highly worthwhile, not only in trimming the search space, but also in eliminating senseless interpretations. Second, the patterns of ill-formedness will depend on the user community and the modality of input. For instance, non-native speakers of a language make dif­ferent errors than native speakers. Typed input has a predominance of typographical/spelling errors; spo­ken input may have more restarted utterances.

As a correlate to the need for more corpora of ill-formed natural language, there is an obvious need to define highly specific heuristics (as meta-rules) to diagnose and recover from each type of ill-formedness. Some of the heuristics should involve clarification dialogue, another area for research.

There are many possible responses given a diag­nosed problem. Consider a simple problem: violation of selection restrictions. In German, the verb "fressen" presupposes that the one eating is an animal. To an input such as "Dieser Mann frisst oft", several recovery strategies could apply:

a) The selection restriction could be ignored.

b) The selection restriction could be generalized for future use.

c) The system could conclude that an error has occur­red, as in the aforementioned language learning environment.

"This man eats often."

d) The system could engage in clarification dialog to determine whether the user intended to use that word.

e) The system could assume the user believes that the man referred to eats like an animal.

The conditions for selecting a strategy need to be studied. An explicit model of the user is needed for deciding the intent of the user and for appropriate recovery from ill-formedness.

Learning the idiosyncrasies of particular users and automatic extension of the system (based on detecting relatively ill-formed input) is very challenging. Some initial steps in this direction have been taken in Car­bonell (1979), but there is much to be done. A signif­icant aspect of the learning problem in this environ­ment is the substantial uncertainty about whether the system has the intended interpretation, and the effect on both the functional and time performance of the system as the abnormal is viewed as more normal (and the search space correspondingly grows).

For syntactic ill-formedness, pure bottom-up pars­ing is intuitively very appealing, since one has descrip­tions of what is present both to the left and the right of the problem(s). The EPISTLE project (Jensen and Heidorn 1983) is employing bottom-up parsing. The advantage of employing top-down strategies, including left-corner parsing strategies, is the strong expecta­tions available when a configuration blocks. Conse­quently, many relaxation strategies and systems in the literature (for example, Hendrix, et al. 1978; Kwasny and Sondheimer 1981; Weischedel and Sondheimer 1982) have been proposed and implemented in that framework. Use of bottom-up strategies offers inter­esting new classes of relaxation, such as rearranging constituents for ordering problems. It is not obvious how the combinatorics of bottom-up strategies will compare to those of top-down strategies. However, developing relaxation techniques for bottom-up proc­essing and extensive empirical studies comparing them to top-down are certainly needed.

One of the most critical problems is control. The need to relax the very rules that constrain the search for an interpretation is like opening Pandora's box. This affects not only the time required to understand an ill-formed input, but also ambiguity through the additional alternatives the system is prepared to ac­cept. There are several aspects to controlling this search. First, the well-formedness constraints should reflect strictly what is normative. Second, the relaxa­tion rules should be made as tight as warranted by patterns of ill-formedness in language use. Third, a partial order on the relaxations should be established. Fourth, not only syntactic constraints and selection restrictions should be used (as in our system) but also pragmatic information to suggest the most promising alternatives. We have begun research on how to use pragmatic knowledge in an information-seeking environment for this purpose; see Carberry (1983, 1984) and Ramshaw and Weischedel (1984). In the environ­ment of messages reporting events, Granger (1983) reports on using expectations based on stereotypical events for this purpose. Extensive empirical studies regarding effective control of the search space are needed.

The acid test for a framework, relaxation heuristics, and control strategies is not relaxing simple tests like subject-verb agreement or diagnosing obvious prob­lems like a word not in the dictionary. Rather the acid test is a wide spectrum of problems, including exam­ples like misspellings/typographical errors that result in a known word, because in this type of example, all of the local evidence can indicate that the incorrect word is perfectly correct. Trawick (1983) has initiat­ed work on such misspelling problems.

Ill-formed input cannot be ignored by natural language processing systems. This paper has suggested a uni­form framework for processing ill-formed input in the hope of providing a basis for standardizing work on ill-formedness.

Our framework has several advantages:

a) Well-formed interpretations are always preferred.

b) Ill-formedness processing is explicitly related to the well-formedness rules.

c) Only the constraint that seems to be violated is relaxed; all other well-formedness constraints are still effective for eliminating senseless interpreta­tions and trimming search.

d) Deviance notes record the aspect that deviates from well-formedness, thus allowing pragmatic inferences by later processing.

e) Though our approach is uniform, it permits encod­ing as much specific knowledge into the diagnosis and recovery procedure as one desires for highly specialized cases.

f) Though this paper has drawn most of its examples from ATN grammars and from case frame process­ing, as argued in Section 3.3., the framework is not dependent on a particular model of language proc­essing.

g) The framework should be applicable to lexical, syntactic, semantic, and pragmatic constraints.

The authors appreciate the helpful suggestions of Sudhir Advani, Robert J. Bobrow, Madeleine Bates, Sandra Carberry, Sheila Coyazo, Giorgio Ingargiola, Stan C. Kwasny, William Mann, William Mark, Geoff Pullum, Lance Ramshaw, Amir Razi, Richard L. Wex-elblat, and William A. Woods in discussions about the research.

Much of the coding for the implementation de­scribed was done by Amir Razi.