Consider the following three instructions:

(la) Go into the other room to get the urn of coffee.

(lb) Before you pick it up, be sure to unplug it.

(lc) When you bring it back here, carry it carefully with both hands.

Let's consider (la). To understand this instruction, an agent must find the connection between the two actions a—go into the other room, and ß—get the urn of coffee. The infinitival to alerts the agent to the fact that a con­tributes to achieving ß. General knowledge about phys­ically getling objects requires that the agent move to the place where the object is located; therefore, the agent will infer that the (most direct) connection between these ac­tions has go into the other room fulfilling this requirement However, this is not enough. An assumption needs to be made for such connection to go through, namely, that the urn is in the other room.

This example shows that to make sense of instructions, an agent must engage in the active computation of the action(s) to be executed, and cannot simply "extract" all such information from the input. This differentiates our work from others', as we will discuss shortly.

Another important point that arises from (la) is that the relation contributes holding between a, described in the matrix clause, and ß, described in the purpose clause, can be specilied either as generation or enablement, as a study of naturally occurring purpose clauses [Di 92a] shows.

Generation was introduced by [Gol70], Informally, if ac­tion a generates action ß, we can say that ß is exe­cuted by executing a. An example is Turning on the light by flipping the switch.

Enablement. Following [P0I86] and [Bal90], action a en­ables action ß if and only if an occurrence of a brings about conditions necessary for the subsequent perfor­mance of ß. In Unscrew the protective plate to expose the box, "unscrew the protective plate" enables "tak­ing the plate off' which generates "exposing the box".

In [P0I86], it is shown that these two relations arc nec­essary to model action descriptions conveyed by Natural Language. We would like to add one further observation: such relations allow us to draw conclusions about action execution too. This is quite useful since we do have to execute (ie„ animate) the input instructions, as our work is taking place in the context of the Animation from Natural Language (AnimNL) project at the University of Pennsyl­vania [WBD*91].

As far as generation is concerned, while two actions are described, only a, the generator, needs to be performed; instead, if a enables ß, after executing a, ß still needs to be executed. In fact, if a enables ß, a has to begin, but not necessarily end, before ß.

The following conclusions can be drawn from the obser­vations in the previous section:

1. NL action descriptions are fairly complex, including modifiers of many different types—see also [WD90]. An action representation formalism must be able to deal with complex descriptions, such as carry it care­fully with both hands; with descriptions at different levels of abstraction, such as go and walk to, or such as cut the square in half and cut the square in half along the diagonal in (2).

2. NL instructions include a wide variety of construc­tions, such as purpose clauses and temporal clauses. Instruction interpretation systems must be able to deal with complex imperatives and with the relations be­tween actions that they express.

3. An instruction interpretation system cannot assume that the descriptions of the actions to be performed are equivalent to the logical forms computed by the parser: such logical forms have to be constrained in various ways, e.g. by computing assumptions, as in (la), or more specific action descriptions, as in (2). Notice that these constraints derive from the interac­tion between the actions to be executed and the goals the agent adopts. It is essential that this interaction is taken into account by such systems.

Work done in the past on understanding instructions has generally concentrated on simple positive commands, and has failed to address some of the desiderata listed above: [VB90] limits the interaction between new and preexist­ing goals to inserting the new goats in the list of goals if their execution does not violate preexisting œnstraints, otherwise they are rejected. [Cha91] proposes a model of instruction interpretation which seems useful at the level of the basic skills an agent is endowed with, but in which there is no internal structure to actions, and no distinction between the agent's actions and goals. [AZC91] instead does assume a rich relation between instructions and pre­existing goal(s). However, instructions arc not continually integrated into the plan the agent is developing; instead they are used as a resource when the stored knowledge about plans cannot be adapted to the situation at hand.

Turning now to our proposal, our approach to these prob­lems includes

1. An action representation formalism based on lack­endoff s Conceptual Structures [Jac90].

2. An action KB that contains simple plans that repre­sent common sense knowledge about actions.

3. A plan graph that represents the structure of the agent's intentions.

We have chosen to use Jackendoff's Conceptual Structures [Jac90] for two reasons. First, as our point of departure is NL, there are the obvious benefits of using a linguistically motivated representational theory, e.g. easing the burden upon the parser to produce such representations [Whi92]. Second, there is significant mileage to be gained from using a decompositional theory of meaning, insofar as the prim­itives effectively capture important generalizations. In Ulis section we introduce the notation and some minor modifi­cations to the theory as presented in [Jac90]. We use Go into the other room as a representative example.

In Jackendoff's theory, an entity may be of ontological type Thing, Place, Path, Event, State, Manner or Property. The conceptual structure for a room is shown in (3a) below:

(3a) [Thing hoom] (3b) [Thing KITCHEN]

Square brackets indicate an entity of type Thing meet­ing the enclosed featural description. Small caps indi­cate atoms in conceptual structure, which serve as links to other systems of representation; for example, the con­ceptual structure for a kitchen (3b) differs from that of a to a body that generates a header. The annotations on the body specify the relations between the subactions; such re­lations include partial temporal ordering, enablement, and possibly others.

From the planning tradition, we retain the notions of qualifiers and effects. Qualifiers are conditions that make an action relevant: for example, unplug x is relevant only if x is plugged.

Notice the importance of using a representation such as Jackendoff's: it helps us capture the common characteris­tics of different actions, e.g. get and carry. The seman­tic representation for carry would also match the generic move-action template, and would add to it a qualification such as

Having such a representation is also useful for comput­ing qualifiers and effects in a systematic way: they can be precompiled from the representation itself. For example, for every action including a component 6 such as we know tliat after 6, j must be at /, therefore we can include this in the effects of the action. Given the further restriction that j cannot be in two places at once, we may infer that j cannot \k at / now, and thus precompute the qualifier.

The plan graph represents the structure of the intentions that the agent adopts as a response to the instructions. It keeps track of the goals the agent is pursuing, of the hier­archical relations between the goals and the actions whose execution achieves such goals, and of various relations be­tween the actions. It also helps interpret the instructions that follow. In (t), establishing the initial goal get the urn of coffee provides the context in which the two following instructions have to be interpreted—a similar strategy is adopted for example by [Kau90]. In Fig. 2, we show the complete structure built after interpreting (1).

A node in a plan graph contains the Conceptual Structure representation of an action, augmented with the consequent slate achieved alter the execution of that action. The arcs represent relations between actions; among them, those relevant to our example arc: temporal, such as precedes in Fig. 2; enablement; generation, and its generalization substep, used when a belongs to a sequence of more than one action that generates ß.

BJackendoff suggests something analogous with his inference rules, which have yet to be formalized.

°In Fig. 2 the labels on the nodes are only mnemonics, and do not represent their real contents.

Al: UE(um. lNtfoliitr-rooin])) A2: DEtum, plüggcd-ln)

There may also be assumptions associated with a plan graph. If an assumption is derived from the qualifiers as­sociated with an action, it is associated with the node de­scribing that action—A2 in Fig. 2; if it is derived while inferring a relation between two actions, it is associated with the corresponding arc—Al.

The plan graph is built by an interpretation algorithm that takes as its input the logical form constructed by the parser. The algorithm works by keeping track of the ac­tive nodes, which include the goal currently in focus, and the nodes just added to the tree. The topmost level of the algorithm invokes different procedures, according to the particular syntactic construction at hand - e.g. the con­struction Do a to do ß will trigger the hypothesis that a either generates or enables ß [Di 92b]. These procedures retrieve the plan(s) associated with the goal currently in focus, and then expand such plans in a hierarchical fash­ion.

These procedures embody various inference processes, that can be characterized either as planning—e.g. plan ex-paasion, subgoaling— or as plan inference—e.g. inferring assumptions, inferring the more abstract goal some actions are supposed to achieve. Space doesn't allow us to go into further delails about the algorithm or the inference pro­cesses; rather, in the next section we will give an example of how assumptions are computed.

We will now show how the assumption that the um is to be found in die other room is made while processing (la), Go into the other room to gel the urn of coffee.

The process begins with the following representation constructed by the parser, where the FOR-function (de­rived from the to -phrase) encodes the contributes relation holding between the go-action a and the get-action ß:

Given Die presence of the to phrase, we know that a may be part of a sequence of actions that generate ß. To pursue this hypothesis, we begin by looking up ß in the action KB. ß matches the general move-action shown in Fig. 1 if the object to be moved j is bound to the urn of coffee:

Next we try to match a with some subaction 7 of ß. a matches the first action 71 in ß if we take [AT(j)J and [IN([otuer-room1)] to be the same place. This is tanta­mount to making the following assumption:

Once the instruction is understood in this way, the two actions may be incorporated into the plan graph as shown in Fig. 2.

One should mention that assumption (11) could of course be wrong, say if there were a note in the next room saying ha ha, it's not really in this room but the next.

Notice that even if there is already an urn of coffee in the current room, die instruction Go into the other room to get the urn of coffee is still understood to refer to an urn in the other room. This contrasts sharply with Go into the other room to wash out the urn of coffee, where the most likely urn is the currently visible one. In the current framework, this difference would be captured in the following way. Unlike in the case of the get -action, the #o-action matches titc following subaction of wash-out :

Therefore, assumption (11) will not be derived, permitting die possibility of Ute urn being in the current room.

We have presented an approach to action representation and iastruction interpretation which we feel is more flexible than previously proposed formalisms: it allows us to use terms at different levels of specificity, and to perform the complex inferences that NL instructions require.

Future research includes exploring how to integrate a hierarchical organization of entities, actions and plans with the action KB.

The system is being implemented in Quintus Prolog, with substantial progress having been made in particular on the parser [Whi92], and on the action KB.

This research was supported by the following grants: DARPA no. N00014-90-J4863, ARO no. BAAL CO-89-C-0031, NSF no. IRI 90-16592, and Hen Franklin no. 91S.3078C-1. We would like to thank all the members of the AnimNL group, and in particular Bonnie Webber, Libby Levison and Chris Geib, for very stimulating and helpful discussions.