In recent years, a number of annotation schemes that encode semantic information have been de­veloped and used to produce data sets for training machine learning algorithms. Semantic markup schemes that have focused on annotating entity types and, more generally, word senses, have been extended to include semantic relationships between sentence elements, such as the seman­tic role (or label) assigned to the argument by the predicate (Palmer et al., 2005; Ruppenhofer et al., 2006; Kipper, 2005; Burchardt et al., 2006; Subi-rats, 2004).

In this task, we take this one step further and attempt to capture the "compositional history" of the argument selection relative to the predicate. In particular, this task attempts to identify the oper­ations of type adjustment induced by a predicate over its arguments when they do not match its se-lectional properties. The task is defined as fol­lows: for each argument of a predicate, identify whether the entity in that argument position satis­fies the type expected by the predicate. If not, then identify how the entity in that position satisfies the typing expected by the predicate; that is, identify the source and target types in a type-shifting or co­ercion operation.

Consider the example below, where the verb re­port normally selects for a human in subject po­sition, as in (la). Notice, however, that through a métonymie interpretation, this constraint can be violated, as demonstrated in (lb).

(1) a. John reported in late from Washington, b. Washington reported in late.

Neither the surface annotation of entity extents and types nor assigning semantic roles associated with the predicate would reflect in this case a cru­cial point: namely, that in order for the typing requirements of the predicate to be satisfied, a type coercion or a metonymy (Hobbs et al., 1993; Pustejovsky, 1991; Nunberg, 1979; Egg, 2005) has taken place.

The SemEval Metonymy task (Markert and Nis-sim, 2007) was a good attempt to annotate such métonymie relations over a larger data set. This task involved two types with their métonymie variants: categories-for-locations (e.g., place-for-people) and categories-for-organizations (e.g., organization-for-members). One of the limitations of this approach, however, is that while appropri­ate for these specialized metonymy relations, the annotation specification and resulting corpus are not an informative guide for extending the annota­tion of argument selection more broadly.

In fact, the metonymy example in (1) is an in­stance of a much more pervasive phenomenon of type shifting and coercion in argument selection. For example, in (2) below, the sense annotation for the verb enjoy should arguably assign similar values to both (2a) and (2b).

Before introducing the specifics of the argument selection and coercion task, we will briefly review our assumptions regarding the role of annotation in computational linguistic systems.

We assume that the features we use for encoding a specific linguistic phenomenon are rich enough to capture the desired behavior. These linguistic descriptions are typically distilled from extensive theoretical modeling of the phenomenon. The de­scriptions in turn form the basis for the annota­tion values of the specification language, which are themselves the features used in a development cycle for training and testing a labeling algorithm over a text. Finally, based on an analysis and eval­uation of the performance of a system, the model of the phenomenon may be revised.

We call this cycle of development the matter methodology (Fig. 1):

Model: Structural descriptions provide theoretically in­formed attributes derived from empirical observations over the data;

Annotate: Annotation scheme assumes a feature set that en­codes specific structural descriptions and properties of the input data;

Train: Algorithm is trained over a corpus annotated with the target feature set; Test: Algorithm is tested against held-out data; Evaluate: Standardized evaluation of results;

Revise: Revisit the model, annotation specification, or algo­rithm, in order to make the annotation more robust and reliable.

Some of the current and completed annotation ef­forts that have undergone such a development cy­cle include PropBank (Palmer et al., 2005), Nom-Bank (Meyers et al., 2004), and TimeBank (Puste­jovsky et al., 2005).

The argument selection and coercion (asc) task involves identifying the selectional mechanism used by the predicate over a particular argument.For the purposes of this task, the possible relations between the predicate and a given argument are re­stricted to selection and coercion. In selection, the argument NP satisfies the typing requirements of the predicate, as in (3):

(3) a. The spokesman denied the statement (proposi-

tion).

b. The child threw the stone (physical object).

c. The audience didn't believe the rumor (proposi­tion).

Coercion occurs when a type-shifting operation must be performed on the complement NP in order to satisfy selectional requirements of the predicate, as in (4). Note that coercion operations may apply to any argument position in a sentence, including the subject, as seen in (4b). Coercion can also be seen as an object of a proposition, as in (4c).

(4) a. The president denied the attack (event —> propo-

sition).

b. The White House (location —> human) denied this statement.

c. The Boston office called with an update (event —>

info).

In order to determine whether type-shifting has taken place, the classification task must then in­volve (1) identifying the verb sense and the asso­ciated syntactic frame, (2) identifying selectional requirements imposed by that verb sense on the target argument, and (3) identifying the semantic type of the target argument.

We prepared the data for this task in two phases: the data set construction phase and the annotation phase (see Fig. 2). The first phase consisted of (1) selecting the target verbs to be annotated and compiling a sense inventory for each target, and (2) data extraction and preprocessing. The pre­pared data was then loaded into the annotation in­terface. During the annotation phase, the annota­tion judgments were entered into the database, and an adjudicator resolved disagreements. The result­ing database was then exported in an XML format.

a. location —> human (accusare, annunciare) b. artifact —> human (annunciare, avvisare) c. event —> location (arrivare, raggiungere) d. artifact —> event (cominciare, completare) e. event —> document (leggere, divorare) f. human —> document (leggere, divorare) g. event —> sound (ascoltare, echeggiare) h. artifact —> sound (ascoltare, echeggiare)

7. The Italian training data contained 1466 in­stances, 381 of which are coercions; the test data had 1463 instances, with 384 coercions.

The test and training data were provided in XML. The relation between the predicate (viewed as a function) and its argument were represented by composition link elements (CompLink), as shown below. The test data differed from the train­ing data in the omission of CompLink elements.

In case of coercion, there is a mismatch between the source and the target types, and both types need to be identified; e.g., The State Department repeatedly denied the attack:

The State Department repeatedly <SELECTOR sid="sl">denied</SELECTOR> the <TARGET id="t1">attack</TARGET>. <CompLink cid="cidl" compType="COERCION" selector_id="si" relatedToTarget="tl" sourceType="EVENT" targetType="PROPOSITION"/>

When the compositional operation is selection, the source and target types must match; e.g., The State Department repeatedly denied the statement:

The State Department repeatedly <SELECTOR sid="s2">denied</SELECTOR> the <TARGET id="t2">statement</TARGET>. <CompLink cid="cid2" compType="SELECTION" selector_id="s2" relatedToTarget="t2" sourceType="PROPOSITION" targetType="PROPOSITION"/>

We received only a single submission for the asc task. The UTDMet system was an SVM-based system with features derived from two main sources: a PageRank-style algorithm over Word-Net hypernyms used to define semantic classes, and statistics from a PropBank-style parse of some 8 million documents from the English Gigaword corpus. The results, shown in Table 2, were computed from confusion matrices constructed for each of four classification tasks for the 1039 link instances in the English test data: determination of argument selection or coercion, identification of the argument source type, identification of the ar­gument target type, and the joint identification of the source/target type pair.

Clearly, the UTDMet system did quite well at this task. The one immediately noticeable outlier is the macro-averaged precision for the joint type, which reflects a small number of miscategoriza-tions of rare types. For example, eliminating the single miscategorized artifact-location link in the submitted test data bumps this score up to a respectable 94%. This large discrepancy can ex­plained by the lack of any coercions with those types in the gold-standard data.

In the absence of any other submissions, it is difficult to provide a point of comparison for this performance. However, we can provide a base­line by taking each link to be a selection whose source and target types are the most common type (event for the gold-standard English data). This yields micro-averaged precision scores of 69% for selection vs. coercion, 33% for source type iden­tification, 37% for the target type identification, and 22% for the joint type.

The performance of the UTDMet system sug­gests that most of the type coercions were identifi­able based largely on examination of lexical clues associated with selection contexts. This is in fact to be expected for the type coercions that were the focus of the English data set. It will be interesting to see how systems perform on the Italian data set and an expanded corpus for English and Italian, where more subtle and complex type exploitations and manipulations are at play. These will hope­fully be explored in future competitions.

In this paper, we have described the Argument Se­lection and Coercion task for SemEval-2010. This task involves identifying the relation between a predicate and its argument as one that encodes the compositional history of the selection process. This allows us to distinguish surface forms that di­rectly satisfy the selectional (type) requirements of a predicate from those that are coerced in context. We described some details of a specification lan­guage for selection, the annotation task using this specification to identify argument selection behav­ior, and the preparation of the data for the task. Finally, we analyzed the results of the task sub­missions.