We first decided on an inventory of semantic rela­tions. Ideally, it should be exhaustive (enable the description of relations between any pair of nomi­nals) and mutually exclusive (each pair of nominals in context should map onto only one relation). The literature, however, suggests that no relation inven­tory satisfies both needs, and, in practice, some trade-off between them must be accepted.

As a pragmatic compromise, we selected nine relations with coverage sufficiently broad to be of general and practical interest. We aimed at avoid­ing semantic overlap as much as possible. We included, however, two groups of strongly related relations (entity-origin / entity-destina­tion and Content-Container / Component-Whole / Member-Collection) to assess mod­els' ability to make such fine-grained distinctions. Our inventory is given below. The first four were also used in SemEval-1 Task 4, but the annotation guidelines have been revised, and thus no complete continuity should be assumed.

Cause-Effect (CE). An event or object leads to an effect. Example: those cancers were caused by radiation exposures

Instrument-Agency (IA). An agent uses an in­strument. Example: phone operator

Product-Producer (PP). A producer causes a product to exist. Example: a factory manu­factures suits

Content-Container (CC). An object is physically stored in a delineated area of space. Example: a bottle full of honey was weighed

Entity-Origin (EO). An entity is coming or is de­rived from an origin (e.g., position or mate­rial). Example: letters from foreign countries

Entity-Destination (ED). An entity is moving to­wards a destination. Example: the boy went to bed

Component-Whole (CW). An object is a com­ponent of a larger whole. Example: my apartment has a large kitchen

Member-Collection (MC). A member forms a nonfunctional part of a collection. Example: there are many trees in the forest

Message-Topic (MT). A message, written or spo­ken, is about a topic. Example: the lecture was about semantics

We defined a set of general annotation guidelines as well as detailed guidelines for each semantic relation. Here, we describe the general guidelines, which delineate the scope of the data to be col­lected and state general principles relevant to the annotation of all relations.

Our objective is to annotate instances of seman­tic relations which are true in the sense of hold­ing in the most plausible truth-conditional inter­pretation of the sentence. This is in the tradition of the Textual Entailment or Information Valida­tion paradigm (Dagan et al., 2009), and in con­trast to "aboutness" annotation such as semantic roles (Carreras and Marquez, 2004) or the BioNLP 2009 task (Kim et al., 2009) where negated rela­tions are also labelled as positive. Similarly, we exclude instances of semantic relations which hold only in speculative or counterfactural scenarios. In practice, this means disallowing annotations within the scope of modals or negations, e.g., "Smoking may/may not have caused cancer in this case."

We accept as relation arguments only noun phrases with common-noun heads. This distin­guishes our task from much work in Information Extraction, which tends to focus on specific classes of named entities and on considerably more fine­grained relations than we do. Named entities are a specific category of nominal expressions best dealt

!The full task guidelines are available at http : / / docs . google.com/View?id=dfhkmm46_0f63mfvf7

with using techniques which do not apply to com­mon nouns. We only mark up the semantic heads of nominals, which usually span a single word, except for lexicalized terms such as science fiction.

We also impose a syntactic locality requirement on example candidates, thus excluding instances where the relation arguments occur in separate sen­tential clauses. Permissible syntactic patterns in­clude simple and relative clauses, compounds, and pre- and post-nominal modification. In addition, we did not annotate examples whose interpretation relied on discourse knowledge, which led to the exclusion of pronouns as arguments. Please see the guidelines for details on other issues, includ­ing noun compounds, aspectual phenomena and temporal relations.

The annotation took place in three rounds. First, we manually collected around 1,200 sentences for each relation through pattern-based Web search. In order to ensure a wide variety of example sentences, we used a substantial number of patterns for each relation, typically between one hundred and several hundred. Importantly, in the first round, the relation itself was not annotated: the goal was merely to collect positive and near-miss candidate instances. A rough aim was to have 90% of candidates which instantiate the target relation ("positive instances").

In the second round, the collected candidates for each relation went to two independent annotators for labeling. Since we have a multi-way classifi­cation task, the annotators used the full inventory of nine relations plus other. The annotation was made easier by the fact that the cases of overlap were largely systematic, arising from general phe­nomena like metaphorical use and situations where more than one relation holds. For example, there is a systematic potential overlap between Content-Container and Entity-Destination depend­ing on whether the situation described in the sen­tence is static or dynamic, e.g., "When I came, the <el>apples</el> were already put in the <e2>basket</e2>." is CC(el, e2), while "Then, the <el>apples</el> were quickly put in the <e2>basket</e2>" is ED(el, e2).

In the third round, the remaining disagreements were resolved, and, if no consensus could be achieved, the examples were removed. Finally, we merged all nine datasets to create a set of 10,717 instances. We released 8,000 for training and kept the rest for testing.

Table 1 shows some statistics about the dataset. The first column (Freq) shows the absolute and rel­ative frequencies of each relation. The second col­umn (Pos) shows that the average share of positive instances was closer to 75% than to 90%, indicating that the patterns catch a substantial amount of "near-miss" cases. However, this effect varies a lot across relations, causing the non-uniform relation distribu­tion in the dataset (first column). After the second round, we also computed inter-annotator agreement (third column, IAA). Inter-annotator agreement was computed on the sentence level, as the per­centage of sentences for which the two annotations were identical. That is, these figures can be inter­preted as exact-match accuracies. We do not report Kappa, since chance agreement on preselected can­didates is difficult to estimate. IAA is between 60% and 95%, again with large relation-dependent variation. Some of the relations were particularly easy to annotate, notably Content-Container, which can be resolved through relatively clear cri­teria, despite the systematic ambiguity mentioned above. Entity-Origin was the hardest relation to annotate. We encountered ontological difficulties in defining both Entity (e.g., in contrast to Effect) and Origin (as opposed to Cause). Our numbers are on average around 10% higher than those re­ported by Girju et al. (2009). This may be a side effect of our data collection method. To gather 1,200 examples in realistic time, we had to seek productive search query patterns, which invited certain homogeneity. For example, many queries for Content-Container centered on "usual sus­pect" such as box or suitcase. Many instances of Member-Collection were collected on the ba­sis of from available lists of collective names.