SemEval-2010 Task 12, Parser Evaluation using Textual Entailment (PETE) (Yuret et al., 2010), was designed as a new, formalism-independent type of parser evaluation scheme. The task is broadly Recognizing Textual Entailment (RTE), but unlike typical RTE tasks, its intention is to fo­cus on purely syntactic entailments, assuming no background knowledge or reasoning ability. For example, given a text (T) The man with the hat was tired., the hypothesis (H) The man was tired. is entailed, but The hat was tired, is not. A cor­rect decision on whether H is entailed can be used as a diagnostic for the parser's analysis of (some aspect of) T. By requiring only a binary decision on the entailment, instead of a full syntactic anal­ysis, a parser can be evaluated while its underlying formalism remains a "black box".

Our system had two components: a parser, and an entailment system which decided whether T en­tails H based on the parser's output. We distin­guish two types of evaluation. Task evaluation, i.e. the official task scoring, indicates whether the entailment decisions - made by the parser and en­tailment system together - tally with the gold stan­dard dataset. Entailment system evaluation, on the other hand, indicates whether the entailment sys­tem is an appropriate parser evaluation tool. In the PETE task the parser is not evaluated directly on the dataset, since the entailment system acts as intermediary. Therefore, for PETE to be a vi­able parser evaluation scheme, each parser must be coupled with an entailment system which accu­rately reflects the parser's analysis of the data.

We used the c&c parser (Clark and Curran, 2007), which can produce output in the form of grammatical relations (GRs), i.e. labelled head-dependencies. For example, (nsubj tired man) for the example in Section 1 represents the fact that the NP headed by man is the subject of the predicate headed by tired. We chose to use the Stanford Dependency GR scheme (de Marneffe et al., 2006), but the same approach should work for other schemes (and other parsers producing GRs).

Our entailment system was very simple, and based on the assumption that H is a simplified ver­sion of T (true for this task though not for RTE in general). We parsed both T and H with the c&c parser. Let grs(S) be the GRs the parser produces for a sentence S. In principle, if grs(H) Ç grs(T), then we would consider H an entailment. In prac­tice, a few refinements to this rule are necessary.

We identified three exceptional cases. First, syntactic transformations between T and H may change GR labels. The most common transforma­tion in this dataset was passivization, meaning that a direct object in T could be a passive subject in H.

Second, H could contain tokens not present in T. Auxiliary verbs were introduced by passivization. Pronouns such as somebody and something were introduced into some H sentences to indicate an NP or other phrase not targeted for evaluation. De­terminers were sometimes introduced or changed, e.g. prices to the prices. Expletive subjects were also sometimes introduced.

A total of 19 systems were submitted. The base­line score for "always YES" was 51.8% accuracy. Our system achieved 72.4% accuracy, which was the highest score among the submitted systems. Table 1 shows the results for our system, as well as SCHWA (University of Sydney), also based on the c&c parser and the next-highest scorer (see Section 6 for a comparison), and the median and lowest scores. The parser found an analysis for

!http://www.eis.upenn.edu/~treebank/ tokenizer.sed.

Tokenize T and H with PTB tokenizer

Parse T and H with C&C parser

Lowercase and lemmatize all tokens

Discard any GR in grs(H) containing a token not in T

YES if core(H) Ç core(T) and grs(H) n grs(T) / 0, NO otherwise

Figure 1 : Full pipeline for parser and entailment system. core(S): the set of core (subject and ob­ject) GRs in grs(S).

99.0% of T sentences and 99.7% of H sentences in the test data.

Table 2 shows the results for our system on the de­velopment data (66 sentences). The parser found an analysis for 100% of sentences and the overall accuracy was 66.7%. In the majority of cases the parser and entailment system worked together to find the correct answer as expected. For example, for Trading in AMR shares was suspended shortly after 3 p.m. EDT Friday and didn't resume. => Trading didn 't resume., the parser produced three GRs for H (tokens are shown lemmatized and low­ercase): (nsubj resume trading), (neg do n't), and (aux resume do). All of these were also in grs(T), and the correct YES decision was made. For Moreland sat brood­ing for a full minute, during which I made each of us a new drink. => Minute is made., the parser produced two GRs for H. One, (auxpass make be) , was ignored because the passive auxiliary be is not in T. The second, pas­sive subject GR (nsubjpass make minute) was equated with a direct object (dobj make minute). This GR was not in grs(T), so the cor­rect NO decision was made.

In some cases a correct YES answer was reached via arguably insufficient positive evi­dence. For He would wake up in the middle of the night and fret about it. => He would wake up., the parser produces incorrect analyses for the VP would wake up for both T and H. However, these GRs are ignored since they are non-core (not sub­ject or object), and a YES decision is based on the single GR match (nsubj would he). This is not entirely a lucky guess, since the entailment system has correctly ignored the odd analyses of would wake up and focused on the role of he as the subject of the sentence. However, especially since the target content word pair was (he, wake), more positive evidence would be desirable. Of the 22 correct YES decisions, only two were truly lucky guesses in that the single match was a determiner; others had at least one core match.

Table 3 shows the breakdown of errors. The largest category was false negatives due to un­bounded dependencies not recovered by the parser, for example It required an energy he no longer possessed to be satirical about his father. => Somebody no longer possessed the energy.. Here the parser fails to recover the direct object re­lation between possess and energy in T. It is known that parsers have difficulty with unbounded depen­dencies (Rimell et al., 2009, from which the un­bounded examples in this dataset were obtained), so this result is not surprising.

The next category was other parser errors. This is a miscellaneous category including e.g. errors on coordination, parenthetical elements, identify­ing the head of a clausal subject, and one due to the POS tagger. For example, for Then at least he would have a place to hang his tools and some­thing to work on. => He would have something to work on., the parser incorrectly coordinated tools and something for T. As a result (dobj have something) was in grs(H) but not grs(T), yield­ing an incorrect NO.

Four errors were due to the entailment system rather than the parser; these will be dicsussed in Section 5. We also identified one sentence where the gold standard entailment appears to rely on extra-syntactic information, or at least informa­tion that is difficult for a parser to recover. This is Index-arbitrage trading is "something we want to watch closely," an official at London's Stock Ex­change said. => We want to watch index-arbitrage trading. Recovering the entailment would require resolving the reference of something, arguably the role of a semantic rather than syntactic module.

We now consider whether our entailment system was an appropriate tool for evaluating the c&c parser on the PETE dataset. It is easy to imag­ine a poor entailment system that makes incorrect guesses in spite of good parser output, or con­versely one that uses additional reasoning to sup­plement the parser's analysis. To be an appropri­ate parser evaluation tool, the entailment system must decide whether the information in H is also contained in the parse of T, without "introducing" or "correcting" any errors.

Assuming our GR-based approach is valid, then given gold-standard GRs for T and H, we expect an appropriate entailment system to result in 100% accuracy on the task evaluation. To perform this oracle experiment we annotated the development data with gold-standard GRs. Using our entailment system with the gold GRs we achieved 90.9% task accuracy. Six incorrect entailment decisions were made, of which one was on the arguably extra-syntactic entailment discussed in Section 4.

Three errors were due to transformations be­tween T and H which changed the GR label or head. For example, consider Occasionally, the children find steamed, whole-wheat grains for ce­real which they call "buckshot". => Grains are steamed.. In T, steamed is a prenominal adjective, with grains as its head; while in H, it is a passive, with grains as its subject. The entailment system did not account for this transformation, although in principle it could have. The other two errors occurred because GRs involving a non-core rela­tion or a pronoun introduced in H, both of which our system ignored, were crucial for the correct entailment decision.

Table 3 shows that with automatically-generated GRs, four errors on the task evaluation were attributable to the entailment system. Three of these were also found in the oracle experiment. The fourth resulted from a POS change between T and H for There was the revolution in Tibet which we pretended did not exist. => The pretended did not exist.. The crucial GR was (nsubj exist pretended) in grs(H), but the entailment system ignored it because the lemmatizer did not give pretend as the lemma for pretended as a noun. This type of error might be prevented by answering NO if the POS of any word changes between T and H, but the implementation is non-trivial since word indices may also change. There were eight POS changes in the development data, most of which did not result in errors. We also observed two cases where the entailment system "corrected" parser errors, yielding a correct entailment decision despite the parser's incorrect analysis of T. When compared with a manual analysis of whether T entailed H based on automatically-generated GRs, the entailment system achieved 89.4% overall accuracy.

We achieved a successful result on the PETE task using a state-of-the-art parser and a simple entail­ment system, which tested syntactic entailments by comparing the GRs produced by the parser for T and H. We also showed that our entailment sys­tem had accuracy of approximately 90% as a tool for evaluating the C&C parser (or potentially any parser producing GR-style output) on the PETE development data. This latter result is perhaps even more important than the task score since it suggests that PETE is worth pursuing as a viable approach to parser evaluation.

The second-highest scoring system, SCHWA (University of Sydney), was also based on the c&c parser and used a similar approach (though using CCG dependency output rather than GRs). It achieved almost identical task accuracy to the Cambridge system, but interestingly with higher accuracy on YES entailments, while our system was more accurate on NO entailments (Table 1). We attribute this difference to the decision crite­ria: both systems required at least one matching relation between T and H for a YES answer; but we additionally answered NO if any core GR in grs(H) was not in grs(T). This difference shows that a GR-based entailment system can be tuned to favour precision or recall.

Finally, we note that although this was a sim­ple entailment system with some dataset-specific characteristics - such as a focus on subject and object relations rather than, say, PP-attachment -these aspects should be amenable to customization or generalization for other related tasks.

The authors were supported by EPSRC grant EP/E035698/1. We thank Matthew Honnibal for his help in producing the gold-standard GRs.