The SemEval 2010 Parser Evaluation using Tex­tual Entailments (pete) task attempts to address the long-standing problems in parser evaluation caused by the diversity of syntactic formalisms and analyses in use. The task investigates the feasibility of a minimalist extrinsic evaluation -that of detecting textual entailment between a truth sentence and a hypothesis sentence. It is extrin­sic in the sense that it evaluates parsers on a task, rather than a direct comparison of their output against some gold standard. However, it requires only minimal task-specific logic, and the proposed entailments are designed to be inferrable based on syntactic information alone.

Our system used the c&c parser (Clark and Curran, 2007a), which uses the Combinatory Cat-egorial Grammar formalism (ccg, Steedman, 2000). We used the CCGbank-style dependency output of the parser (Hockenmaier and Steedman, 2007), which is a directed graph of head-child re­lations labelled with the head's lexical category and the argument slot filled by the child.

We divided the dependency graphs of the truth and hypothesis sentences into predicates that con­sisted of a head word and its immediate children. For instance, the parser's analysis of the sentence Totals include only vehicle sales reported in pe­riod might produce predicates like include(Totals, sales), only (include), and reported(sales). If at least one such predicate matches in the two parses, we predict entailment. We consider a single pred­icate match sufficient for entailment because the lexical categories and slots that constitute our de­pendency labels are often different in the hypothe­sis sentence due to the generation process used in the task.

The single predicate heuristic gives us an over­all accuracy of 70% on the test set. Our precision and recall over the test set was 68% and 80% re­spectively giving an F-score of 74%.

To investigate how many of the errors were due to parse failures, and how many were failures of our entailment recognition process, we manually annotated the 66 development truth sentences with gold standard ccg derivations. This established an upper bound of 87% F-score for our approach.

This upper bound suggests that there is still work to be done before the system allows trans­parent evaluation of the parser. However, cross-framework parser evaluation is a difficult problem: previous attempts to evaluate the c&c parser on grammatical relations (Clark and Curran, 2007b) and Penn Treebank-trees (Clark and Curran, 2009) have also produced upper bounds between 80 and 90% F-score. Our pete system was much easier to produce than either of these previous attempts at cross-framework parser evaluation, suggesting that this may be a promising approach to a diffi­cult problem.

Combinatory Categorial Grammar (ccg, Steed­man, 2000) is a lexicalised grammar formalism based on combinatory logic. The grammar is di­rectly encoded in the lexicon in the form of combi­natory categories that govern how each word com­bines with its neighbours. The parsing process de­termines the most likely assignment of categories to words, and finds a sequence of combinators that allows them to form a sentence.

A sample ccg derivation for a sentence from the test set is shown in Figure 1. The category for each word is indicated beneath it. It can be seen that some categories take other categories as ar­guments; each argument slot in a category is num­bered based on the order of application, from latest to earliest. For example:

Figure 2 shows how the argument slots are mapped to dependencies. The first two columns list the predicate words and their categories, while the second two show how each argument slot is filled. For example, in the first row, only has the category (S\NP)\(S\NP), with argument slot 1 filled by include). It is these dependencies that form the basis for our predicates in this task.

Recent work has seen the development of high-performance parsers built on the ccg formalism. Clark and Curran (2007a) demonstrate the use of techniques like adaptive supertagging, parallelisa-tion and a dynamic-programming chart parsing al­gorithm to implement the c&c parser, a highly efficient ccg parser that performs well against parsers built on different formalisms (Rimell et al., 2009). We use this parser for the pete task.

The performance of statistical parsers is largely a function of the quality of the corpora they are trained on. For this task, we used models derived from the CCGbank corpus - a transformation of the Penn Treebank (Marcus et al., 1993) including ccg derivations and dependencies (Hockenmaier, 2003a). It was created to further ccg research by providing a large corpus of appropriately anno­tated data, and has been shown to be suitable for the training of high-performance parsers (Hocken­maier, 2003b; Clark and Curran, 2004).

Our system used the c&c parser to parse the truth and hypothesis sentences. We took the dependen­cies generated by the parser and processed these to generate predicates encoding the canonical form of the head word, its required arguments, and their order. We then attempted to unify the predicates from the hypothesis sentence with the predicates in the truth sentence. A successful unification of predicates a and b occurs when the head words of a and b are identical and their argument slots are also identical. If any predicate from the hypothe­sis sentence unified with a predicate from the truth sentence, our system returned yes, otherwise the system returned no.

We used the 66 sentence development set to tune our approach. While analysing the hypoth­esis sentences, we noticed that many examples re-

Table 1 : Final results over the test set placed nouns from the truth sentence with indefi­nite pronouns such as someone or something (e.g. Someone bought something). In most of these cases the indefinite would not be present in the truth sen­tence at all, so to deal with this we converted in­definite pronouns into wildcard markers that could be matched to any argument. We also incorporated sensitivity to passive sentences by adjusting the ar­gument numbers of dependents.

In its most naive form our system is heavily biased towards excellent recall but poor preci­sion. We evaluated a number of heuristics to prune the predicate space and selected those which im­proved the performance over the development set. Our final system used the part-of-speech tags gen­erated by the parser to remove predicates headed by determiners, prepositions and adjectives. We note that even after predicate pruning our system is still likely to return better recall performance than precision, but this discrepancy was masked in part by the nature of the development set: most hy­potheses are short and so the potential number of predicates after pruning is likely to be small. The final predicates generated by the system for the ex­ample derivation given in Figure 1 after heuristic pruning are:

only(include) reported(sales) include(totals, sales)

We report results over the 301 sentence test set in Table 1. Our overall accuracy was 70%, and per­formance over yes entailments was roughly 20% higher than accuracy over no entailments. This bias towards yes entailments is a reflection of our single match heuristic that only required one pred­icate match before answering yes. Our system performed nearly 20% better than the baseline sys­tem (all YES responses) and placed second overall in the task evaluation.

Table 2 shows our results over the development corpus. The 17% drop in accuracy and 8% drop in F-score between the development data and the test data suggests that our heuristics may have over-fitted to the limited development data. More so­phisticated heuristics over a larger corpus would be useful for further fine-tuning our system.

We constructed a system to evaluate the c&c parser using textual entailments. We converted the parser output into a set of predicate structures and used these to establish the presence of entailment. Our system achieved an overall accuracy of 79% on the development set and 70% over the test set. The gap between our development and test accu­racies suggests our heuristics may have been over-fitted to the development data.

Our investigation using gold-standard depen­dencies established an upper bound of 87% on the development set for our approach to the task. While this is not ideal, we note that previous ef­forts at cross-parser evaluation have shown that it is a difficult problem (Clark and Curran (2007b) and Clark and Curran (2009)). We conclue that the concept of a minimal extrinsic evaluation put forward in this task is a promising avenue for formalism-independent parser comparison.