We have identified 644 examples of VPE from the Brown Corpus and the Wall Street Journal Corpus of the Perm Treebank. Since the parsing schemes used in the Penn Treebank do not explicitly label VPE occurrences, it is difficult to ensure that all oc­currences of VPE in the Treebank are found. However, this data set is the first large set of ellipsis examples we are aware of, and it provides a solid empirical foundation not only for the current study, but for future research on ellipsis.

We used several techniques to identify ellipsis occurrences in the Treebank, all involving the tree pattern-matching utility tgrep} In the Wall Street Journal Corpus, the -NONE- category is used to represent a variety of empty expressions, including VPE. We searched for the following pattern:

which resulted in a set of 260 examples from the Wall Street Journal corpus.

1 The utility tgrep is written by Rich Pito of the University of Pennsylvania, and is distributed together with the Penn Treebank CD-ROM.

All the examples of VPE were coded for the correct antecedent by two human coders. We performed some comparisons of coder responses with one another, based on our three success criteria. On a sample of 162 examples, the results were as shown in Table 3.

In this sample, there was only one example in which coder agreement failed ac­cording to the Head Overlap criterion:

(1) Gold still acts as a haven when uncertainty prevails in the financial markets as it did yesterday.

2 This percentage was computed using the parsed version of both the Brown Corpus and Wall Street Journal Corpus. The sample contained 155,000 words, out of a total of approximately 4,799,845 in the Treebank.

3 The numbers for the missed examples reflect a post hoc analysis of the program output, rather than comparison with coder files.

To test the performance of the system, we first obtained a coded file, which indicates a human coder's preferred antecedent for each example. Then we compared the output of the system with the coder's selections.

As mentioned in Section 1, we define three criteria for success:

1. Head Overlap: either the head verb of the system choice is contained in the coder choice, or the head verb of the coder choice is contained in the system choice.

To illustrate these criteria, we give three examples, one for each success criterion. Note that the success criteria are increasingly strict—if an example satisfies Exact Match, it will also satisfy the other two criteria, and if an example satisfies Head Match, it will also satisfy Head Overlap.

Example: Head Overlap

(14) In July, Par and a 60% owned unit agreed to plead guilty in that inquiry, as did another former Par official.

System output: plead guilty in that inquiry

Coder selection: agreed to plead guilty in that inquiry

According to Head Overlap, the system choice is correct, since its head verb, plead, is contained in the coder selection. This would not be considered correct according to Head Match, since the head of the coder selection is agreed.

Example: Head Match (15) The question is, if group conflicts still exist, as undeniably they do,

System output: exist Coder selection: still exist

Here, both the system output and the coder selection have the head verb exist, but there is not an exact, word-for-word match.

Example: Exact Match

(16) It is difficult if not impossible for anyone who has not pored over the thousands of pages of court pleadings and transcripts to have a worthwhile opinion on the underlying merits of the controversy. Certainly I do not.

After identifying 644 examples of VPE in the Treebank, we reserved 96 randomly selected examples from the Wall Street Journal corpus for a blind test. In Table 4, we give results for the blind test and for the entire Penn Treebank, and we report separate figures on the Brown Corpus and Wall Street Journal Corpus. As a baseline, we also report results (Table 5) on a simple recency-based approach: the most recent VP is always chosen. No Preference Factors or filters are applied.

The difference between the VPE-RES performance and the baseline is statistically significant by all three criteria, based on a x analysis, p < .001.

In Tables 6, 7, and 8, we present results on each major subpart of the program. For this evaluation, we used the Exact Match criterion. We evaluated subparts in three ways: first, we began with the baseline (recency) approach, and activated a single additional component, to see how the system performance changed based on that component. Second, we began with the complete system, and deactivated a single component. Finally, we evaluated system components in an incremental fashion, beginning with Post-Filter, then activating Syntactic Filter with Post-Filter still activated, etc. The Com­posite Factor is a combination of Post-Filter, Syntactic Filter, and Clause-Rel.

17 Since the blind test examples are all taken from the Wall Street Journal corpus, it is most appropriate to compare the blind test results directly to the results on the Wall Street Journal Corpus. Not surprisingly, the blind test results are slightly lower than the results on the complete Wall Street Journal Corpus, since this contains the examples that functioned as training data.

The most important system component is the Composite Factor, which is a combination of the Syntactic Filter, the Post-Filter, and Clause-Rel. The contribution of Clause-Rel is not evident individually; if it is the only factor activated together with Recency-Only, performance in the complete corpus actually declines from 29.2% to 28.1%. However, this is because Clause-Rel requires the Syntactic Filter to make a contribution. This can be observed from the fact that Composite performs better than its individual com­ponents. Also, when Clause-Rel is the deactivated factor, performance declines from 75.9% to 72.8%. The Parallelism Preference Factors, Aux-Match and Parallel-Match, also make an important contribution: when they are activated in the incremental anal­ysis, there are 22 additional correct selections in the complete corpus, an improvement of 3.4%.

Many of the errors occurring under the Exact Match criterion involve alternatives that are virtually identical in meaning, as in the following example:

(17) Stephen Vincent Benet's John Brown's Body [vp comes immediately to mind] [PP in this connection], as does John Steinbeck's The Grapes Of Wrath and Carl Sandburg's The People, Yes.

Here, VPE-RES selected comes immediately to mind, since the PP in this connection is parsed as a sister to the VP. One coder selected comes immediately to mind in this connection, while the other coder made the same selection as VPE-RES. It is difficult to see any difference in meaning between the two choices.

Because of examples like this, we believe Head Overlap or Head Match are prefer­able criteria for success. Even with the Head Match criterion, there are errors that involve very subtle differences, such as the following example:

(18) We were there at a moment when the situation in Laos threatened to ignite another war among the world's giants. Even if it did not, how would this little world of gentle people cope with its new reality of grenades and submachine guns?

The coder selected ignite another war among the world's giants, while VPE-RES selected threatened to ignite another war among the world's giants.

Some errors result from problems with the Syntactic Filter. The following example illustrates a case of antecedent containment that is not recognized by the filter as currently formulated.

(19) All the generals who held important commands in World War 2, did not write books. It only seems as if they did.

The VPE-RES system incorrectly selects seems as the antecedent, because it does not recognize that the VP headed by seems improperly contains the VPE.