The field of Computational Linguistics (CL) has both moved towards applications and towards large data sets. These developments call for a rigorous methodology for creating so-called lingware: linguis­tic data such as lexica, grammars, tree-banks, as well as software processing it. Experience from Soft­ware Engineering has shown that the earlier deficien­cies are detected, the less costly their correction is. Rather than being a post-development effort, quality-evaluation must be an integral part of development to make the construction of lingware more efficient (e.g., cf. (EAGLES, 1996) for a general evaluation framework and (Ciravegna et al., 1998) for the ap­plication of a particular software design methodol­ogy to linguistic engineering). This paper presents the adaptation of a particular Software Engineering (SE) method, instrumentation, to Grammar Engi­neering (GE). Instrumentation allows to determine which test item exercises a certain piece of (software or grammar) code.

The paper first describes the use of instrumenta­tion in SE, then discusses possible realizations in uni­fication grammars, and finally presents two classes of applications.

Systematic software testing requires a match be­tween the test subject (module or complete system) and a test suite (collection of test items, i.e., sam­ple input). This match is usually computed as the percentage of code items exercised by the test suite.

Depending on the definition of a code item, vari­ous measures are employed, for example (cf. (Het-zel, 1988) and (EAGLES, 1996, Appendix B) for overviews):

statement coverage percentage of single state­ments exercised

branch coverage percentage of arcs exercised in control flow graph; subsumes statement cover­age

path coverage percentage of paths exercised from start to end in control flow graph; subsumes branch coverage; impractical due to large (often infinite) number of paths

condition coverage percentage of (simple or ag­gregate) conditions evaluated to both true and false (on different test items)

Testsuites are constructed to maximize the tar­geted measure. A test run yields information about the code items not exercised, allowing the improve­ment of the testsuite.

The measures are automatically obtained by in­strumentation: The test subject is extended by code which records the code items exercised during pro­cessing. After processing the testsuite, the records are used to compute the measures.

Measures from SE cannot simply be transferred to unification grammars, because the structure of (im­perative) programs is different from (declarative) grammars. Nevertheless, the structure of a grammar (formalism) allows to define measures very similar to those employed in SE.

constraint coverage is the quotient # constraints exercised # constraint m grammar where a constraint may be either a phrase-structure or an equational constraint, depend­ing on the formalism.

disjunction coverage is the quotient

# disjunctions covered # disjunctions in grammar where a disjunction is considered covered when all its alternative disjuncts have been separately-exercised. It encompasses constraint coverage. Optional constituents and equations have to be treated as a disjunction of the constraint and an empty constraint (cf. Fig.2 for an example).

interaction coverage is the quotient # disjunct combinations exercised # legal disjunct combinations where a disjunct combination is a complete set of choices in the disjunctions which yields a well-formed grammatical structure. As with path coverage, the set of legal disjunct combination typically is infinite due to recur­sion. A solution from SE is to restrict the use of recursive rules to a fixed number of cases, for example not using the rule at all, and using it only once.

The goal of instrumentation is to obtain informa­tion about which test cases exercise which grammar constraints. One way to record this information is to extend the parsing algorithm. Another way is to use the grammar formalism itself to identify the disjuncts. Depending on the expressivity- of the for­malism used, the following possibilities exist:

atomic features Assuming a unique numbering of disjuncts, an annotation of the form DISJUNCT-nn = + can be used for marking. To determine whether a certain disjunct was used in constructing a solution, one only needs to check whether the associated feature occurs (at some level of embedding) in the solution.

set-valued features If set-valued features are available, one can use a set-valued fea­ture DISJUNCTS to collect atomic sym­bols representing one disjunct each: DISJUNCT-nn € DISJUNCTS, which might the collection of exercised disjuncts.

multiset of symbols To recover the number of times a disjunct is used, one needs to leave the unification paradigm, because it is very difficult to count with unification grammars. We have used a special feature of our grammar develop­ment environment: Following the LFG spirit of different projections, it provides a projection for

I 4-6 (f ADJUNCT); }.

DISJUNCT-005 € o*:} }.

symbolic marks, which is formally equivalent to a multiset of symbols associated with the com­plete solution (structural embedding plays no role; see (Frank et al., 1998) for applications). In this way, we can collect from the root node of each solution the set of all disjuncts exercised, together with a usage count.

Consider the LFG grammar rule in Fig.l. Con­straint coverage would require test items such that every category in the VP is exercised; a sequence of V NP PP would suffice for this measure. Disjunction coverage also requires to take the empty disjuncts into account: NP and PP are optional, so that four items are needed to achieve full disjunction cover­age on the phrase structure part of the rule. Due to the disjunction in the PP annotation, two more test items are required to achieve full disjunction cover­age on the complete rule. Fig.2 shows the rule from Fig.l with instrumentation.

Traditionally, a testsuite is used to improve (or maintain) a grammar's quality (in terms of cover­age and overgeneration). Using instrumentation, one may extend this usage by looking for sources of overgeneration (cf. Sec.4.3), and may also improve the quality of the testsuite, in terms of coverage (cf. Sec.4.1) and economy (cf. Sec.4.2).

Complementing other work on testsuite construc­tion (cf. Sec.4.4), I will assume that a grammar is already available, and that a testsuite has to be constructed or extended. While one may argue that grammar and testsuite should be developed in paral­lel, such that the coding of a new grammar disjunct is accompanied by the addition of suitable test cases, and vice versa, this is seldom the case. Apart from the existence of grammars which lack a testsuite, there is the more principled obstacle of the evolu­tion of the grammar, leading to states where previ­ously necessary rules silently loose their usefulness, because their function is taken over by some other rules, structured differently. This is detectable by-instrumentation, as discussed in Sec.4.1.

On the other hand, once there is a testsuite, it has to be used economically, avoiding redundant tests. Sec.4.2 shows that there are different levels of re­dundancy in a testsuite, dependent on the specific grammar used. Reduction of this redundancy can speed up the test activity, and give a clearer picture of the grammar's performance.

If the disjunction coverage of a testsuite is 1 for some grammar, the testsuite is complete w.r.t. this gram­mar. Such a testsuite can reliably be used to mon­itor changes in the grammar: Any reduction in the grammar's coverage will show up in the failure of some test case (for negative test cases, cf. Sec.4.3).

If the testsuite is not complete, instrumentation can identify disjuncts which are not exercised. These might be either (i) appropriate, but untested, dis­juncts calling for the addition of a test case, or (ii) in­appropriate disjuncts, for which a grammatical test case exercising them cannot be constructed.

Checking completeness of our local testsuite of 1787 items, we found that only 1456 out of 3730 grammar disjuncts in our German grammar were tested, yielding = 0.39 (the TSNLP testsuite containing 1093 items tests only 1081 disjuncts, yielding T^g = 0.28). Fig.3 shows an example of a gap in our testsuite (there are no examples of circumpositions), while Fig.4 shows an inapproppri-ate disjunct thus discovered (the category ADVadj has been eliminated in the lexicon, but not in all rules). Another error class is illustrated by Fig.5, which shows a disjunct that can never be used due to an LFG coherence violation; the grammar is in­consistent here.

I Pcircum 4-=t:

I ADVadj 4=f

ADVstd 4=f

Besides being complete, a testsuite must be econom­ical, i.e., contain as few items as possible. Instru­mentation can identify redundant test cases, where redundancy can be defined in three ways:

similarity There is a set of other test cases which jointly exercise all disjunct which the test case under consideration exercises.

equivalence There is a single test case which ex­ercises exactly the same combination (s) of dis­juncts.

strict equivalence There is a single test case which is equivalent to and, additionally, exer­cises the disjuncts exactly as often as, the test case under consideration.

Fig.6 shows equivalent test cases found in our testsuite: Example 1 illustrates the distinction be­tween equivalence and strict equivalence; the test cases contain different numbers of attributive adjec­tives. Example 2 shows that our grammar does not make any distinction between adverbial usage and secondary (subject or object) predication.

The reduction we achieved in size and processing time is shown in Table 1, which contains measure­ments for a test run containing only the parseable test cases, one without equivalent test cases (for ev­ery set of equivalent test cases, one was arbitrar­ily selected), and one without similar test cases. The last was constructed using a simple heuristic: Starting with the sentence exercising the most dis­juncts, working towards sentences relying on fewer disjuncts, a sentence was selected only if it exercised a disjunct which no previously selected sentence ex­ercised. Assuming that a disjunct working correctly suite, but receive no analysis since the grammatical function PREEDAT is not defined as such in the configuration section.

A different application of instrumentation is the tai­loring of a general grammar to specific genres. All-purpose grammars are plagued by lexical and struc­tural ambiguity that leads to overly long runtimes. If this ambiguity could be limited, parsing efficiency would improve. Instrumenting a general grammar allows to automatically derive specialized subgram-mars based on sample corpora. This setup has sev­eral advantages: The larger the overlap between gen­res, the larger the portion of grammar development work that can be recycled. The all-purpose grammar is linguistically more interesting, because it requires an integrated concept, as opposed to several sepa­rate genre-specific grammars.

I will discuss two ways of improving the efficiency of parsing a sublanguage, given an all-purpose uni­fication grammar. The first consists in deleting un­used disjuncts, while the second uses a staged pars­ing process. The experiments are only sketched, to indicate the applicability of the instrumentation technique, and not to directly compete with other proposals on grammar specialization. For example, the work reported in (Rayner and Samuelsson, 1994; Samuelsson, 1994) differs from the one presented be­low in several aspects: They induce a grammar from a treebank, while I propose to annotate the gram­mar based on all solutions it produces. No criteria for tree decomposition and category specialization are needed here, and the standard parsing algorithm can be used. On the other hand, the efficiency gains are not as big as those reported by (Rayner and Samuelsson, 1994).

Given a large sample of a genre, instrumentation al­lows you to determine the likely constructions of that genre. Eliminating unused disjuncts allows faster

Descriptor Content Coverage

HC-DE Copier/Printer User Manual 89%

WHB Car Maintenance Instructions 76%

NEWS-SC Verb-final subclauses from News 75%

parsing due to a smaller grammar. An experiment was conducted with several corpora as detailed in Table 2. There was some effort to cover the corpus HC-DE, but no grammar development based on the other corpora. The NEWS-SC corpus is part the corpus of verb-final sentences used by (Beil et al., 1999).

A training set of 1000 sentences from each cor­pus was parsed with an instrumented base gram­mar. From the parsing results, the exercised gram­mar disjuncts were extracted and used to construct a corpus-specific reduced grammar. The reduced grammars were then used to parse a test set of an­other 1000 sentences from each corpus. Table 3 shows the performance improvement on the corpora: It gives the size of the grammars in terms of the number of rules (with regular expression right-hand sides and feature annotation), the number of arcs (corresponding to unary or binary rules with dis­junctive feature annotation), and the number of dis­juncts (unary or binary rules with unique feature annotation). The number of mismatches counts the sentences for which the solution (s) obtained differed from those obtained with the base grammar, while the number of additions counts the sentences which did not receive a parse with the base grammar due to resource limitations (runtime or memory), but re­ceived one with the reduced grammar. The other columns give timings to process the total corpus, and the longest and average processing time per sen­tence; time is in seconds. The last column gives the average number of solutions per sentence.

Due to the sampling of a genre, the grammars obtained can only be approximate. To determine the relation of the sample size to the quality of the grammar obtained, the coverage of random fragment grammars was measured in the following way: Ran­domly select a number of sentences from the to­tal corpus, construct (in the same way as described above for the reduced grammar) a fragment gram­mar, and determine its coverage on the test set from the respective corpus. The graphs in Fig.8 show how the coverage and runtime relate to the number of sentences on which the fragment grammars are based. The leftmost data point (x value 0) describes the performance of the reduced grammar on the training set, while the rightmost data point describes its performance on the test set. The data points in between represent fragment grammars based on as many sentences as given by the x axis value.

The results reported here represent the minimal performance gain due to the fact that the construc­tion of reduced and fragment grammars are not based on the correct solutions for the training sen­tences, but rather on all solutions produced by the base grammar. The construction of a large-scale treebank with manually verified solutions is under way but has not yet progressed far enough to serve as input for this experiment. Even with this system­atic, but curable error, the reduction reduces overall processing by a factor of four. The number of solu­tions is constant because only unused disjuncts are eliminated; this will change if the treebank solutions are used to construct the reduced grammar.

I have presented the adaptation of code instrumenta­tion to Grammar Engineering, discussing measures and implementations, and sketching several applica­tions together with preliminary results.

The main application is to improve grammar and testsuite by exploring the relation between both of them. Viewed this way, testsuite writing can ben­efit from grammar development because both de­scribe the syntactic constructions of a natural lan­guage. Testsuites systematically list these construc­tions, while grammars give generative procedures to construct them. Since there are currently many-more grammars than testsuites, we may re-use the work that has gone into the grammars for the im­provement of testsuites.

Other applications of instrumentation are possi­ble; genre adaptation was discussed in some depth. On a more general level, one may ask whether other methods from SE may fruitfully apply to GE as well, possibly in modified form. For example, the static analysis of programs, e.g., detection of unreachable code, could also be applied for grammar develop­ment to detect unusable rules.

L. Balkan and F. Fouvry. 1995. Corpus-based test suite generation. TSNLP-WP 2.2, University- of Essex.

L. et al. Balkan. 1994. Test Suite Design Annotation Scheme. TSNLP-WP2.2, University- of Essex.

F. Beil, G. Carroll, D. Prescher, S. Riezler, and M. Rooth. 1999. Inside-outside estimation of a lexicalized PCFG for german. In Proc. 37th An­nual Meeting of the ACL. Maryland.

F. Ciravegna, A. Lavelli, D. Petrelli, and F. Pianesi. 1998. Developing language reesources and appli­cations with GEPPETTO. In Proc. 1st Int'l Conf. on Language Resources and Evaluation, pages 619-625. Granada/Spain, 28-30 May 1998.

EAGLES. 1996. Evaluation of Natural Language Processing Systems. Final Report EAG-EWG-PR.2.

D. Flickinger, J. Nerbonne, I. Sag, and T. Wa-sow. 1987. Toward Evaluation of NLP Systems. Hewlett-Packard Laboratories, Palo Alto/CA.

A. Frank, T.H. King, J. Kuhn, and J. Maxwell. 1998. Optimality theory style constraint rank­ing in large-scale LFG grammar. In Proc. of the LFG98 Conference. Brisbane/AUS, Aug 1998, CSLI Online Publications.

W.C. Hetzel. 1988. The complete guide to software testing. QED Information Sciences, Inc. Welles-ley/MA 02181.

R.M. Kaplan and J. Bresnan. 1982. Lexical-functional grammar: A formal system for gram­matical representation. In J. Bresnan and R.M. Kaplan, editors, The Mental Representation of Grammatical Relations, pages 173-281. Cam­bridge, MA: MIT Press.

S. Lehmann and S. Oepen. 1996. Tsnlp - test suites for natural language processing. In Proc. 16th Int'l Conf. on Computational Linguistics, pages 711-716. Copenhagen/DK.

K. Netter, S. Armstrong, T. Kiss, J. Klein, and S. Lehman. 1998. Diet - diagnostic and eval­uation tools for nip applications. In Proc. 1st Int'l Conf. on Language Resources and Evalua­tion, pages 573-579. Granada/Spain, 28-30 May 1998.

S. Oepen and D.P. Flickinger. 1998. Towards sys­tematic grammar profiling:test suite techn. 10 years afte. Journal of Computer Speech and Lan­guage, 12:411-435.

M. Rayner and D. Carter. 1996. Fast parsing us­ing pruning and grammar specialization. In Proc. 34th Annual Meeting of the ACL. Santa Cruz, USA.

M. Rayner and C. Samuelsson. 1994. Corpus-based grammar specialization for fast analysis. In M­S. Agnäs, H. Alshawi, I. Btrean, D. Carter, and K. Ceder, editors, Spoken Language Translator: First-Year Report, pages 41-54. Report CRC-043, Cambridge/UK: SRI International.

C. Samuelsson. 1994. Grammar spezialization through entropy thresholds. In Proc. 32nd Annual Meeting of the ACL.