(5) The essays were passed through the computer, under the control of the program which collected data about the proxes. The output was as appears in Figure 2.

PEG-1A Output

Figure 2 shows a piece of output from PEG-IA. Line A shows the way a sentence from the student essay is re­written in 12-character double-precision computer "words" and stored in memory. Line B shows the summary of data for that sentence just analyzed. The first number is the essay identification. The other numbers of Line B are some counts from that sentence. Line C shows a summary of these counts, across sentences, for this whole essay. And Line D are these measures transformed in a number of simple ways, and ready for input into the final analysis.

(6) These scores were then analyzed for their mul­tivariate relationship to the human ratings, were weighted appropriately, and were used to maximize the prediction of the expert human ratings. This was all done by use of standard multiple-regression programs.

The first analyses produced results as shown in Table 1. Here it is possible to read the list of proxes (Col. A), 'and their correlation, after transformation, with the human judgments of overall quality (Col. B). Col. C shows their contribution to the total multiple regression, and Col. D indicates the test-retest relia­bility of the proxes themselves, as discovered from two essays written by the same students, with about a month between writings.

Indeed, the steps in this hypothetical essay grader are close to the heart of the most persistent and troublesome problems in linguistics. Is it necessary that sentences be syntactically analyzed before mapping into deep structure? What is the proper role of semantics in such deep structure? How can the outside knowledge of the reader be incorporated into the machine analysis? (For some discussion of this pro­blem, see Quillian, 1966). In general, how may we in­corporate some of the intuitive richness which the literate hurian brings to his reading?

It is not expected that workers in essay grading will suddenly resolve all such questions. They may be recognized as those which so trouble linguists as to contribute to the recent official pessimism, in the United States, about the future of mechanical transla­tion. After 15 years of effort, mechanical translation is still regarded as disappointing in quality, and vir­tually no sustained output of any. machine program would be ordinarily mistaken for the work of a professional human translator.

On the other hand, the earliest attempts at essay grading by computer have,in a very limited way, leaped ahead of machine translation. And if the expert human ratings of high school essays may be regarded as an acceptable goal, then the machine program appears to have reached such a goal already. For that matter, improved performance, even superior to that of the in­dividual human expert, appears to be immediately practi­cable as well.

The explanation of this advantage, of course, is that the- problem of essay grading as attacked in the current work is much easier than the problem of machine translation. In translation, every nuance of the input string should be accounted for in the output string. In essay grading, only" a certain portion of the input text needs to be accounted for, and the output .does not depend on the existence of any large tlanguage-generating system. High quality machine translation apparently de­mands a fair portion of the total language-manipulating capability of the human, but essay grading may use only a fraction of it, and may process language in ways quite different from that of ther human being. For example, our present programs have to date largely ignored order and sequence in the essays, although to the human the order of words is, of course, of crucial and unceasing import­ance.

Since essay grading can work with such fractional information, then, why pursue the deeper analysis of Table 4? Clearly, the purpose is not entirely the same as it would be for the usual linguist. At any discrete time in research, what is sought is not necessarily the perfect humanoid behavior, but rather those portions of that behavior which, given any current state of the art, will contribute optimally to efficient and practicable improvements in output. Indeed, regardless of the eventual perfection of deep linguistic behavior, for any specific application to essay grading, at any one moment, large portions of such available behavior may be irrele­vant, just as it seems that ordinary human language processing does not usually call for our full linguistic ëffort.

Yet we regard it as eventually important to be able to perform these various kinds of advanced machine analysis when required. Therefore, the eventual uses of the ideal essay analyzer may require analytic capa­bility as deep as may be imagined. Writing out suitable comments for the student, for example, will in some cases, tax any system which may be foreseen.

Even approximate solutions to these problems, how­ever, though unsatisfactory for certain scientific pur­poses, could make important contributions to the educa­tional description and evaluation of essays. For such evaluation is itself probabilistic, limited by imperfect asymptotes of writer consistency and rater agreement. And such evaluation therefore does not require, to be practicable and satisfactory, the same deterministic perfection which has continued to elude and frustrate researchers -in mechanical translation. There is a fund­amental difference in goals, which must be realized. As has been demonstrated here, the output from much cruder statistical programs has already reached a quality not too remote from usefulness. The more advanced strate­gies currently seem, at least to the present workers, bright with promise. '. ,

Daigon, Arthur. Computer Grading of English Composition. The English Journal, January, 1966, 46-52.

Harris, Z. S. Discourse analysis. Language, 1952, 8_ (4) , 474-493.

Computer Search for Traits of Opinionation, Vague­ness, and Specificity-Distinction in Student Essays. Paper read at the Annual Meeting of the American Psychological Association, Washington, D.C., September 2, 1967.

Kuno, Susumu. Some characteristics of the Multiple-Path Syntactic Analyzer. Language Data Processing, Cambridge:. Harvard Computation Laboratory, 1964. C6, 1-8.

Marcotte, Donald. The.Computer Analysis of Cliche Be­havior in Student Writing. Paper read at the Annual Meeting of the American Educational Research Association, New York, February 18, 1967.

McNemar, Quinn; Psychological Statistics, 3rd ed. New York: Wiley, 1962.

Olney, John and Londe, D. A research plan for investi­gating English discourse structure with particular attention to anaphoric relationships. Tech Memo mm-(L)-3256. Santa Monica, California: System Development Corporation. November 22, 1966. 17 p.

Page, Ellis B. The Imminence of Grading Essays by Computer. Phi Delta Kappan, January, 1966, 238-243.

Page, Ellis B. Grading Essays by Computer: Progress Report. Proceedings of the 1966 Invitational Con­ference on Testing Problems. Prxnceton, N.J.: Educational Testing Service, 1967. Pp. 87-100.

Paulus, Dieter. Problems of Nonlinearity in Grading Essays. Paper read at the Annual Meeting of the American Educational Research Association, New York, February 16, 1967a.

Paulus, Dieter. Feedback in Project Essay Grade. Paper read at the Annual Meeting of the American Psycholog­ical Association, Washington, D.C., September 2, 1967b.

Quillian, M. Ross. Semantic Memory. Cambridge, Mass.: Bolt Beranek and Newman, 1966.