<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="229"/><title>Modeling Organization in Student Essays</title><pubinfo>Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing,pages 229-239, MIT, Massachusetts, USA, 9-11 October 2010. ©2010 Association for Computational Linguistics</pubinfo><author surname="Persing" givenname="Isaac"><org  name="University of Texas at Dallas" country="USA" city="Richardson"/></author><author surname="Davis" givenname="Alan"><org  name="University of Texas at Dallas" country="USA" city="Richardson"/></author><author surname="Ng" givenname="Vincent"><org  name="University of Texas at Dallas" country="USA" city="Richardson"/></author></firstpageheader><frontmatter><p><b>Modeling Organization in Student Essays</b></p><p><b>Isaac Persing </b>and <b>Alan Davis </b>and <b>Vincent Ng</b></p><p>Human Language Technology Research Institute University of Texas at Dallas Richardson, TX 75083-0688 (persingq,alan,vince}@hlt.utdallas.edu</p></frontmatter><abstract>Automated essay scoring is one of the most important educational applications of natural language processing. Recently, researchers have begun exploring methods of scoring es­says with respect to particular dimensions of quality such as coherence, technical errors, and relevance to prompt, but there is rela­tively little work on modeling organization. We present a new annotated corpus and pro­pose heuristic-based and learning-based ap­proaches to scoring essays along the organi­zation dimension, utilizing techniques that in­volve sequence alignment, alignment kernels, and string kernels. </abstract></header><body><section number="1" title="Introduction"><p>Automated essay scoring, the task of employing computer technology to evaluate and score writ­ten text, is one of the most important educational applications of natural language processing (NLP) (see Shermis and Burstein (2003) and Shermis et al. (2010) for an overview of the state of the art in this task). Recent years have seen a surge of interest in this and other educational applications in the NLP community, as evidenced by the panel discussion on "Emerging Application Areas in Computational Linguistics" at NAACL 2009, as well as increased participation in the series of workshops on "Innova­tive Use of NLP for Building Educational Applica­tions". Besides its potential commercial value, au­tomated essay scoring brings about a number of rel­atively less-studied but arguably rather challenging discourse-level problems that involve the computa­tional modeling of different facets of text structure, such as content, coherence, and organization.</p><p>A major weakness of many existing essay scor­ing engines such as IntelliMetric (Elliot, 2001) and Intelligent Essay Assessor (Landauer et al., 2003) is that they adopt a holistic scoring scheme, which summarizes the quality of an essay with a single score and thus provides very limited feedback to the writer. In particular, it is not clear which di­mension of an essay (e.g., coherence, relevance) a score should be attributed to. Recent work ad­dresses this problem by scoring a particular dimen­sion of essay quality such as coherence (Miltsakaki and Kukich, 2004), technical errors, and relevance to prompt (Higgins et al., 2004). Automated sys­tems that provide instructional feedback along mul­tiple dimensions of essay quality such as <i>Criterion </i>(Burstein et al., 2004) have also begun to emerge.</p><p>Nevertheless, there is an essay scoring dimension for which few computational models have been de­veloped — <i>organization. </i>Organization refers to the structure of an essay. A high score on organization means that writers introduce a topic, state their po­sition on that topic, support their position, and con­clude, often by restating their position (Silva, 1993). A well-organized essay is structured in a way that logically develops an argument. Note that organi­zation is a different facet of text structure than co­herence, which is concerned with the transition of ideas at both the global (e.g., paragraph) and local (e.g., sentence) levels. While organization is an im­portant dimension of essay quality, state-of-the-art essay scoring software such as e-rater V.2 (Attali and Burstein, 2006) employs rather simple heuristic-based methods for computing the score of an essay along this particular dimension.</p><p>Our goal in this paper is to develop a compu­tational model for the organization of student essays.<page local="2" global="230"/> While many models of text coherence have been developed in recent years (e.g., Barzilay and Lee (2004), Barzilay and Lapata (2005), Soricut and Marcu (2006), Elsner et al. (2007)), the same is not true for text organization. One reason is the avail­ability of training and test data for coherence mod­eling. Coherence models are typically evaluated on the sentence ordering task, and hence training and test data can be generated simply by scrambling the order of the sentences in a text. On the other hand, it is not particularly easy to find poorly organized texts for training and evaluating organization models. We believe that student essays are an ideal source of well- and poorly-organized texts. We evaluate our organization model on a data set of 1003 essays an­notated with organization scores.</p><p>In sum, our contributions in this paper are two­fold. First, we address a less-studied discourse-level task — predicting the organization score of an essay — by developing a computational model of organi­zation, thus establishing a baseline against which fu­ture work on this task can be compared. Second, we annotate a subset of our student essay corpus with organization scores and make this data set publicly available. Since progress in organization modeling is hindered in part by the lack of a publicly anno­tated corpus, we believe that our data set will be a valuable resource to the NLP community.</p></section><section number="2" title="Corpus Information"><p>We use as our corpus the 4.5 million word Interna­tional Corpus of Learner English (ICLE) (Granger et al., 2009), which consists of more than 6000 es­says written by university undergraduates from 16 countries and 16 native languages who are learners of English as a Foreign Language. 91% of the ICLE texts are argumentative. The essays we used vary greatly in length, containing an average of 31.1 sen­tences in 7.5 paragraphs, averaging 4.1 sentences per paragraph. About one quarter of the essays had five or fewer paragraphs, and another quarter contained nine or more paragraphs. Similarly, about one quar­ter of essays contained 24 or fewer sentences and the longest quarter contained 36 or more sentences</p><p>We selected a subset consisting of 1003 essays from the ICLE to annotate and use for training and testing of our model of essay organization. While</p><p>Table 1 : Some examples of writing topics.</p><p><i>narrative </i>writing asks students to compose descrip­tive stories, <i>argumentative </i>(also known as <i>persua­sive) </i>writing requires students to state their opinion on a topic and to validate that opinion with convinc­ing arguments. For this reason, we selected only ar­gumentative essays rather than narrative pieces, be­cause they contain the discourse structures and kind of organization we are interested in modeling.</p><p>To ensure representation across native languages of the authors, we selected mostly essays written in response to topics which are well-represented in multiple languages. This avoids many issues that may arise when certain vocabulary is used in re­sponse to a particular topic for which essays written by authors from only a few languages are available. Table 1 shows three of the twelve topics selected for annotation. Fifteen native languages are represented in the set of essays selected for annotation.</p></section><section number="3" title="Corpus Annotation"><p>To develop our essay organization model, human an-notators scored 1003 essays using guidelines in an essay annotation rubric. Annotators evaluated the organization of each essay using a numerical score from 1 to 4 at half-point increments. This contrasts with previous work on essay scoring, where the cor­pus is annotated with a binary decision (i.e., <i>good </i>or <i>bad) </i>for a given scoring dimension (e.g., Higgins et al. (2004)). Hence, our annotation scheme not only provides a finer-grained distinction of organization quality (which can be important in practice), but also makes the prediction task more challenging.<page local="3" global="231"/></p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Topic</p></td><td class="cell"><p>Languages</p></td><td class="cell"><p>Essays</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Most university degrees are</p></td><td class="cell"><p>13</p></td><td class="cell"><p>147</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>theoretical and do not prepare</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>students for the real world.</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>They are therefore of very lit-</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>tle value.</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>The prison system is out-</p></td><td class="cell"><p>11</p></td><td class="cell"><p>103</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>dated.   No civilized society</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>should punish its criminals: it</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>should rehabilitate them.</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>In his novel <i>Animal Farm,</i></p></td><td class="cell"><p>10</p></td><td class="cell"><p>82</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>George Orwell wrote "All</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>men are equal but some are</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>more equal than others." How</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>true is this today?</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>The meaning of each integer score was described and discussed in detail. Table 2 shows the descrip­tion of each score for the organization dimension.</p><p>Our annotators were selected from over 30 appli­cants who were familiarized with the scoring rubric and given sample essays to score. The six who were most consistent with the expected scores were given additional essays to annotate. To ensure consistency in scoring, we randomly selected a large subset of our corpus (846 essays) to have graded by two differ­ent annotators. Analysis of these doubly annotated essays reveals that, though annotators only exactly agree on the organization score of an essay 29% of the time, the scores they apply are within 0.5 points in 71% of essays and within 1.0 point in 93% of es­says. Additionally, if we treat one annotator's scores as a gold standard and the other annotator's scores as predictions, the predicted scores have a mean er­ror of 0.54 and a mean squared error of 0.50. Table 3 shows the number of essays that received each of the seven scores for organization.</p></section><section number="4" title="Function Labeling"><p>As mentioned before, a high score on organization means that writers introduce a topic, support their position, and conclude. If one or more of these ele­ments are missing or if they appear out of order (e.g., the conclusion appears before the introduction), the resulting essay will typically be considered poorly organized. Hence, knowing the <i>discourse function</i> <i>label </i>of each paragraph in an essay would be help­ful for predicting its organization score.</p><p>Two questions naturally arise. First, how can we obtain the discourse function label of each para­graph? One way is to automatically acquire such labels from a corpus of student essays where each paragraph is annotated with its discourse function label. To our knowledge, however, there is no pub­licly available corpus that is annotated with such in­formation. As a result, we will resort to labeling a paragraph with its function label heuristically.</p><p>Second, which paragraph function labels would be most useful for scoring the organization of an es­say? Based on our linguistic intuition, we identify four potentially useful paragraph function labels: In­troduction, Body, Rebuttal, and Conclusion. Table 4 gives the descriptions of these labels.</p><p>Setting aside for the moment the problem of ex­actly how to predict an essay's organization score given its paragraph sequence, the problem of ob­taining paragraph labels to use for this task still re­mains. As mentioned above, we adopt a heuristic ap­proach to paragraph function labeling. The question, then, is: what kind of knowledge sources should our heuristics be based on? We have identified two types of knowledge sources that are potentially useful for paragraph labeling. The first of these are positional, dealing with where in the essay a paragraph appears. So for example, the first paragraph in an essay is likely to be an Introduction, while the last is likely to be a Conclusion. A paragraph in any other posi­tion, on the other hand, is more likely to be a Body or Rebuttal paragraph.</p><table caption="Table 2: Descriptions of the meaning of each score." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Score</p></td><td class="cell"><p>Description of Essay Organization</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>4</p></td><td class="cell"><p>essay is <b>well structured </b>and is organized in a way that logically develops an argument</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3</p></td><td class="cell"><p>essay is <b>fairly well structured </b>but could somewhat benefit from reorganization</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2</p></td><td class="cell"><p>essay is <b>poorly structured </b>and would greatly benefit from reorganization</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p>essay is <b>completely unstructured </b>and re­quires major reorganization</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 4: Descriptions of paragraph function labels." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Label</p></td><td class="cell"><p>Name</p></td><td class="cell"><p>Paragraph Type</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>I</b></p></td><td class="cell"><p>Introduction</p></td><td class="cell"><p>introduces essay topic and states author's position and main ideas</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>B</b></p></td><td class="cell"><p>Body</p></td><td class="cell"><p>provides reasons, evidence, and examples to support main ideas</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>C</b></p></td><td class="cell"><p>Conclusion</p></td><td class="cell"><p>summarizes and concludes ar­guments made in body para­graphs</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>R</b></p></td><td class="cell"><p>Rebuttal</p></td><td class="cell"><p>considers counter-arguments to thesis or main ideas</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 3: Distributionoforganizationscores." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>score</p></td><td class="cell"><p>1.0</p></td><td class="cell"><p>1.5</p></td><td class="cell"><p>2.0</p></td><td class="cell"><p>2.5</p></td><td class="cell"><p>3.0</p></td><td class="cell"><p>3.5</p></td><td class="cell"><p>4.0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>essays</p></td><td class="cell"><p>24</p></td><td class="cell"><p>14</p></td><td class="cell"><p>35</p></td><td class="cell"><p>146</p></td><td class="cell"><p>416</p></td><td class="cell"><p>289</p></td><td class="cell"><p>79</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="4" global="232"/><p>A second potentially useful knowledge source in­volves the types of sentences appearing in a para­graph. This idea presupposes that, like paragraphs, sentences too can have discourse function labels in­dicating the logical role they play in an argument. The sentence label schema we propose, which is de­scribed in Table 5, is based on work in discourse structure by Burstein et al. (2003), but features addi­tional sentence labels.</p><p>To illustrate why these sentence function labels may be useful for paragraph labeling, consider a paragraph containing a Thesis sentence. The pres­ence of a Thesis sentence is a strong indicator that the paragraph containing it is either an Introduction or Conclusion. Similarly, a paragraph containing Rebuttal or Solution sentences is more likely to be a Body or Rebuttal paragraph.</p><p>Hence, to obtain a paragraph's function label, we need to first label its sentences. However, we are faced with the same problem: how can we ob­tain the sentence function labels? One way is to learn them from a corpus where each sentence is manually annotated with its sentence function la­bel, which is the approach adopted by Burstein et al. (2003). However, this annotated corpus is not publicly available. In fact, to our knowledge, there is no publicly-available corpus that is annotated with sentence function labels. Consequently, we adopt a heuristic approach to sentence function labeling.</p><p>Overall, we created a knowledge-lean set of heuristic rules labeling paragraphs and sentences. Because many of the paragraph labeling heuristics depend on the availability ofsentence labels, we will describe the sentence labeling heuristics first. For each sentence function label x, we identify several features whose presence increases our confidence that a given sentence is an example of x. So for example, the presence of any of the words "agree", "think", or "opinion" increases our confidence that the sentence they occur in is a Thesis. Ifthe sentence instead contains words such as "however", "but", or "argue", these increase our confidence that the sentence is a Rebuttal. The features we examine for sentence labeling are not limited to words, how­ever. Each content word the sentence shares with the essay prompt gives us evidence that the sentence is a restatement of the prompt. Having searched a sentence for all these clues, we finally assign the sentence the function label having the most support among the clues found.</p><p>The heuristic rules for paragraph labeling are sim­ilar in nature, though they depend heavily on the labels of a paragraph's component sentences. If a paragraph contains Thesis, Prompt, or Background sentences, the paragraph is likely to be an Introduc­tion. However, if a paragraph contains Main Idea, Support, or Conclusion sentences, it is likely to be a Body paragraph. Finally, as mentioned previously, some positional information is used in labeling para­graphs. For example, a paragraph that is the first paragraph in an essay is likely to be an Introduction, but a paragraph that is neither the first nor the last is likely to be either a Rebuttal or Body paragraph. After searching a paragraph for all these features, we gather the pieces of evidence in support of each paragraph label and assign the paragraph the label having the most support.<footnote anchor="1"/><page local="5" global="233"/></p><footnote label="1">Space limitations preclude a complete listing of these para-</footnote><table caption="Table 5: Descriptions of sentence function labels." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Label</p></td><td class="cell"><p>Name</p></td><td class="cell"><p>Sentence Function</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>P</b></p></td><td class="cell"><p>Prompt</p></td><td class="cell"><p>restates the prompt given to the author and contains no new material or opinions</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>T</b></p></td><td class="cell"><p>Transition</p></td><td class="cell"><p>shifts the focus to new topics but contains no meaningful information</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>H</b></p></td><td class="cell"><p>Thesis</p></td><td class="cell"><p>states the author's position on the topic for which he/she is arguing</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>M</b></p></td><td class="cell"><p>Main Idea</p></td><td class="cell"><p>asserts reasons and foundational arguments that support the thesis</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>E</b></p></td><td class="cell"><p>Elaboration</p></td><td class="cell"><p>further explains reasons and ideas but contains no evidence or examples</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>S</p></td><td class="cell"><p>Support</p></td><td class="cell"><p>provides evidence and examples to support the claims made in other statements</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>c</p></td><td class="cell"><p>Conclusion</p></td><td class="cell"><p>summarizes and concludes the entire argument or one of the main ideas</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>R</b></p></td><td class="cell"><p>Rebuttal</p></td><td class="cell"><p>considers counter-arguments that contrast with the thesis or main ideas</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>O</b></p></td><td class="cell"><p>Solution</p></td><td class="cell"><p>puts to rest the questions and problems brought up by counter-arguments</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>u</p></td><td class="cell"><p>Suggestion</p></td><td class="cell"><p>proposes solutions the problems brought up by the argument</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><section number="5" title="Heuristic-Based Organization Scoring"><p>Having applied labels to each paragraph in an es­say, how can we use these labels to predict the es­say's score? Recall that the importance of each para­graph label stems not from the label itself, but from the sequence of labels it appears in. Motivated by this observation, we exploit a technique that is com­monly used in bioinformatics — <i>sequence align­ment. </i>While sequence alignment has also been used in text and paraphrase generation (e.g., Barzilay and Lee (2002; 2003)), it has not been extensively ap­plied to other areas of language processing, includ­ing essay scoring. In this section, we will present two heuristic approaches to organization scoring, one based on aligning <i>paragraph sequences </i>and the other on aligning <i>sentence sequences.</i></p><subsection number="5.1" title="Aligning Paragraph Sequences"><p>As mentioned above, our first approach to heuristic organization scoring involves aligning paragraph se­quences. Specifically, this approach operates in two steps. Given an essay e in the test set, we (1) find the k essays in the training set that are most similar to e via paragraph sequence alignment, and then (2) pre­dict the organization score of e by aggregating the scores of its k nearest neighbors obtained in the first step. Below we describe these two steps in detail.</p><p>First, to obtain the k nearest neighbors of e, we employ the Needleman-Wunsch alignment algo­rithm (Needleman and Wunsch, 1970), which com­putes a similarity score for any pair of essays by finding an optimal alignment between their para­graph sequences. To illustrate why we believe se­quence alignment can help us determine which es­says are most similar, consider two example es­says. One essay, which we will call IBBBC, begins with an Introductory paragraph, follows it with three Body paragraphs, and finally ends with a Conclud­ing paragraph. Another essay CRRRI begins with a paragraph stating its Conclusion, follows it with three Rebuttal paragraphs, and ends with a para­graph Introducing the essay's topic. We can tell by a casual glance at the sequences that any reasonable similarity function should tell us that they are not</p><p>graph and sentence labeling heuristics. See our website at http://www.hlt.utdallas.edu/~alan/ICLE/ for the complete list of heuristics.</p><p>very similar. The Needleman-Wunsch alignment al­gorithm has this effect since the score of the align­ment it produces would be hurt by the facts that (1) there is not much overlap in the sets of paragraph labels each contains, and (2) the paragraph labels they do share (I and C) do not occur in the same order. The resulting alignment would therefore con­tain many mismatches or indels.<footnote anchor="2"/></p><p>If we now consider a third essay whose para­graph sequence could be represented as IBRBC, a good similarity function should tell us that IBBBC and IBRBC are very similar. The Needleman-Wunsch alignment score between the two paragraph sequences has this property, as the alignment al­gorithm would discover that the two sequences are identical except for the third paragraph label, which could be mismatched for a small penalty. We would therefore conclude that the IBBBC and IBRBC es­says should receive similar organization scores.</p><p>To fully specify how to find the k nearest neigh­bors of an essay, we need to define a similarity func­tion between paragraph labels. In sequence align­ment, similarity function <i>S</i>(i, j) tells us how likely it is that symbol i (in our case, a paragraph label) will be substituted with another symbol j. While we expect that in an alignment between high-scoring essays, an Introduction paragraph is most likely to be aligned with another Introduction paragraph, how much worse should the alignment score be if an In­troduction paragraph needs to be mismatched with a Rebuttal paragraph or replaced with an indel? We solve this problem by heuristically defining the sim­ilarity function as follows: S(i, j) = 1 when i = j, S) = —1 when i = j, and also S(i, —) = S(—, i) = —1, where '—' is an indel. In other words, the similarity function encourages the align­ment between two identical function labels and dis­courages the alignment between two different func­tion labels, regardless of the type of function labels.</p><p>After obtaining the k nearest neighbors of e, the next step is to predict the organization score of e by aggregating the scores of its k nearest neighbors into one number. (Note that we know the organization score of each nearest neighbor, since they are all taken from the training set.<page local="6" global="234"/>) One natural way to do this would be to take the mean, median, or mode of its k nearest neighboring essays from the training set. Hence, our first heuristic method Hp for scoring organization has three variants.</p><footnote label="2">In pairwise sequence alignment, a mismatch occurs when one symbol has to be substituted for another to make two se­quences match. An indel indicates that in order to transform one sequence to match another, we must either insert a symbol into one sequence or delete a symbol from the other sequence.</footnote></subsection><subsection number="5.2" title="Aligning Sentence Sequences"><p>An essay's paragraph sequence captures information about its organization at a high level, but ignores much of its lower level structure. Since we have also heuristically labeled sentences, it now makes sense to examine the sequences of sentence function labels within an essay's paragraphs. The intuition is that at least some portion of an essay's organization score can be attributed to the organization of the sentence sequences of its component paragraphs.</p><p>To address this concern, we propose a second heuristic approach to organization scoring. Given a test essay e, we first find for each <i>paragraph </i>in e the k <i>paragraphs </i>in the training set that are most similar to it. Specifically, each paragraph is repre­sented by its sequence of <i>sentence </i>function labels. Given this paragraph representation, we can find the k nearest neighbors of a paragraph by applying the Needleman-Wunsch algorithm described in the pre­vious subsection to align <i>sentence </i>sequences, using the same similarity function we defined above.</p><p>Next, we score each paragraph pi by aggregating the scores of its k nearest neighbors obtained in the first step, assuming the score of a nearest neighbor paragraph is the same as the organization score of the training set essay containing it. As before, we can employ the mean, median, or mode to aggregate the scores of the nearest neighbors of <i>pi.</i></p><p>Finally, we predict the organization score of e by aggregating the scores of its paragraphs obtained in the second step. Again, we can employ mean, me­dian, or mode to aggregate the scores. Since we have three ways ofaggregating the scores ofa paragraph's nearest neighbors and three ways of aggregating the resulting paragraph scores, this second method <i>Hs</i>for scoring organization has nine variants.</p></subsection></section><section number="6" title="Learning-Based Organization Scoring"><p>In the previous section, we proposed two heuris­tic approaches to organization scoring, one based on aligning paragraph label sequences and the other based on aligning sentence label sequences. In the process of constructing these two systems, however, we created a lot of information about the essays which might also be useful for organization scoring, but which the heuristic systems are unable to exploit. To remedy the problem, we introduce three learning-based systems which abstract the additional infor­mation we produced in three different ways. In each system, we use the SVM1ight (Joachims, 1999) im­plementation of regression support vector machines (SVMs) (Cortes and Vapnik, 1995) to train a regres-sor because SVMs have been frequently and suc­cessfully applied to a variety of NLP problems.</p><subsection number="6.1" title="Linear Kernel"><p>Owing to the different ways we presented of com­bining the scores of an essay's nearest neighbors, the paragraph label sequence alignment approach has three variants, and its sentence label sequence alignment counterpart has nine. Unfortunately, these heuristic approaches suffer from two major weak­nesses. First, it is not intuitively clear which of these 12 ways for predicting an essay's organiza­tion score is clearly better than the others. Second, it is not clear that the k nearest neighbors of an es­say will always be similar to it with respect to or­ganization score. While we do expect the alignment scores between good essays with reasonable para­graph sequences to be high, poorly organized es­says by their nature have more random paragraph sequences. Hence, we have no intuition about the k nearest neighbors of a poor essay, as it may have as high an alignment score with another poorly orga­nized essay as with a good essay.</p><p>Our solution to these problems is to use the orga­nization scores obtained by the 12 heuristic variants as features in a linear kernel SVM learner. We be­lieve that using the estimates given by all the 12 vari­ants of the two heuristic approaches rather than only one of them addresses the first weakness mentioned above. The second weakness, on the other hand, is addressed by treating the organization score predic­tions obtained by the nearest neighbor methods as features for an SVM learner rather than as estimates of an essay's organization score.</p><p>The approach we have just described, however, does not exploit the full power of linear kernel<page local="7" global="235"/></p><p>SVMs. One strength of linear kernels is that they make it easy to incorporate a wide variety of dif­ferent types of features. In an attempt to further enhance the prediction capability of the SVM re-gressor, we will provide it with not only the 12 fea­tures derived from the heuristic-based approaches, but also with two additional types offeatures.</p><p>First, to give our learner more direct access to the information we used to heuristically predict es­say scores, we can extract <i>paragraph label subse-quences<footnote anchor="3"/> </i>from each essay and use them as features. To illustrate the intuition behind these features, con­sider two paragraph subsequences: Introduction-Body and Rebuttal-Introduction. It is fairly typi­cal to see the first subsequence, I-B, at the begin­ning of a good essay, so its occurrence should give us a small amount of evidence that the essay it oc­curs in is well-organized. The presence of the sec­ond subsequence, R-I, however, should indicate that its essay's organization is poor because, in general, a good essay should not give a Rebuttal before an In­troduction. Because we can envision subsequences of various lengths being useful, we create a binary presence or absence feature in the linear kernel for each paragraph subsequence of length 1, 2, 3, 4, or 5 appearing in the training set.</p><p>Second, we employ <i>sentence label subsequences </i>as features in the linear kernel. Recall that when describing our alignment-based nearest neighbor organization score prediction methods, we noted that an essay's organization score may be partially attributable to how well the sentences within its paragraphs are organized. For example, if one of an essay's paragraphs contains the sentence la­bel subsequence Main Idea-Elaboration-Support-Conclusion this gives us some evidence that the es­say is overall well-organized since one of its compo­nent paragraphs contains this reasonably-organized subsequence. An essay with a paragraph contain­ing the subsequence Conclusion-Support-Thesis-Rebuttal, however, is likely to be poorly orga­nized because this is a poorly-organized subse­quence. Since sentence label subsequences of dif­fering lengths may be useful for score prediction, we create a binary presence or absence feature for each sentence label subsequence of length 1, 2, 3, 4, or 5 in the training set.</p><footnote label="3">Note that a subsequence is not necessarily contiguous.</footnote><p>While the number of nearest neighbor features is manageable, the presence of a large number of fea­tures can sometimes confuse a learner. For that rea­son, we do feature selection on the two types of subsequence features, selecting only 100 features for each type that has the highest information gain (see Yang and Pedersen (1997) for details). We call the system resulting from the use of these three types of features <i>Rlnps </i>because it uses <b>R</b>egression with <b>l</b>inear kernel to predict essay scores, and it uses <b>n</b>earest neighbor, <b>p</b>aragraph subsequence, and <b>s</b>entence subsequence features.</p></subsection><subsection number="6.2" title="String Kernel"><p>In a traditional learning setting, the feature set em­ployed by an off-the-shelf learning algorithm typ­ically consists of <i>flat </i>features (i.e., features whose values are discrete- or real-valued, as the ones de­scribed in the Linear Kernel subsection). Advanced machine learning algorithms such as SVMs, on the other hand, have enabled the use of <i>structured </i>fea­tures (i.e., features whose values are structures such as parse trees and sequences), owing to their ability to employ <i>kernels </i>to efficiently compute the similar­ity between two potentially complex structures.</p><p>Perhaps the most obvious advantage of employ­ing structured features is <i>simplicity. </i>To understand this advantage, consider learning in a traditional set­ting. Recall that we can only employ flat features in this setting, as we did with the linear kernel. Hence, if we want to use information from a parse tree as features, we will need to design heuristics to extract the desired parse-based features from parse trees. For certain tasks, designing a good set of heuris­tics can be time-consuming and sometimes difficult. On the other hand, SVMs enable a parse tree to be employed directly as a structured feature, obvi­ating the need to design heuristics to extract infor­mation from potentially complex structures. How­ever, structured features have only been applied to a handful of NLP tasks such as semantic role labeling (Moschitti, 2004), syntactic parsing and named en­tity identification (Collins and Duffy, 2002), relation extraction (Bunescu and Mooney, 2005), and coref-erence resolution (Versley et al., 2008). Our goal here is to explore this rarely-exploited capability of SVMs for the task of essay scoring.</p><page local="8" global="236"/><p>While the vast majority of previous NLP work on using structured features have involved tree ker­nels, we employ a kernel that is rarely investigated in NLP: <i>string kernels </i>(Lodhi et al., 2002). Informally, a string kernel aims to efficiently compute the sim­ilarity between two strings (or sequences) of sym­bols based on the similarity of their subsequences. We apply string kernels to essay scoring as follows: we represent each essay using its paragraph function label sequence, and employ a string kernel to com­pute the similarity between two essays based on this representation. Typically, a string kernel takes as in­put two parameters: K (which specifies the length of the subsequences in the two strings to compare) and A (which is a value between 0 and 1 that spec­ifies whether matches between non-contiguous sub­sequences in the two strings should be considered as important as matches between contiguous subse­quences). In our experiments, we select values for these parameters in a somewhat arbitrary manner. In particular, since A ranges between 0 and 1, we sim­ply set it to 0.5. For K, since in the flat features we considered all paragraph label sequences of lengths from 1 to 5, we again take the middle value, setting it to 3. We call the system using this kernel Rs be­cause it uses a <b>R</b>egression SVM with a <b>s</b>tring kernel to predict essay scores.</p></subsection><subsection number="6.3" title="Alignment Kernel"><p>In general, the purpose of a kernel function is to measure the similarity between two examples. The string kernel we described in the previous subsec­tion is just one way of measuring the similarity of two essays given their paragraph sequences. While this may be the most obvious way to use paragraph sequence information from a machine learning per­spective, our earlier use of the Needleman-Wunsch algorithm suggests a more direct way of extracting structured information from paragraph sequences.</p><p>More specifically, recall that the Needleman-Wunsch algorithm finds an optimal alignment be­tween two paragraph sequences, where an opti­mal alignment is defined as an alignment having the highest possible alignment score. The optimal alignment score can be viewed as another similar­ity measure between two essays. As such, with some slight modifications, the alignment score be­tween two paragraph sequences can be used as the kernel value for an Alignment Kernel.<footnote anchor="4"/> We call the system using this kernel <i>Ra </i>because it uses a <b>R</b>egression SVM with an <b>a</b>lignment kernel to pre­dict essay scores.</p></subsection><subsection number="6.4" title="Combining Kernels"><p>Recall that the flat features are computed using a lin­ear kernel, while the two types of structured features are computed using string and alignment kernels. If we want our learner to make use of more than one of these types of features, we need to employ a <i>compos­ite </i>kernel to combine them. Specifically, we define and employ the following composite kernel:</p><doubt alpha="50.0" length="2" tooSmall="False" monospace="0.0">1n</doubt><doubt alpha="50.0" length="24" tooSmall="False" monospace="0.0">Kc(Fl,F2) =-YJKt(Fl,F2),</doubt><p>where F1 and F2 are the full set of features (contain­ing both flat and structured features) that represent the two essays under consideration, <i>Ki </i>is the ith ker­nel we are combining, and n is the number of kernels we are combining. To ensure that each kernel under consideration contributes equally to the composite kernel, each kernel value Ki(F<b>i, </b>F2) is normalized so that its value falls between 0 and 1.</p></subsection></section><section number="7" title="Evaluation"><subsection number="7.1" title="Evaluation Metrics"><p>We designed three evaluation metrics to measure the error of our organization scoring system. The sim­plest metric, S1, is perhaps the most intuitive. It measures the frequency at which a system predicts the wrong score out of the seven possible scores. Hence, a system that predicts the right score only 25% of the time would receive an <i>S1 </i>score of 0.75.</p><p>The S2 metric is slightly less intuitive than S1, but no less reasonable. It measures the average distance between the system's score and the actual score. This metric reflects the idea that a system that estimates scores close to the annotator-assigned scores should be preferred over a system whose esti­mations are further off, even if both systems estimate the correct score at the same frequency.</p><p>Finally, the S3 evaluation metric measures the average square of the distance between a system's organization score estimations and the annotator-assigned scores.<page local="9" global="237"/> The intuition behind this system is that not only should we prefer a system whose es­timations are close to the annotator scores, but we should also prefer one whose estimations are not too frequently very far away from the annotator scores. These three scores are given by:</p><footnote label="4">In particular, we note that for theoretical reasons, a kernel function must always return a non-negative value. The align­ment score function does not have this property, so we increase all alignment scores until their theoretical minimum value is 0.</footnote><doubt alpha="40.0" length="5" tooSmall="False" monospace="0.0">11N1N</doubt><p>where Ai and Ei are the annotator assigned and sys­tem estimated scores respectively for essay i, and N is the number of essays. Since many of the systems we have described assign test essays real-valued or­ganization scores, to obtain <i>Ei </i>for system S1 we round the outputs of each system to the nearest of the seven scores the human annotators were permit­ted to assign (1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0).</p><p>To test our system, we performed 5-fold cross val­idation on our 1003 essay set, micro-averaging our results into three scores corresponding to the three scoring metrics described above.</p></subsection><subsection number="7.2" title="Results and Discussion"><p><b>The average baseline. </b>As mentioned before, there is no standard baseline for organization modeling against which we can compare our systems. To start with, we employ a simple "average" baseline. <i>Avg </i>computes the average organization score of essays in the training set and assigns this score to each test set essay. Results of this baseline are shown in row 1 of Table 6. Though simple, this baseline is by no means easy-to-beat, since 41% of the essays have a score of 3, and 96% of the essays have a score that is within one point of 3.</p><p><b>Heuristic baselines. </b>Recall that we have 12 ver­sions of the two heuristic approaches to organization prediction. Space limitations preclude a discussion of the results of all these versions, so instead, to ob­tain the strongest baseline results, we show only the best results achieved by the three versions based on aligning paragraph label sequences in row 2 (Hp) and the best results achieved by the nine versions based on aligning sentence label sequences in row 3 (Hs) of Table 6. It is clear from the results that the <i>H</i><b>p </b>systems yielded the best baseline predictions under all three scoring metrics, performing signif­icantly better than both the <i>Avg </i>and <i>Hs </i>systems (p &lt; 0.01) with respect to the S2 and S3 metrics, but its S1 performance is less significant with re­spect to <i>Avg (p &lt; </i>0.1) and is indistinguishable at even the p &lt; 0.1 level from Hs.<footnote anchor="5"/> In general, how­ever, it appears to be the case that systems based on aligning paragraph label sequences achieve better results than systems that attempt to align sentence label sequences.</p><p><b>Learning-based approaches. </b>Rows 4-6 of Table 6 show the results we obtained using each of the three single-kernel systems. When compared to the best baseline, these results suggest that <i>H</i><b>p </b>is a pretty good heuristic approach to organization scoring. In fact, only one of these three learning-based sys­tems <i>(Rlnps) </i>performs better than Hp under the three scoring metrics, and in each case, the difference be­tween the two is not significant even at p &lt; 0.1. This suggests that, even though <i>Rl</i><b>nps </b>performs slightly better than Hp, the only major benefit we have ob­tained by using the linear kernel is that it has made it unnecessary for us to choose between the 12 pro­posed heuristic systems.</p><p>Considering that the second best one-kernel sys­tem, Rs, does not have access to any of the near­est neighbor features, which have already proven useful, its performance seems reasonably good in that its performance is at least better than the Avg system. This suggests that, even though Rs does not perform exceptionally, it is extracting some use­ful information for organization scoring from the heuristically assigned paragraph label sequences. The best one-kernel system, Rl<b>nps, </b>however, is significantly better than Rs with respect to all three scoring metrics, with p &lt; 0.<page local="10" global="238"/>1 for S1 and p &lt; 0.05 for S2 and S3. By contrast, it initially appears that the alignment kernel is not extracting any useful information from these paragraph sequences at all, since its S1, S2, and S3 scores are all much worse than all of the baseline systems. The second best one-kernel system Rs performs significantly better than Ra at p &lt; 0. 01 for all three scoring metrics.</p><footnote label="5">All significance tests are two-tailed paired t-tests.</footnote><table caption="Table 6: System Performance" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>System</b></p></td><td class="cell"><p><i>si</i></p></td><td class="cell"><p><i>S2</i></p></td><td class="cell"><p><i>S3</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p><i>Avg</i></p></td><td class="cell"><p>.585</p></td><td class="cell"><p><i>All</i></p></td><td class="cell"><p>.348</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>2</p></td><td class="cell"><p><i>Hp</i></p></td><td class="cell"><p>.548</p></td><td class="cell"><p>.339</p></td><td class="cell"><p>.198</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>3</p></td><td class="cell"><p><i>Hs</i></p></td><td class="cell"><p>.575</p></td><td class="cell"><p>.397</p></td><td class="cell"><p>.329</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>4</p></td><td class="cell"><p><i>Rlnps</i></p></td><td class="cell"><p>.520</p></td><td class="cell"><p>.331</p></td><td class="cell"><p>.186</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>5</p></td><td class="cell"><p><i>Rs</i></p></td><td class="cell"><p>.577</p></td><td class="cell"><p>.369</p></td><td class="cell"><p>.222</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>6</p></td><td class="cell"><p><i>Ra</i></p></td><td class="cell"><p>.686</p></td><td class="cell"><p>.519</p></td><td class="cell"><p>.429</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>7</p></td><td class="cell"><p><i>RlSfips</i></p></td><td class="cell"><p>.534</p></td><td class="cell"><p>.332</p></td><td class="cell"><p>.187</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>8</p></td><td class="cell"><p><i>Rl&amp;nps</i></p></td><td class="cell"><p>.541</p></td><td class="cell"><p>.332</p></td><td class="cell"><p>.178</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>9</p></td><td class="cell"><p><i>Rsa</i></p></td><td class="cell"><p>.517</p></td><td class="cell"><p>.325</p></td><td class="cell"><p>.177</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>10</p></td><td class="cell"><p><i>Rlsdfips</i></p></td><td class="cell"><p>.517</p></td><td class="cell"><p>.323</p></td><td class="cell"><p>.175</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Next, we explore the impact ofcomposite kernels, which allow our learners to make use of multiple types of flat and structured features. Specifically, the results shown in rows 7-9 are obtained by combin­ing two kernels at a time. These experiments reveal the surprising result that the two worst performing single-kernel systems, Rs and Ra, when combined into Rsa, yield the best two-kernel system results, which are significant with respect to the best one-kernel system results under S3 at p &lt; 0.1. This re­sult suggests that these two different methods of ex­tracting information from paragraph sequences pro­vide us with different kinds of evidence useful for organization scoring, although neither method by it­self was exceptionally useful. Though Rsa does not have any access to nearest neighbor informa­tion, it still performs significantly better than Hp at p &lt; 0. 05 under S1 and S3.</p><p>While we have already pointed out that Rsa is the best composite two-kernel system, it is not clear which of Rls<b>nps </b>and Rla<b>nps </b>is second-best. Neither system consistently performs better than the other under all three scoring metrics, and the differences between them are not significant even at p &lt; 0.1. It is clear only that Rsa is better than both, as its scores are statistically significantly better at p &lt; 0.01 with respect to Rls<b>nps </b>and Rla<b>nps </b>under at least one of the three scoring metrics in each case.</p><p>Finally, in the last row of Table 6, we combine all three kernels into one SVM learner. The most important lesson we learn from this experiment is that each of the three kernels provides the learner with a different kind of useful information, so that a composite kernel using all three sources of in­formation performs better than any system using fewer kernels. Although the improvements over the best two-kernel system (Rsa) and one-kernel sys­tem (Rl<b>nps) </b>are small, they are still statistically sig­nificant at p &lt; 0.1 under one of the scoring metrics,</p><p>S3. When we compare this combined system to the best baseline (H<b>p),</b><b> </b>we discover the improvements derived from the three-kernel system are significant improvements over it at p &lt; 0.05 and p &lt; 0.01 with respect to S1 and S3 respectively.</p><p><b>Feature analysis. </b>To better understand which of the three flat features (nearest neighbors, paragraph label sequences, or sentence label sequences) con­tributes the most to the linear kernel portion of the systems' performances, we analyze the three fea­ture types on Rl<b>nps </b>using the backward elimination feature selection algorithm. First, we remove each of the three feature groups independently from the Rlnps's feature set and determine which of the three removals yields the best performance according to each scoring metric. Next, among the remaining two feature groups, we repeat the same step, remov­ing each of the two groups independently from the feature set to determine which of the two removals yields the best performance.</p><p>While space limitations preclude showing the ac­tual numbers, the trend is consistent among all three scoring metrics: the first feature type to remove is paragraph sequences (meaning that they are the least important) and the last to remove is the near­est neighbor features. Nevertheless, performance al­ways drops when a feature type is removed, indicat­ing that all three feature types contribute positively to overall performance. The fact that flat paragraph sequence features proved to be least useful high­lights the importance of the structured methods we presented for using paragraph sequence information.</p></subsection></section><section number="8" title="Conclusions"><p>We have investigated the relatively less-studied problem of modeling the organization in student es­says. The contributions of our work include the novel application of two techniques from bioinfor-matics and machine learning — sequence align­ment and string kernels, as well as the introduc­tion of alignment kernels — to essay scoring. We showed that each technique makes a significant con­tribution to a scoring system, and we hope that this work will increase awareness ofthese powerful tech­niques among NLP researchers. Finally, to stimulate work on this problem, we make our corpus of anno­tated essays available to other researchers.</p><page local="11" global="239"/></section><section title="Acknowledgments"><p>We thank the three reviewers for their comments. Our six annotators, Andrew Hubbs, Karin Khoo, Jayne Koath, Christopher Maier, Andrew Mallon, and Cory Thornton, all deserve numerous thanks, because without the countless hours they each spent annotating hundreds of essays, none of the research described in this paper would have been possible.</p></section><references><p>Yigal Attali and Jill Burstein. 2006. Automated es­say scoring with e-rater V.2. <i>Journal of Technology, Learning, and Assessment, </i>4(3).</p><p>Regina Barzilay and Mirella Lapata. 2005. Modeling local coherence: An entity-based approach. In <i>Pro­ceedings of the ACL, </i>pages 141-148.</p><p>Regina Barzilay and Lillian Lee. 2002. Bootstrapping lexical choice via multiple-sequence alignment. In <i>Proceedings ofEMNLP, </i>pages 164-171.</p><p>Regina Barzilay and Lillian Lee. 2003. Learning to paraphrase: An unsupervised approach using multiple-sequence alignment. In <i>HLT-NAACL 2003: Main Pro­ceedings, </i>pages 16-23.</p><p>Regina Barzilay and Lillian Lee. 2004. Catching the drift: Probabilistic content models, with applications to generation and summarization. In <i>HLT-NAACL 2004: Main Proceedings, </i>pages 113-120.</p><p>Razvan Bunescu and Raymond Mooney. 2005. A short­est path dependency kernel for relation extraction. In <i>Proceedings ofHLT/EMNLP, </i>pages 724-731.</p><p>Jill Burstein, Martin Chodorow, and Claudia Leacock. 2004. Automated essay evaluation: The <i>Criterion </i>on­line writing evaluation service. <i>AI Magazine, </i>25(3), 27-36.</p><p>Jill Burstein, Daniel Marcu, and Kevin Knight. 2003. Finding the write stuff: Automatic identification of discourse structure in student essays. <i>IEEE Intelligent Systems, </i>18(1):32-39.</p><p>Michael Collins and Nigel Duffy. 2002. New ranking algorithms for parsing and tagging: Kernels over dis­crete structures, and the voted perceptron. In <i>Proceed­ings ofthe ACL, </i>pages 263-270.</p><p>Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. <i>Machine Learning, </i>20(3):273-297.</p><p>Scott Elliot. 2001. IntelliMetric: From here to validity. Paper presented at the annual meeting ofthe American Educational Research Association, Seattle, WA.</p><p>Micha Elsner, Joseph Austerweil, and Eugene Charniak. 2007. A unified local and global model for discourse coherence. In <i>NAACL HLT 2007: Proceedings of the Main Conference, </i>pages 436-443.</p><p>Sylviane Granger, Estelle Dagneaux, Fanny Meunier, and Magali Paquot. 2009. <i>International Corpus of Learner English (Version 2). </i>Presses universitaires de Louvain.</p><p>Derrick Higgins, Jill Burstein, Daniel Marcu, and Clau­dia Gentile. 2004. Evaluating multiple aspects of co­herence in student essays. In <i>HLT-NAACL 2004: Main Proceedings, </i>pages 185-192. Thorsten Joachims.   1999.   Making large-scale SVM learning practical. In Bernhard Scholkopf and Alexan­der Smola, editors, <i>Advances in Kernel Methods - Sup­port Vector Learning, </i>pages 44-56. MIT Press. Thomas K. Landauer, Darrell Laham, and Peter W. Foltz.  2003.  Automated scoring and annotation of essays with the Intelligent Essay AssessorTM. In <i>Auto­mated Essay Scoring: A Cross-Disciplinary Perspec­tive, </i>pages 87-112. Huma Lodhi, Craig Saunders, John Shawe-Taylor, Nello Cristianini, and Christopher J. C. H. Watkins. 2002.</p><p>Text classification using string kernels. <i>Journal ofMa-chine Learning Research, </i>2:419-444. Eleni Miltsakaki and Karen Kukich. 2004. Evaluation of text coherence for electronic essay scoring systems. <i>Natural Language Engineering, </i>10(1):25-55. Alessandro Moschitti. 2004. A study on convolutionker­nels for shallow statistic parsing. In <i>Proceedings ofthe</i> <i>ACL, </i>pages 335-342.</p><p>Saul Ben Needleman and Christian Dennis Wunsch. 1970. A general method applicable to the search for similarities in the amino acid sequence oftwo proteins. <i>Journal of Molecular Biology, </i>48(3):443-453, March. Mark Shermis and Jill Burstein. 2003. <i>Automated Essay Scoring: A Cross-DisciplinaryPerspective. </i>Lawrence Erlbaum Associates, Inc., Mahwah, NJ. Mark Shermis, Jill Burstein, Derrick Higgins, and Klaus Zechner.  2010. Automated essay scoring: Writing assessment and instruction. In <i>International Encyclo­pedia ofEducation (3rd edition), </i>pages 20-26. Tony Silva. 1993. Toward an understanding of the dis­tinct nature of L2 writing: The ESL research and its implications. 27(4):657-677. Radu Soricut and Daniel Marcu. 2006. Discourse gener­ation using utility-trained coherence models. In <i>Pro­ceedings of the COLING/ACL 2006 Main Conference Poster Sessions, </i>pages 803-810. Yannick Versley, Alessandro Moschitti, Massimo Poe-sio, and Xiaofeng Yang. 2008. Coreference systems based on kernels methods. In <i>Proceedings ofCOL-</i><b>ING, </b>pages 961-968. Yiming Yang and Jan O. Pedersen. 1997. A comparative study on feature selection in text categorization. In <i>Proceedings ofICML, </i>pages 412-420.</p></references></body></article>