<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="900"/><title>Statistical Estimation of Word Acquisition with Application to Readability Prediction</title><author surname="Kidwell" givenname="Paul"><org  name="Purdue University" country="USA" city="West Lafayette"/></author><author surname="Lebanon" givenname="Guy"><org  name="Georgia Institute of Technology" country="USA" city="Atlanta"/></author><author surname="Collins-Thompson" givenname="Kevyn"><org  name="Microsoft Research" country="USA" city="Redmond"/></author></firstpageheader><frontmatter><p><b>Statistical Estimation of Word Acquisition with Application to Readability Prediction</b></p><p><b>Paul Kidwell</b></p><p>Department of Statistics Purdue University West Lafayette, IN</p><p>kidwe11pau1Ogmai1.c om</p><p><b>Guy Lebanon</b></p><p>College of Computing rgia Institute of Technology Atlanta, GA</p><p>lebanonOcc.gatech.edu</p><p><b>Kevyn Collins-Thompson</b></p><p>Microsoft Research Redmond, WA</p><p>kevynctomicrosoft.com</p></frontmatter><abstract>Models of language learning play a cen­tral role in a wide range of applica­tions: from psycholinguistic theories of how people acquire new word knowledge, to information systems that can automati­cally match content to users' reading abil­ity. We present a novel statistical ap­proach that can infer the distribution of a word's likely acquisition age automati­cally from authentic texts collected from the Web. We then show that combining these acquisition age distributions for all words in a document provides an effective semantic component for predicting read­ing difficulty of new texts. We also com­pare our automatically inferred acquisition ages with norms from existing oral stud­ies, revealing interesting historical trends as well as differences between oral and written word acquisition processes. </abstract></header><body><section number="1" title="Introduction"><p>Word acquisition refers to the temporal process by which children learn the meaning and understand­ing of new words. Some words are acquired at a very early age, some are acquired at early pri­mary school grades, and some are acquired at high school or even later in life as the individual under­goes experiences related to that word. A related concept to acquisition age is document grade level readability which refers to the school grade level of the document's intended audience. It applies in situations where documents are written with the expressed intent of being understood by children in a certain school grade. For example, textbooks authored specifically for fourth graders are said to have readability grade level four.</p><p>We develop and evaluate a novel statistical model that draws a connection between document grade level readability and age acquisition distri­butions. Based on previous work in the area, we define a model for document readability using a logistic Rasch model and the quantiles of the ac­quisition age distributions. We then proceed to in­fer the age acquisition distributions for different words from document readability data collected by crawling the web.</p><p>We examine the inferred acquisition distribu­tions from two perspectives. First, we analyze and contrast them with previous studies on oral word acquisition, revealing interesting historical trends as well as differences between oral and written word acquisition processes. Second, the inferred acquisition distributions serve as parameters for the readability model, which enables us to predict the readability level of novel documents.</p><p>To our knowledge, this is the first published study of a method to 'reverse-engineer' individ­ual word acquisition statistics from graded texts. By obtaining such a fine-grained model of how language evolves over time, we obtain a new, rich source of semantic features for a document. The increasing amounts of content available from the Web and other sources also means that these flexible models of authentic usage can be eas­ily adapted for different tasks and populations. Our work serves to complement the growing body of research using statistics and machine learn­ing for language learning tasks, and has appli­cations including predicting reading difficulty for Web pages and other non-traditional documents, reader-specific example and question generation for lexical practice in intelligent tutoring systems, and analysis tools for language learning research.</p></section><section number="2" title="A Model for Document Readability and Word Acquisition"><p>For a fixed word and a fixed population of indi­viduals T the age of acquisition (AoA) distribu­tion <i>pw</i><i> </i>represents the age at which word <i>w</i><i> </i>was</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">900</doubt><p><i>Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, </i>pages 900-909, Singapore, 6-7 August 2009. ©2009 ACL and AFNLP<page local="2" global="901"/></p><p>acquired by the population. Existing AoA norm studies almost universally summarize AoA ratings in terms of two parameters: mean and standard deviation, ignoring higher-level moments such as skew. For direct comparison with these studies we follow this convention and thus our goal is to esti­mate AoA for a word <i>w</i><i> </i>in terms of mean <i>ßw</i><i> </i>and standard deviation <i>aw</i><i> </i>parameters using the (trun­cated) normal distribution where the proportionality constant ensures that the distribution is normalized over the range of ages under consideration e.g., <i>t </i>G [6,18] for school grades. It is important to note that our model is not restricted by the assumption of (1) and can be readily extended to the Gamma family of distribu­tions, if modeling asymmetric spread in the distri­bution is appropriate.</p><doubt alpha="43.8" length="16" tooSmall="False" monospace="0.0">e-{t-tiw)2/{2al)</doubt><doubt alpha="40.7" length="27" tooSmall="False" monospace="0.0">pw{t)ocN(t;nw,aw) =-- - (1)</doubt><p>For a fixed vocabulary <i>V </i>of distinct words the age acquisition distributions for all words <i>w </i>G <i>V </i>are defined using <i>2\V\ </i>parameters</p><doubt alpha="41.2" length="17" tooSmall="False" monospace="0.0">{(Hw,aw):weV}.(2)</doubt><p>These parameters, which are the main objects of interest, can in principle be estimated from data using standard statistical techniques. Unfortu­nately, data containing explicit acquisition ages is very difficult to obtain reliably. Explicit word ac­quisition data is based on interviewing adults re­garding their age acquisition process during child­hood and so may be unreliable and difficult to ob­tain for a large representative group of people.</p><p>On the other hand, it is possible to reliably col­lect large quantities of readability data defined as pairs of documents and ages of intended audience. As we demonstrate later in the paper, such data may be automatically obtained by crawling spe­cialized resources on the Web. We demonstrate how to use such data to estimate the word acqui­sition parameters (2) and to use the estimates to predict future readability ages.</p><p>Traditionally, document readability has been defined in terms of the school grade level at which a large portion of the words have been acquired by most children (Chall and Dale, 1995). We pro­pose the following interpretation ofthat definition, which is made appropriate for quantitative studies by taking into account the inherent randomness in the acquisition process.</p><p><b>Definition 1. </b><i>A</i><i> </i><i>document</i><i> d = (w\,... ,wm) </i><i>is</i><i> </i><i>said</i><i> </i><i>to</i><i> </i><i>have</i><i> </i>(1 — <i>e\, </i>1 — £2)<i>-readability</i><i> </i><i>level</i><i> </i><i>t</i><i> </i><i>if</i><i> </i><i>by</i><i> </i><i>age</i><i> </i><i>t</i><i> </i><i>no</i><i> </i><i>less</i><i> </i><i>than</i><i> </i>1 — £1 <i>percent</i><i> </i><i>of</i><i> </i><i>the</i><i> </i><i>words</i><i> </i><i>in</i><i> </i><i>d</i><i> </i><i>have</i><i> </i><i>been</i><i> </i><i>acquired</i><i> </i><i>each</i><i> </i><i>by</i><i> </i><i>no</i><i> </i><i>less</i><i> </i><i>than</i><i> </i>1 — £2 <i>percent</i><i> </i><i>of</i><i> </i><i>the</i><i> </i><i>population.</i></p><p>We denote by <i>qw</i><i> </i>the quantile function of the cdf corresponding to the acquisition distribution <i>pw.</i><i> </i>In other words, <i>qw{r) </i>represents the age at which <i>r</i><i> </i>percent of the population <i>T</i><i> </i>have acquired word <i>w. </i>Despite the fact that it does not have a closed form, it is a continuous and smooth function of the parameters <i>fj,w, </i><i>aw</i><i> </i>in (1) (assuming <i>T</i><i> </i>is infinite) and can be tabulated before inference begins.</p><p>Following Definition 1 we define a logistic Rasch readability model:</p><p><i>P(d </i>is <i>(s, </i>r)-readable at age <i>t)</i> where <i>qd(s, r) </i>is the <i>s </i>quantile of <i>{qWi{r) :</i><i> i = </i>1,..., m}. An equivalent formulation to (3) that makes the probability model more explicit is <i>P(d </i>is (s, r)-readable at age <i>t)</i></p><doubt alpha="25.0" length="12" tooSmall="False" monospace="0.0">log--------—</doubt><doubt alpha="62.5" length="32" tooSmall="False" monospace="0.0">1 —P{dis(s, r)-readable at aget)</doubt><doubt alpha="29.4" length="17" tooSmall="False" monospace="0.0">= 9(qd(s,r)-t)(3)</doubt><doubt alpha="44.4" length="18" tooSmall="False" monospace="0.0">=exp(9(qd(s,r)-t))</doubt><doubt alpha="58.8" length="17" tooSmall="False" monospace="0.0">l+eMO(Qd(s,r)-t)Y</doubt><p>In other words, the probability of a document <i>d</i><i> </i>being (s, <i>r)</i>-readable increases exponentially with <i>Qd(s,r) </i>which is the age at which <i>s </i>percent of the words in <i>d</i><i> </i>have been acquired each by <i>r</i><i> </i>percent of the population.</p><p>The parameter <i>r</i><i> </i>= 1 — £2 determines what it means for a word to be acquired and is typically considered to be a high value such as 0.8. The parameter <i>s</i><i> </i><i>=</i><i> </i>1 — <i>e\</i><i> </i>determines how many of the document words need to be acquired for it to be readable. It can be set to a high value such as 0.9 if a very precise understanding is required for readability but can be reduced when a more mod­est definition of readability applies.</p><p>We note that due to the discreteness of the set {&lt;?«&gt;; (j) : i = 1, • • •, m}, neither <i>qd(s, r) </i>nor the loglikelihood are differentiable in the parameters (2). This raises some practical difficulties with respect to the computational maximization of the likelihood and subsequent estimation of (2). How­ever, for long documents containing a large num­ber of words, <i>qd(s, r) </i>is approximately smooth which motivates a maximum likelihood procedure using gradient descent on a smoothed version of <i>qd(s).</i><page local="3" global="902"/><i> </i>Alternative optimization techniques which do not require smoothness may also be used.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">901</doubt><p>In the case of a normal distribution (1) we have that a word is acquired by <i>r </i>percent of the pop­ulation at age <i>w = ß </i>+ $_1(r)cr, where $ is the cumulative distribution function (cdf) of the normal distribution. To investigate the distribu­tion of acquisition ages we assume that the /x, <i>a </i>parameters corresponding to different words in a document are drawn from Gamma distributions <i>ß ~ G(cti,ßi) </i>and <i>a ~ G(a2,ß2)- </i>The normal and Gamma distributions are chosen in part be­cause they are flexible enough to model many sit­uations and also admit good statistical estimation theory. Noting that $_1(r)(T ~ <i>G(a2,^~<footnote anchor="1"/>(r)ß2), </i>we can write the distribution of the acquisition ages as the following convolution <i>f<footnote anchor="1"/></i><i> tai~le Wz</i> which reverts to a Gamma when <i>ß\</i><i> </i><i>=</i><i> </i><i>ß2.</i><i></i></p><doubt alpha="50.0" length="14" tooSmall="False" monospace="0.0">wai+a2-1g—w/ß2</doubt><doubt alpha="45.5" length="22" tooSmall="False" monospace="0.0">fw{w)=T(ai)T(a2)ß^ß^ *</doubt><doubt alpha="44.4" length="9" tooSmall="False" monospace="0.0">(ß1-ß2)tw</doubt><doubt alpha="37.5" length="8" tooSmall="False" monospace="0.0">/—7-n-dt</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">JO</doubt><p>The distribution of the s-percentile of <i>fw, </i>which amounts to (r, s)-readability of documents, can be analyzed by combining <i>fw </i>above with a standard normal approximation of order statistics (e.g., (David andNagaraja, 2003))</p><p><b>V </b><i>m\fw{Fw (p</i><i>WJ</i></p><p>where m is the document length and <i>Fw</i><i> </i>is the cdf corresponding to <i>fw-</i></p><p>Figure 1 shows the relationship between docu­ment length and confidence interval (CI) width in readability prediction. It contrasts the CI widths for model based intervals and empirical intervals. In both cases, documents of lengths larger than 100 words provide CI widths shorter than 1 year. This finding is also noteworthy as it provides empirical support for the long-standing 'rule-of-thumb' that readability measures become unreli­able for passages of less than 100 words (Fry, 1990).</p></section><section number="3" title="Experimental Results"><p>Our experimental study is divided into three parts. The first part examines the word acquisition dis­tributions that were estimated based on readabil­ity data. The second part compares the estimated</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">3.0-</doubt><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">2.5-</doubt><doubt alpha="0.0" length="14" tooSmall="False" monospace="0.0">50 100 150 200</doubt><p>Document Length</p><p>Figure 1: A comparison of model (dashed) vs. em­pirical (solid) 95% confidence interval widths as a function of document length (r = 0.9 and <i>s = </i>0.7). CI widths were computed using 1000 Monte Carlo samples generated from the <i>fw </i>model fit to the data and from the empirical distribution. Word distributions correspond to a 1577 word document written for a 7th grade audience taken from the Web 1-12 corpus.</p><p>(written) acquisition ages with oral acquisition ages obtained from interview studies reported in the literature. The third part focuses on using the estimated word acquisition distributions to predict document readability. These three experimental studies are described in the three subsections be­low.</p><p>In our experiments we used three readability datasets. The corpora were compiled by crawl­ing web pages containing documents authored for audiences of specific grade levels. The Web 1-12 data contains 373 documents, with each doc­ument written for a particular school grade level in the range 1-12. The Weekly Reader (WR) dataset, was obtained by crawling the commercial website www.wrtoolkit.com after receiving spe­cial permission. That dataset contains a total of 1780 documents, with 4 readability levels rang­ing from 2 to 5 indicating the school grade lev­els of the intended audience. A total of 788 doc­uments with readability between grades 2 and 5 and having length greater than 50 words were se­lected from 1780 documents. The Reading A-Z dataset, contains a set of 215 documents was ob­tained from Reading A-Z.com, spanning grade 1 through grade 6.</p><p>The grade levels in these three corpora, which correspond to US school grades, were either ex­plicitly specified by the organization or authors who created the text, or implicit in the class­room curriculum page where the document was acquired.<page local="4" global="903"/> The pages were drawn from a wide range of subject areas, including history, science, geography, and fiction.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">902</doubt><p>To reduce the possibility of overfitting, we used a common feature selection technique of eliminat­ing words appearing in less than 4 documents. In the experiments we used maximum likelihood to estimate the model parameters cr^) <i>'■</i><i> </i><i>w</i><i> </i>G <i>V}</i><i> </i>for the Rasch model (3). The maximum likeli­hood was obtained using a non-smooth coordinate descent procedure.</p><subsection number="3.1" title="Estimation of Word Acquisition Distributions"><p>Figure 2 displays the inferred age acquisition dis­tributions and empirical word appearances of three words: thought (left), multitude (middle), and assimilate (right). In these plots, the em­pirical cdf of word appearances is indicated by a piecewise constant line while the probability den­sity function of the estimated AoA distribution is indicated by a dashed line. The vertical line in­dicates the 0.8 quantile of the AoA distribution which corresponds to the grade by which 80% of the children have acquired the word.</p><p>The word assimilation appears in 2 doc­uments having 12th grade readability. The high grade level of these documents results in a high es­timated acquisition age and the paucity of observa­tions leads to a large uncertainty in this estimate as seen by the variance of the acquisition age distri­bution. The word thought appears several times in multiple grades. It is first observed in the 1st grade and not again until the 4th grade resulting in an estimated acquisition age falling between the two. The variance of this acquisition distribution is relatively small due to the frequent use of this word. The empirical cdf shows that multitude is used in grades 6, 8, and 9. Relative to thought and assimilation the word multitude was used less and more frequently respectively, which leads to an acquisition age distribution with a larger variance than that of thought and smaller than that of assimilation.</p><p>The relationship in Figure 2 between the em­pirical word appearances and the age acquisition distribution demonstrates the following behavior: (a) The variance of the age acquisition distribu­tion goes down as the word appears in more documents, and (b) the mean of the AoA distribution tends to be lower than the mean of the empirical word appearance distribution, and in many cases even smaller than the first grade in which the word appeared. This is to be expected as authors use specific words only after they believe the words were acquired by a large portion of the intended audience.</p></subsection><subsection number="3.2" title="Comparison with Oral Studies"><p>Among the related work in the linguistic commu­nity, are several studies concerning oral acquisi­tions of words. These studies estimate the age at which a word is acquired for oral use based an interview processes with participating adults. We focus specifically on the seminal study of ac­quisition ages performed by Gilhooly and Logie (GL) (1980) and made available through the MRC database (Coltheart, 1981).</p><p>There are some substantial differences between these previous studies and our approach. We an­alyze the age acquisition process through docu­ment readability which leads to a written, rather than oral, notion of word acquisition. Further­more, our estimates are based on documents writ­ten with a specific audience in mind, while the pre­vious studies are based on interviewing adults re­garding their childhood word acquisition process which is arguably less reliable due to the age dif­ference between the acquisition and the interview. Finally, the GL study was performed in the late 1970s while our study uses contemporary internet data. Conceivably, the word acquisition process changed over the past 3 decades.</p><p>Despite these differences, it is interesting to contrast our inferred age acquisitions with the GL study and consider the differences and similari­ties. Figure 3 displays the relationship between the GL age of acquisition (AoA) and the acquisi­tion ages obtained from readability data based on the <i>s = </i>0.8 quantile. Some correlation is present (r<footnote anchor="2"/> = 0.34) but the two measures differ consid­erably. As expected, the acquisition ages obtained from written readability data tend to be higher than the oral studies. The distributions of differences between the GL acquisition ages and the ones in­ferred from the readability data appears in Fig­ure 4.</p><p>Comparing the acquisition ages obtained from readability data to the GL study results in a mean absolute error of 0.9 to 1.5, depending on the spe-<page local="5" global="904"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">903</doubt><doubt alpha="46.7" length="15" tooSmall="True" monospace="0.0">mu: 9sigma: 3.4</doubt><doubt alpha="0.0" length="23" tooSmall="False" monospace="0.0">5 10 15 5 10 15 5 10 15</doubt><p>Figure 2: A comparison of empirical word appearances and AoA distributions for three words: thought (left), multitude (middle), and assimilation (right). The empirical cdf of word ap­pearances appears as a piecewise constant line and the estimated pdf is indicated by the dashed curve with its 0.8 quantile indicated by a vertical line.</p><p>cine value of the Rasch parameter <i>9. </i>Interestingly, the tendency for the written acquisition age to ex­ceed the oral one diminishes as the grade level in­creases. This represents the notion that at higher grades words are acquired in both oral and written senses at the same age.</p><p><b>Predicted versus Oral Acquisition Age</b></p><doubt alpha="83.3" length="6" tooSmall="False" monospace="0.0">GL AoA</doubt><p>Figure 3 : A scatter plot <i>(s = </i>80, <i>n = </i>50) of pre­dicted age of acquisition versus Gilhooly and Lo-gie's values reveals the tendency for the written estimate to exceed the oral estimate (r<footnote anchor="2"/> = 0.34).</p><p>A comparison to two more recent studies con­firms relationships that are similar to those ob­served with GL AoA. The Bristol Norm study (Stadthagen-Gonzalez and Davis, 2006) was per­formed in an identical way to the GL study and comparing the lists of acquisition ages results in a mean absolute error of approximately 0.5 which is much lower than the .9 to 1.5 relative to GL. The recent AoA list of Cortese and Khanna (2008) showed an increase in correlation relative to the GL study (r<footnote anchor="2"/> = 0.43) potentially reflecting change in the acquisition process due to temporal effects.</p><doubt alpha="82.4" length="68" tooSmall="True" monospace="0.0">Residual Distribution: Predicted AoA versus Oral AoA S-percentile=80</doubt><doubt alpha="0.0" length="6" tooSmall="True" monospace="0.0">20-15-</doubt><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">0I-</doubt><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">110-</doubt><doubt alpha="0.0" length="2" tooSmall="True" monospace="0.0">5-</doubt><doubt alpha="7.1" length="28" tooSmall="False" monospace="0.0">o-   ■-1-&gt;-&gt;—&gt;-&gt;-&gt;—&gt;-&gt;-&gt;—L -</doubt><doubt alpha="0.0" length="11" tooSmall="True" monospace="0.0">-4 -2 0 2 4</doubt><doubt alpha="76.5" length="34" tooSmall="True" monospace="0.0">Error (Predicted AoA - Actual AoA)</doubt><p>Figure 4: The difference distribution between the GL and the inferred AoA from Web 1-12 is skewed to the right as would be expected since written AoA is higher than oral AoA. Relaxing the definition of readability by decreasing <i>s </i>re­sults in higher inferred acquisition ages. Values of s in [0.5,0.9] produced reasonable results, with <i>s = </i>0.65 achieving smallest mean absolute error.</p><p>Those words that have the same written and verbal acquisition age are partially attributable to those words learned prior to first grade. Many words are learned between the ages of 2 and 5, while reading materials are typically not assigned a grade level of less than 1 or age 6. Approxi­mately 40% of the words assigned the same grade level by both Gilhooly and our prediction had an AoA of 1st grade.</p><p>In some cases, the ages of acquisition obtained from readability data is actually lower than the ages reported in the older oral studies. This phe­nomenon is likely caused by a combination of a shift in educational standards, a change in so­cial standards, or estimation errors due to sample size and modeling assumptions. Approximately 30 years have passed since Gilhooly and Logie's study was conducted.<page local="6" global="905"/> Specifically, society has made efforts to enhance the safety and health of children and to increase the attention to science education in very early grades. For example, the word drug appeared in writing 0.94 grades ear­lier than the age in which it was acquired orally according to the GL study. The newer Bristol Norm study confirms this observation as it pre­dicts a decrease in grade level for drug of 0.88 over GL as well. A similar decrease in acqui­sition age relative to the GL norms was noted for many other words such as hypothesis, conclusion, engineer, diet, exercise, and vitamin.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">904</doubt></subsection><subsection number="3.3" title="Global Readability Prediction"><p>Once acquisition age distributions are available, whether estimated statistically from data or ob­tained from a survey, they may be used to predict the grade level of novel documents. Specifically, the model predicts readability level <i>t*</i><i> </i>for a novel document <i>d </i>if it is the minimal grade for which readability is established:</p><doubt alpha="61.0" length="41" tooSmall="False" monospace="0.0">t*=min{i :P(dis readable at aget) &gt; ß{t)}</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(5)</doubt><p>where <i>ß{t) </i>is a parameter describing the strictness of the readability requirement. Note that we allow <i>ß(t) </i>to vary as a function of time (grade level). We discuss the justification for this below.</p><p>A critical issue for reading difficulty predic­tion is how to handle words that appear in a new document that have never been seen in the train­ing/development texts. In a statistical approach, the solution to this smoothing problem has two steps. First, we must decide how much total proba­bility mass to allocate to all unknown words. Sec­ond, we must decide how to subdivide this total mass for individual words or classes of words us­ing word-specific priors.</p><p>Our experience suggests that the first step of estimating total probability mass is particularly important: the likelihood of seeing an unknown word increases as a function of total vocabulary size, which is continuously growing with time. We model this by defining the following dynamic threshold</p><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">ß{t)</doubt><doubt alpha="30.3" length="33" tooSmall="False" monospace="0.0">exp(ai — 0.5) 1 + exp(ai — 0.5) '</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(6)</doubt><p>We learn the growth rate parameter <i>a </i>in (6) from the data at the same time as we learn the read­ability model's quantile parameters <i>s</i><i> </i><i>=</i><i> </i>1 <b>— </b><i>e\,</i><i> r = </i>1 <b>— </b>Ê2. The range of the resulting <i>ß(t) </i>is typically 0.5 in lower grades, increasing to 0.9 in higher grades. We discuss fitting these parameters and their optimal values further in Sec. 3.3.1. We found that using any fixed <i>ß </i>value for all grades was generally much less effective than a dynamic <i>ß{t) </i>threshold, and so we focus on the latter in our evaluation.</p><p>For the second (word-specific) smoothing step, we simply assign uniform probability across grades, once the total unseen mass is determined. More sophisticated word-specific priors incorpo­rating word length, morphological features, se­mantic clusters and so on are certainly possible and an interesting direction for future work.</p><p>In the following section we conduct three exper­iments involving readability prediction. First, we confirm the effectiveness of the AoA-based model compared to other predictive models. Second, we examine how prediction effectiveness is affected when our learned (written) acquisition ages are re­placed with existing oral AoA norms. Third, we examine the ability of our model to generalize to new content by training and testing on different (non-overlapping) corpora.</p><subsubsection number="3.3.1" title="Effectiveness of Readability Prediction"><p>In order to assess the effectiveness of our model in predicting the readability grade levels of novel documents we apply the model to two corpora. First, we use the Web 1-12 corpus to learn opti­mal parameter values for <i>a , r, </i>and <i>s </i>and then as­sess prediction error using a test-training paradigm for the proposed model, Naive Bayes, and support vector regression. Second, the trained model is ap­plied with to the Reader A-Z corpus and the results are compared with alternative semantic variables. Because corpora can vary significantly in text ho­mogeneity, amount of noise, document size, and other factors, training and testing across different corpora - rather than relying on cross-validation with a single pooled dataset - gives valuable in­formation about how a prediction method might be expected to perform on data with widely differ­ent characteristics. This particular choice of Web 1-12 for training and ReadingA-Z for testing was arbitrary.</p><p>To evaluate the best values for the <i>a </i>parameter in (6) and <i>s, r </i>parameters in Definition 1 we gen-<page local="7" global="906"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">905</doubt><p>Figure 5 : Mean absolute error (MAE) and correla­tion coefficient as functions of the quantile param­eter <i>s </i>at optimal levels of <i>a </i>and <i>r, </i>averaged over 100 training/test samples. The MAE is displayed as the solid line and is aligned with the left axis while the correlation is displayed as a dashed line and is aligned with the right axis. 90% bootstrap confidence intervals are displayed.</p><p>erated 100 independent test and training samples and computed the mean absolute prediction error (MAE) and the correlation coefficient between the predicted and actual levels. Figure 5 (left) shows these two quantities: in each group of three lines, the top and bottom lines delineate the upper and lower 90% confidence bounds for the middle line. Each middle line gives mean error or correlation as a function of the quantile parameter <i>s </i>at opti­mal levels of <i>r </i>and a, averaged over the 100 train­ing/test samples. The optimal value of <i>s </i>for both quantities is around 0.6 (0.65 for the MAE). The optimal value for parameter <i>a </i>was approximately 1.55. The best MAE is 1.4 which compares favor­ably to the 2.92 MAE obtained by always predict­ing Grade 6 which is the optimal "dumb" classifier in the sense that of all constant predictors it pro­vides the smallest expected MSE over a uniform grade distribution as is the case with the Webl-12 corpus. Figure 6 is a scatter plot comparing predicted grades vs. actual grades, with a strong correlation of 0.89.</p><p>We compared the predictions of model (3) to two standard classifiers: naive Bayes and support vector regression (SVR). SVR was applied twice using different sets of features - once with the doc­ument word frequencies and once with the esti-</p><p>Figure 6: The scatter plot demonstrates the strong relationship between predicted and actual global readability levels.</p><p>Figure 7: A comparison of mean absolute error (MAE) across prediction algorithms shows the age of acquisition model compares favorably. The confidence bounds (LB,UB) were computed by re­peating each model building procedure 100 times.</p><p>mated AoA percentiles for the document words. The document word frequency vector is compa­rable to the semantic component of the machine learning approach used by (Heilman et al., 2008). The 75-25 training-test model building paradigm was used over documents from grades 1 to 12 to obtain predicted values. The MAE for these predictors and their 90% confidence intervals are shown in Figure 7. Predicting readability using word frequencies had inferior performance, with the naive Bayes model performing poorly and the SVR and Rasch model obtaining MAE around 1.4.</p><p>In the second experiment, we compared our model to published correlation results (Collins-Thompson and Callan, 2005) for multiple alter­native semantic variables using the same Reading A-Z corpus, with the results shown in Fig. 8. De­tails on these semantic variables, which have been used in previous statistical learning approaches, are available in the same study. Interestingly, the correlation of the model was comparable to ex-<page local="8" global="907"/></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">906</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Prediction Rule</p></td><td class="cell"><p>MAE</p></td><td class="cell"><p>LB</p></td><td class="cell"><p>UB</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Age of Acquisition</p></td><td class="cell"><p>1.40</p></td><td class="cell"><p>1.19</p></td><td class="cell"><p>1.67</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Naive Bayes</p></td><td class="cell"><p>1.98</p></td><td class="cell"><p>1.71</p></td><td class="cell"><p>2.26</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SVR (word frequency)</p></td><td class="cell"><p>1.86</p></td><td class="cell"><p>1.69</p></td><td class="cell"><p>2.06</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>SVR (AoA percentiles)</p></td><td class="cell"><p>1.36</p></td><td class="cell"><p>1.22</p></td><td class="cell"><p>1.58</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Grade 6</p></td><td class="cell"><p>2.92</p></td><td class="cell"><p><b>-</b></p></td><td class="cell"><p><b>-</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Figure 8: Comparison of the correlation of AoA and other semantic variables with grade level for the Reading A-Z corpus, showing the AoA model with the dynamic threshold compares well to ex­isting methods. The competitor methods used are from (Collins-Thompson and Callan, 2005) and comprise the Smoothed Unigram, UNK (rel­ative to revised Dale-Chall), TYPE (number of unique words), MLF (mean log frequency), and FK (Flesch-Kincaid readability).</p><p>isting variables, but did vary depending upon the source of AoA. Note that because the Reading A-Z texts were assigned grades by their creators using some of the same semantic variables (e.g. Type), it is not surprising that those variables perform es­pecially well on this dataset.</p><p>High quality readability prediction is a worth­while result in itself; however, we can also use the prediction mechanism to study the validity of Def­inition 1 and the Rasch model. We do so by apply­ing other predictive algorithms using the inferred acquisition age distribution for each document as the predictor variables and comparing the MAE with the MAE obtained by the estimated Rasch model. In particular, we examine the performance of support vector regression (SVR) using the esti­mated AoA percentiles for each document as pre­dictor variables. The results displayed in Fig­ure 7 show that SVR and the dynamic threshold prediction rule perform similarly well, suggesting that Definition 1 and the Rasch model are suitable models for readability prediction.</p></subsubsection><subsubsection number="3.3.2" title="Prediction with Existing Acquisition Age Norms"><p>We now examine how predicting readability of novel documents using acquisition ages obtained in surveys perform in comparison to the ages ob­tained from the maximum likelihood estimation.</p><p>We use the GL and Bristol age of acquisition norms. The intersection of AoA norm data and the Web Corpus are 1217 and 1012 words respectively for the GL and Bristol measure; additionally, the highest grade level associated with these word sets</p><p>Figure 9: The Gilhooly and Logie AoA norms and the Bristol norms are independent sources for ages of acquisition. A comparison of the prediction quality using these norms shows two things: 1) the definition provides comparable prediction quality using expert norms, and 2) the dynamic threshold <i>ß{t) </i>improves prediction over the static threshold (optimal s-th percentile) for the norms.</p><p>Figure 10: The readability of WR documents was predicted using 4 sources of AoA data. The pa­rameters of the prediction model were fit using only the Web data, or the WR data, or both sources in the case of the GL and Bristol norms AoA data.</p><p>are eight and seven respectively. When applying the prediction rule using AoA norms <i>r </i>is implic­itly selected in the norming process as the result is a single value instead of a distribution. Interest­ingly, the optimal ranges of s-percentile, from 92 to 100, were the same for both the GL and Bristol norms. Table 9 shows that the prediction accuracy obtained using the GL Norms was almost identical to that obtained with the inferred AoA, while the Bristol Norms performed as well as some of the competitor procedures.</p></subsubsection><subsubsection number="3.3.3" title="Prediction Effectiveness across Different Corpora"><p>To provide additional evidence for our model's ability to generalize to new corpora, we exam­ine how the learned <i>r </i>and <i>s </i>values vary when the model is learned on one corpus and evaluated on another, and how this affects the accuracy of the readability prediction.</p><p>Figure 10 demonstrates the corpus used for tun­ing the readability prediction has a large impact on the quality of the prediction. Comparing the MAE of the readability predictions on WR data when the age of acquisition is inferred from Web data to the MAE when the AoA is inferred from WR data shows the error rate more than doubles from 0.<page local="9" global="908"/>90 to 1.89. The increase in error rate also appears when the age of acquisition for WR data is predicted using the AoA norm data. In this case the prediction was performed using the parameters identified when the model was trained on Web data and when the model was trained on WR data. In each case a tendency to overfit appears as the MAE increases from 1.14 to 2.05 for the GL norms and 1.34 to 1.57 for the Bristol norms. Interestingly, the Bristol norms perform better on WR data when fit using the Web data, while the GL norms per­form better when fit using the WR data.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">907</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Correlation</p></td><td class="cell"><p></p></td><td class="cell"><p>Correlation</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>GL (Web)</p></td><td class="cell"><p>.65</p></td><td class="cell"><p>UNK</p></td><td class="cell"><p>.78</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>GL (WR)</p></td><td class="cell"><p>.40</p></td><td class="cell"><p>Type</p></td><td class="cell"><p>.86</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Bristol (Web)</p></td><td class="cell"><p>.76</p></td><td class="cell"><p>MLF</p></td><td class="cell"><p>.49</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Bristol (WR)</p></td><td class="cell"><p>.57</p></td><td class="cell"><p>FK</p></td><td class="cell"><p>.30</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Inferred (Web)</p></td><td class="cell"><p>.59</p></td><td class="cell"><p>Unigram</p></td><td class="cell"><p>.63</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>S-th</p></td><td class="cell"><p>Dynamic</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Prediction Rule</p></td><td class="cell"><p>Percentile</p></td><td class="cell"><p>Threshold</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Age of Acquisition</p></td><td class="cell"><p>1.69</p></td><td class="cell"><p>1.40</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>GL Norms</p></td><td class="cell"><p>1.73</p></td><td class="cell"><p>1.42</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Bristol Norms</p></td><td class="cell"><p>1.97</p></td><td class="cell"><p>1.79</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>AoA</p></td><td class="cell"><p></p></td><td class="cell"><p>Weekly</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Source</p></td><td class="cell"><p>Web 1-12</p></td><td class="cell"><p>Reader</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Inferred (Weekly Reader)</p></td><td class="cell"><p>-</p></td><td class="cell"><p>.91</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Inferred (Web 1-12)</p></td><td class="cell"><p>1.89</p></td><td class="cell"><p>-</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>GL</p></td><td class="cell"><p>2.05</p></td><td class="cell"><p>1.14</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Bristol</p></td><td class="cell"><p>1.57</p></td><td class="cell"><p>1.34</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></subsubsection></subsection></section><section number="4" title="Related Work"><p>Age of acquisition for word reading and under­standing has been extensively studied as a learn­ing factor in the psycholinguistics literature, where AoA norms have been obtained using surveys. Ex­amples of relevant literature are (Gilhooly and Lo-gie, 1980; Zevin and Seidenberg, 2002). Our ap­proach differs by connecting AoA to readability through Definition 1 and using readability data to estimate AoA norms from large amounts of au­thentic language data. A related study is that by Crossley et al. (2007) who used AoA to help dis­criminate between authentic and simplified texts for second-language readers.</p><p>In the past decade, there has been renewed in­terest in corpus-based statistical models for read­ability prediction. One example is the popular Lexile measure (Stenner, 1996) which uses word frequency statistics from a large English corpus. Collins-Thompson and Callan (2005) introduced a new approach based on statistical language mod­eling, treating a document as a mixture of lan­guage models for individual grades. Further re­cent refinements in methods for readability predic­tion include using machine learning methods such as Support Vector Machines (Schwann and Os-tendorf, 2005), log-linear models (Heilman et al., 2008), A;-NN classifiers and combining semantic and grammatical features (Heilman et al., 2007).</p><p>The growing number of features investigated by these machine learning approaches reflect the fact that reading difficulty is a complex phenomenon involving many factors, from semantic difficulty (vocabulary) to syntax and discourse complex­ity, reader background, and others. While a fullfeatured comparison between previous approaches that includes AoA features would be very inter­esting, our goal in this study was to provide a clear analysis of the most fundamental factor of readability, semantic difficulty, which accounts for 80-90% of the variance in readability prediction scores (Chall and Dale, 1995). Because AoA is a semantic, vocabulary-based representation, we compare its effectiveness with the correspond­ing <i>semantic</i><i> </i><i>components</i><i> </i>from previous machine-learning approaches in Sec. 3.3.1.</p></section><section number="5" title="Discussion"><p>While there have been several recent studies re­garding word acquisition and readability our work is the first to provide a quantitative connection be­tween these two concepts in a statistically mean­ingful way. The core assumption that we make is Definition 1 which is consistent with standard readability definitions e.g., (Chall and Dale, 1995) and states that document readability level is deter­mined by most people understanding most words.</p><p>The connection between word acquisition and readability is both intuitive and useful. It allows two degrees of freedom <i>s</i><i> </i><i>=</i><i> </i>1 <b>— </b><i>e\</i><i> </i>and <i>r</i><i> </i>= 1 <b>— </b><i>e2</i><i></i>to handle situations where different readability no­tions exist. Experiments validate the model and demonstrate interesting trends in word acquisi­tions as compared to older oral acquisition stud­ies. Experimental results show that the proposed model is also effective in terms of predicting read­ability level of documents on multiple datasets. It compares favorably to naive Bayes and sup­port vector regression, the latter being one of the strongest regression baselines.</p></section><section title="Acknowledgments"><p>The authors thank Joshua Dillon for downloading the weekly reader data and pre-processing it. The work described in this paper was funded in part by NSF grant DMS-0604486.</p></section><references><p>J. S. Chall and E. Dale. 1995. <i>Readability Revisited: The New Dale-Chall Readability Formula. </i>Brook-line Books, Brookline, MA.</p><p>K. Collins-Thompson and J. Callan. 2005. Predicting reading difficulty with statistical language models.</p><p><i>J. of the American Soc. for Info. Science and Tech., </i>56(13):598-605.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">908</doubt><page local="10" global="909"/><p>M. Coltheart. 1981. The MRC psycholinguistic database. <i>Quarterly Journal of Experimental Psy­chology, </i>33A:497-505.</p><p>M. Cortese and M. Khanna. 2008. Age acquisition ratings for 3000 monosyllabic words. <i>Behavior Re­search Methods, </i>40:791-794.</p><p>S. A. Crossley, P. M. McCarthy, and D. S. McNa-mara. 2007. Discriminating between second lan­guage learning text-types. In <i>Proc. of the Twenti­eth International Florida Artificial Intelligence Re­search Society Conference.</i></p><p>H. A. David and H. N. Nagaraja. 2003. <i>Order Statis­tics. </i>Wiley, Marblehead, MA.</p><p>E. Fry. 1990. A readability formula for short passages.</p><p><i>Journal ofReading.</i></p><p>K. J. Gilhooly and R. H. Logie. 1980. Age of acquisi­tion, imagery, concreteness, familiarity and ambigu­ity measures for 1944 words. <i>Behaviour Research Methods and Instrumentation, </i>12:395-427.</p><p>M. Heilman, K. Collins-Thompson, J. Callan, and M. Eskenazi. 2007. Combining lexical and gram­matical features to improve readability measures for first and second language texts. In <i>Proc. of the Hu­man Language Technology Conference.</i></p><p>M. Heilman, K. Collins-Thompson, and M. Eskenazi. 2008. An analysis of statistical models and features for reading difficulty prediction. In <i>The 3rd Work­shop on Innovative Use ofNLP for BuildingEduca-tional Applications.</i></p><p>S. E. Schwann and M. Ostendorf. 2005. Reading level assessment using support vector machines and sta­tistical language models. In <i>Proc. ofthe Association ofComputational Linguistics.</i></p><p>H. Stadthagen-Gonzalez and C. J. Davis. 2006. The bristol norms for age of acquisition, imageabil-ity, and familiarity.   <i>Behavior Research Methods,</i> 38:598-605.</p><p>A. J. Stenner. 1996. <i>Measuring reading comprehen­sion with the Lexile Framework. </i>Metametrics, Inc.,</p><p>Durham, NC.</p><p>J. D. Zevin and M. S. Seidenberg. 2002. Age of acqui­sition effects in word reading and other tasks. <i>Jour­nal of Memory and Language, </i>47(1): 1-29.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">909</doubt></references></body></article>