<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="700"/><title>The Imagination of Crowds: Conversational AAC Language Modeling using Crowdsourcing and Large Data Sources</title><pubinfo>Proceedingsof the 2011 Conference on Empirical Methods in Natural Language Processing,pages 700-711, Edinburgh, Scotland, UK, July 27-31, 2011. ©2011 Association for Computational Linguistics</pubinfo><author surname="Vertanen" givenname="Keith"><org  name="University of St. Andrews" country="United Kingdom" city="St. Andrews"/></author><author surname="Kristensson" givenname="Per Ola"><org  name="University of St. Andrews" country="United Kingdom" city="St. Andrews"/></author></firstpageheader><frontmatter><p><b>The Imagination of Crowds: Conversational AAC Language Modeling using</b></p><p><b>Crowdsourcing and Large Data Sources</b></p><p><b>Keith Vertanen</b></p><p>Department of Computer Science Princeton University</p><p>vertanen@princeton.edu</p><p><b>Per Ola Kristensson</b></p><p>School of Computer Science University of St Andrews</p><p>pok@st-andrews.ac.uk</p></frontmatter><abstract>Augmented and alternative communication (AAC) devices enable users with certain com­munication disabilities to participate in every­day conversations. Such devices often rely on statistical language models to improve text entry by offering word predictions. These predictions can be improved if the language model is trained on data that closely reflects the style of the users' intended communica­tions. Unfortunately, there is no large dataset consisting of genuine AAC messages. In this paper we demonstrate how we can crowd-source the creation of a large set of fictional AAC messages. We show that these messages model conversational AAC better than the cur­rently used datasets based on telephone con­versations or newswire text. We leverage our crowdsourced messages to intelligently select sentences from much larger sets of Twitter, blog and Usenet data. Compared to a model trained only on telephone transcripts, our best performing model reduced perplexity on three test sets of AAC-like communications by 60­82% relative. This translated to a potential keystroke savings in a predictive keyboard in-terfaceof 5-11%. </abstract></header><body><section number="1" title="Introduction"><p>Users with certain communication disabilities rely on augmented and alternative communication (AAC) devices to take part in everyday conversa­tions. Often these devices consist of a predictive text input method coupled with text-to-speech out­put. Unfortunately, the text entry rates provided by</p><p>AAC devices are typically low, between 0.5 and 16 words-per-minute (Trnka et al., 2009).</p><p>As a consequence, researchers have made nu­merous efforts to increase AAC text entry rates by employing a variety of improved language model­ing techniques. Examples of approaches include adapting the language model to recently used words (Wandmacher et al., 2008; Trnka, 2008), using syn­tactic information (Hunnicutt, 1989; Garay-Vitoria and Gonzalez-Abascal, 1997), using semantic in­formation (Wandmacher and Antoine, 2007; Li and Hirst, 2005), and modeling topics (Lesher and Rinkus, 2002; Trnka et al., 2006). For a recent sur­vey, see Garay-Vitoria and Abascal (2006).</p><p>While such language model improvement tech­niques are undoubtedly helpful, certainly they can all benefit from starting with a long-span language model trained on large amounts of closely matched data. For AAC devices this means closely modeling everyday face-to-face communications. However, a long-standing problem in the field is the lack of good data sources that adequately model such AAC communications. Due to privacy-reasons and other ethical concerns, there is no large dataset consist­ing of genuine AAC messages. Therefore, previous research has used transcripts of telephone conversa­tions or newswire text. However, these data sources are unlikely to be an ideal basis for AAC language models.</p><p>In this paper we show that it is possible to signif­icantly improve conversational AAC language mod­eling by first crowdsourcing the creation of a fic­tional collection of AAC messages on the Amazon Mechanical Turk microtask market. Using a care-<page local="2" global="701"/></p><p><b>Due to a medical condition or accident, imagine you can't talk or type on a normal keyboard. Instead, you use a special communication device that speaks for you. You operate this device by pushing a burton whenever your desired letter is highlighted. By repeatedly pushing the button, you can spell out words, phrases or entire sentences.</b></p><p><b>Invent a fictitious (but plausible) communication you might make using your device. Think of the things you might want to say to your family, friends, care­givers, and people you meet in the community. Please proceed quickly and accurately. Do NOT include any private information (such as real email addresses, phone numbers, or names). Invent a new communication for each task of this type.</b></p><p><b>Write as if you were actually using your communication device to speak for you. Do NOT write about your actions or state of mind.</b></p><p>fully designed microtask we collected 5890 mes­sages from 298 unique workers. As we will see, word-for-word these fictional AAC messages are better at predicting AAC test sets than a wide-range of other text sources. Further, we demonstrate that Twitter, blog and Usenet data outperform telephone transcripts or newswire text.</p><p>While our crowdsourced AAC data is better than other text sources, it is too small to train high-quality long-span language models. We therefore investi­gate how to use our crowdsourced collection to in­telligently select AAC-like sentences from Twitter, blog and Usenet data. We compare a variety of different techniques for doing this intelligent selec­tion. We find that the best selection technique is the recently proposed cross-entropy difference method (Moore and Lewis, 2010). Using this method, we build a compact and well-performing mixture model from the Twitter, blog and Usenet sentences most similar to our crowdsourced data.</p><p>We evaluate our mixture model on four different test sets. On the three most AAC-like test sets, we found substantial reductions in not only perplexity but also in potential keystroke savings when used in a predictive keyboard interface. Finally, to aid other AAC researchers, we have publicly released our crowdsourced AAC collection, word lists and best-performing language models<footnote anchor="1"/>.</p></section><section number="2" title="Crowdsourcing AAC-like Messages"><p>As we mentioned in the introduction, there are un­fortunately no publicly available sources of gen­uine conversational AAC messages. We conjectured we could create surrogate data by asking workers on Amazon Mechanical Turk to imagine they were a user of an AAC device and having them invent things they might want to say. While crowdsourcing is commonly used for simple human computation tasks, such as labeling images and transcribing au­dio, it is an open research question whether we can leverage workers' creativity to invent plausible and useful AAC-like messages. In this section, we de­scribe our carefully constructed microtask and com­pare how well our collected messages correspond to communications from actual AAC users.</p><footnote label="1">http://www.aactext.org/imagine/</footnote><figure caption="Figure 1: The interface for HITs of type 1 in our crowdsourced data collection."></figure><figure caption="Figure 2: The interface for HITs of type 2 in our crowdsourced data collection."></figure><subsection number="2.1" title="Collection Tasks"><p>To collect our data, we used two different types of human intelligence tasks (HITs). In type 1, the workers were told to imagine that due to an accident or medical condition they had to use a communica­tion device to speak for them. Workers were asked to invent a plausible communication. Workers were prevented from pasting text. After several pilot ex­periments, we arrived at the instructions shown in figure 1.</p><p>In type 2, a worker first judged the plausibility of a communication written by a previous worker (figure 2). After judging, the worker was asked to "invent a completely new communication" as if the worker was the AAC user. Workers were pre­vented from pasting text or typing the identical text as the one just judged. The same communication</p><p>Due to a medical condition or accident imagine your friend Pat can't talk or type on a normal keyboard. Instead, Pat uses a special communication device that speaks. Pat operates this de\ice by pushing a button whenever the device highlights their nest desired letter. By repeatedly pushing the button, Pat spells out words, phrases or entire sentences.</p><p>You will be presented with a short message. First you need to decide if the message is a plausible message Pat might write using such a communication device. Next you will invent your own communication as if you were Pat using the device. Think of the things you might want to say to your family, friends, care-gr.<footnote anchor="7"/>ers: and people you meet in the community-Please proceed quickly and accurately. Do NOT include any private information (such as real email addresses, phone numbers, ornâmes). Invent a new communication for each HIT of this type. Write as if you were actually using the communication device to speak for you. Do NOT write about your actions or state of mind.</p><p>Text: Are you going to club?</p><p>Is the above a plausible message that Pat may have written using the communication device'<footnote anchor="1"/></p><doubt alpha="62.5" length="8" tooSmall="False" monospace="0.0">. Yes No</doubt><p>Not sure<page local="3" global="702"/></p><p>was judged by three separate workers. In this work we did not make use of these judgments.</p></subsection><subsection number="2.2" title="Data Cleaning"><p>While most workers produced plausible and often creative communications, some workers entered ob­vious garbage. These workers were identified by a quick visual scan of the submitted communications. We rejected the work of 9% of the workers in type 1 and 4% of the workers in type 2. After removing these workers, we had 2481 communications from type 1 and 4440 communications from type 2.</p><p>After combining the data from all accepted HITs, we conducted further semi-automatic data clean­ing. We first manually reviewed communications sorted by worker. We removed workers whose text was non-fluent English or not plausible (e.g. some workers entered news headlines or proverbs). Iden­tical communications from the same worker were removed. We removed communications with an out-of-vocabulary (OOV) rate of over 20% with re­spect to a large word list of 330K words obtained from human-edited dictionaries<footnote anchor="2"/>. We also removed communications that were all in upper case, con­tained common texting abbreviations (e.g. "plz", "ru", "2day"), communications over 80 characters, and communications with excessive letter repeti­tions (e.g. "yippeeee"). After cleaning, we had 5890 messages from 298 unique workers.</p></subsection><subsection number="2.3" title="Results"><p>Tables 1 and 2 show some example communications obtained in each HIT type. Sometimes, but not al­ways, type 2 resulted in the worker writing a similar communication as the one judged. This is a mixed blessing. While it may reduce the diversity of com­munications, we found that workers were more ea­ger to accept HITs of type 2. The average HIT com­pletion time was also shorter, 24 seconds in type 2 versus 36 seconds in type 1. While we initially paid $0.04/HIT for both types, we found in subsequent rounds that we could pay $0.02/HIT for type 2. We also had to reject less work in type 2 and qualita­tively found the communications to be more AAC-like. Since workers had to imagine themselves in a</p><footnote label="2">We combined Wiktionary, Webster's dictionary provided by Project Gutenberg, the CMU pronouncing dictionary and GNU aspell.</footnote><p>Is the dog friendly?</p><p>Can I have some water please?</p><p>I need to start making a shopping list soon.</p><p>What I would really like right now is a plate of fruit.</p><p>Who will drive me to the doctor's office tomorrow?</p><table caption="Table 1: Example communications from type 1."></table><p><b>Can you bring my slippers?</b></p><p>I am cold, is there another blanket. How did Pam take the news? Bring the fuzzy slippers here. <b>Did you have breakfast? </b>why are you so late? I am pretty hungry, can we go eat? I had bacon eggs and hashbrowns for breakfast.</p><p>Table 2: Example communications from type 2. The text in bold is the message workers judged. It is fol­lowed in plain text by the workers' new messages.</p><p>very unfamiliar situation, it appears that providing a concrete example was helpful to workers.</p></subsection></section><section number="3" title="Comparison of Training Sources"><p>In this section, we compare the predictive perfor­mance of language models trained on our Turk AAC data with models trained on other text sources. We use the following training sets:</p><p>• News - Newspaper articles from the CSR-III (Graff et al., 1995) and Gigaword corpora (Graff, 2003). 60M sentences, 1323M words.</p><p>• WIKIPEDIA - Current articles and discussion threads from a snapshot of Wikipedia (January 3,</p><p>2008) . 24M sentences, 452M words.</p><p>• USENET - Messages from a Usenet corpus (Shaoul and Westbury, 2009). 123M sentences,</p><p>1847M words.</p><p>• Switchboard - Transcripts of 2217 telephone conversations from the Switchboard corpus (God­frey et al., 1992). Due to its conversational style, this corpus has been popular for AAC language modeling (Lesher and Rinkus, 2002; Trnka et al.,</p><p>2009) . 0.2M sentences, 2.6M words.</p><doubt alpha="62.7" length="83" tooSmall="False" monospace="0.0">•BLOG - Blog posts from the ICWSM corpus (Burton et al., 2009). 25M sentences, 387M</doubt><p>words.</p><page local="4" global="703"/><p>• Twitter - We collected Twitter messages via the streaming API between December 2010 and March 2011. We used the free Twitter stream which provides access to 5% of all tweets. Twit­ter may be particularly well suited for modeling AAC communications as tweets are short typed messages that are often informal person-to-person communications. Twitter has previously been proposed as a candidate for modeling conversa­tions, see for example Ritter et al. (2010). 7M sentences, 55M words.</p><p>• TurkTrain - Communications from 80% of the workers in our crowdsourced collection. 4981 sentences, 24860 words.</p><p>Wikipedia, Usenet, Blog and Twitter all consisted of raw text that required significant filter­ing to eliminate garbage, spam, repeated messages, XML tags, non-English text, etc. Given the large amount of data available, our approach was to throw away any text that did not appear to be a sensible English sentence. For example, we eliminated any sentence having a large number of words not in our</p><doubt alpha="60.7" length="28" tooSmall="False" monospace="0.0">330K word list.3.1 Test Sets</doubt><p>We evaluated our models on the following test sets:</p><p>• Comm - Sentences written in response to hy­pothetical communication situations collected by Venkatagiri (1999). We removed nine sentences containing numbers. This set is used throughout the paper. 251 sentences, 1789 words.</p><p>• Specialists - Context specific phrases sug­gested by AAC specialists<footnote anchor="3"/>. This set is used throughout the paper. 952 sentences, 3842 words.</p><p>• TurkDev - Communications from 10% of the workers in our crowdsourced collection (disjoint from TurkTrain and TurkTest). This set will be used for initial evaluations and also to tune our models. 551 sentences, 2916 words.</p><p>• TurkTest - Communications from 10% of the workers in our crowdsourced collection (disjoint from TurkTrain and TurkDev). This set is used only in the final evaluation section. 563 sen­tences, 2721 words.</p><p>Test set</p><p>Sentence</p><footnote label="3">http://aac.unl.edu/vocabulary.html</footnote><p>C om m I love your new haircut.</p><p>C om m How many children do you have?</p><p>Specialists Are you sure you don't mind?</p><p>Specialists I'll keep an eye on that for you</p><p>SwitchTest yeah he's a good actor though</p><p>SwitchTest what did she have like</p><table caption="Table 3: Examples from three of our test sets."></table><p>• SwitchTest - Transcripts of three Switchboard conversations (disjoint from the Switchboard training set). This is the same set used in Trnka et al. (2009). We dropped one sentence containing a dash. This set is only used in the final evaluation section. 59 sentences, 508 words.</p><p>TurkDev and TurkTest contain text similar to table 1 and 2. Table 3 shows some examples from the other three test sets. Sentences in Comm tended to be richer in vocabulary and subject matter than those in Specialists. The Specialists sentences tended to be general phrases that avoided mention­ing specific situations, proper names, etc. Sentences in SwitchTest exhibited phenomena typical of human-to-human voice conversations (filler words, backchannels, interruptions, etc).</p><subsection number="3.2" title="Language Model Training"><p>All language models were trained using the SRILM toolkit (Stolcke, 2002). All models used interpo­lated modified Kneser-Ney smoothing (Kneser and Ney, 1995; Chen and Goodman, 1998). In this sec­tion, we trained 3-gram language models with no count-cutoffs. All text was converted to lowercase and we removed punctuation except for apostrophes. We believe punctuation would likely slow down a user's conversation for only a small potential advan­tage (e.g. improving text-to-speech prosody).</p><p>All models used a vocabulary of 63K words in­cluding an unknown word. We obtained our vocab­ulary by taking all words occurring in TurkTrain and all words occurring four or more times in the Twitter training set. We restricted our vocabu­lary to words from our large list of 330K words. This restriction prevented the inclusion of com­mon misspellings prevalent in many of our train­ing sets. Our 63K vocabulary resulted in low OOV<page local="5" global="704"/></p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">X</doubt><doubt alpha="25.0" length="4" tooSmall="False" monospace="0.0">q_ ;</doubt><p>q.</p><doubt alpha="25.0" length="4" tooSmall="False" monospace="0.0">0 ö)</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">I</doubt><doubt alpha="0.0" length="51" tooSmall="False" monospace="0.0">4     6    8    10   12   14   16   18   20   22 24</doubt><p>Words of training data (K)</p><p>(a) TurkDev test set</p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">x</doubt><doubt alpha="25.0" length="4" tooSmall="False" monospace="0.0">a. ;</doubt><p>0 q.</p><doubt alpha="50.0" length="2" tooSmall="False" monospace="0.0">ö)</doubt><p>(b) Comm test set (c) Specialists test set</p><p>Figure 3: Perplexity of language models trained on the same amount of data from different sources. The perplexity is the average of 20 models trained on random subsets of the training data (one standard deviation error bars).</p><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">o o</doubt><doubt alpha="66.7" length="3" tooSmall="True" monospace="0.0">IT)</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">o</doubt><p>rates for all test sets: Comm 0%, Specialists 0.05%, TurkDev 0.1%, TurkTest 0.07%, and SwitchTest 0.8%.</p></subsection><subsection number="3.3" title="Small Training Size Experiment"><p>We trained language models on each dataset, vary­ing the number of training words from 4K to 24K (the limit of the TurkTrain set). For each dataset and training amount, we built 20 different models by choosing sentences from the full training set at ran­dom. We computed the mean and standard deviation of the per-word perplexity of the set of 20 models.</p><p>As shown in figure 3, word-for-word the Turk-Train data was superior for our three most AAC-like test sets. Thus it appears our crowdsourcing pro­cedure was successful at generating AAC-like data. Twitter was consistently the second best. Blog, Usenet and Switchboard also performed well.</p></subsection><subsection number="3.4" title="Large Training Size Experiment"><p>The previous experiment used a small amount of training data. We selected the best three datasets having tens of millions of words of training data: Usenet, Blog, and Twitter. As in the previ­ous experiment, we computed the mean and stan­dard deviation of the per-word perplexity of a set of 20 models. Increasing the amount of training data substantially reduced perplexity compared to our small TurkTrain collection (figure 4). Tweets were clearly well suited for modeling AAC-like text as 3M words of Twitter data was better than 40M words of Blog data.</p></subsection><subsection number="3.5" title="Comparison with Real AAC Data"><p>Beukelman et al. (1984) analyzed the communica­tions made by five nonspeaking adults over 14 days. All users were experienced using a tape-typewriter AAC device. Beukelman gives a ranked list of the top 500 words, the frequency of the top 20 words, and statistics calculated on the communications.</p><p>For the top 10 words in Beukelman's AAC user data, we computed the probability of each word in our various datasets (figure 5). As shown, some words such as "to" and "a" occur with similar fre­quency across all datasets. Some words such as "the" are overrepresented in data such as news text. Other words such as "I" and "you" are much more variable. Our Turk data has the closest matching frequency for the most popular word "I". Interest­ingly, our Turk data shows a much higher probabil­ity for "you" than the AAC data. We believe this re­sulted from the situation we asked workers to imag­ine (i.e. communicating via a letter-at-a-time scan­ning interface). Workers presumed in such a situa­tion they would need to ask others to do many tasks. We observed many requests in the data such as "Can (a) TurkDev test set (b) Comm test set (c) Specialists test set<page local="6" global="705"/></p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Wikipedia</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Usenet</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>Switchboard</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>Blog E Twitter</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>TurkTrain</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>\</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>V</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>^ K</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">o o</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">o</doubt><doubt alpha="40.0" length="10" tooSmall="False" monospace="0.0">^HHHl"-—•-</doubt><doubt alpha="0.0" length="8" tooSmall="False" monospace="0.0">10 20 30</doubt><p>Words of training data (M)</p><p>Figure 4: Perplexity of language models trained on increasing amounts of data from three different training sources. Results on the TurkDev, Comm and Specialists test sets.</p><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">£0</doubt><doubt alpha="80.0" length="5" tooSmall="False" monospace="0.0">IE re</doubt><doubt alpha="33.3" length="6" tooSmall="False" monospace="0.0">-a 0 o</doubt><p>Q.</p><doubt alpha="50.0" length="2" tooSmall="False" monospace="0.0">I0</doubt><doubt alpha="16.7" length="6" tooSmall="False" monospace="0.0">.5? 'E</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">3 0</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">you</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">the</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">my</doubt><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">and</doubt><figure caption="Figure 5: The unigram probabilities of the top 10 words reported by Beukelman et al. (1984)."></figure><p>you change my sheets?" and "Can you walk the dog for me?"</p><p>Beukelman reports 33% of all communications could be made using only the top 500 words. The same 500 words allowed writing of 34% of our Turk communications. Other datasets exhibited much lower percentages. Note that this is at least partially due to the longer sentences present in some datasets. Unfortunately, Beukelman does not report the aver­age communication length. Our Turk communica­tions were 5.0 words on average. The next shortest dataset was Twitter with 7.5 words per communi­cation. Despite their short average length, only 10% of Tweets could be written using the top 500 words.</p><p>Beukelman reports that 80% of words in the AAC users' communications were in the top 500 words. 81% of the words in our crowdsourced data were in this word list. For comparison, only 65% of words in our Twitter data were in the 500 word vocabu­lary. While our TurkTrain set contains only 2141 unique words, this may in fact be good since it has been argued that rare words have received too much attention in AAC (Baker et al., 2000).</p></subsection></section><section number="4" title="Using Large Datasets Effectively"><p>In the previous section, we found our crowdsourced data was good at predicting AAC-like test sets. However, in order to build a good long-span lan­guage model, we would require millions of such communications. Crowdsourcing such a large col­lection would be prohibitively expensive. There­fore, we instead investigated how to use our crowd-sourced data to intelligently select AAC-like data from other large datasets. For large datasets, we used Twitter, Blog and Usenet as they were both large and well-matched to AAC data.</p><subsection number="4.1" title="Selecting AAC-like Data"><p>For each training sentence, we calculated three val­ues:</p><p>• WER - The minimum word error rate between the training sentence and one of the crowdsourced</p><doubt alpha="40.0" length="5" tooSmall="True" monospace="0.0">Q. Q.</doubt><doubt alpha="75.0" length="4" tooSmall="False" monospace="0.0">g ar</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">cd</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">lo</doubt><doubt alpha="0.0" length="1" tooSmall="True" monospace="0.0">0</doubt><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">40</doubt><doubt alpha="0.0" length="4" tooSmall="True" monospace="0.0">0.07</doubt><doubt alpha="0.0" length="4" tooSmall="True" monospace="0.0">0.06</doubt><doubt alpha="0.0" length="4" tooSmall="True" monospace="0.0">0.01</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">it</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">in</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">is</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">to</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">a</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Usenet</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>p.</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Blog</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>§</b></p></td><td class="cell"><p>E Twittter</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Usenet</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>1</p></td><td class="cell"><p><b>p.</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>L</i></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Ht</p></td><td class="cell"><p>B Twitter</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Usenet</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>p.</b></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Diog</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>B Twitter</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>L</p></td><td class="cell"><p>■ AAC users, Beukelman</p><p>■ Turk workers</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>J i</p></td><td class="cell"><p>■ Switchboard</p><p>■ Twitter</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>L J</p></td><td class="cell"><p>■ Blog</p><p>■ Usenet</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>k..................J .....</b>i.....</p></td><td class="cell"><p>■ Wikipedia News</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="7" global="706"/><doubt alpha="60.0" length="5" tooSmall="True" monospace="0.0">Q. CL</doubt><doubt alpha="93.3" length="15" tooSmall="True" monospace="0.0">Entropy pruning</doubt><p>Count cutoff pruning a Cross-entropy selection</p><doubt alpha="92.3" length="13" tooSmall="True" monospace="0.0">WER selection</doubt><doubt alpha="91.7" length="24" tooSmall="True" monospace="0.0">Cross-entropy difference</doubt><doubt alpha="40.0" length="5" tooSmall="True" monospace="0.0">Q. Q_</doubt><doubt alpha="0.0" length="15" tooSmall="False" monospace="0.0">0 5 10 15 20 25</doubt><p>Language model parameters (M)</p><p>(a) Twitter</p><footnote label="7">Count cutoff pruning 9 Cross-entropy selection</footnote><p>ko o h° °f o oooooooooc c c o a</p><doubt alpha="71.4" length="7" tooSmall="False" monospace="0.0">(b)Blog</doubt><doubt alpha="83.3" length="48" tooSmall="True" monospace="0.0">v Count cutoff pruning ■ Cross-entropy selection</doubt><p>(c) Usenet communications. This is the minimum number of words that must be inserted, substituted or deleted to transform the training sentence into a Turk-Train sentence divided by the number of words in the TurkTrain sentence. For example, the training sentence "I didn't sleep well Monday night either" was given a WER of 0.33 because two word-changes transformed it into a message written by a worker: "I didn't sleep well last night".</p><figure caption="Figure 6: Perplexity onTurkDevusing different data selection and pruning techniques."></figure><doubt alpha="100.0" length="2" tooSmall="True" monospace="0.0">CO</doubt><doubt alpha="100.0" length="2" tooSmall="True" monospace="0.0">CD</doubt><doubt alpha="50.0" length="6" tooSmall="False" monospace="0.0">O   Oo</doubt><doubt alpha="71.4" length="7" tooSmall="False" monospace="0.0">ojo.o o</doubt><footnote label="0">o</footnote><doubt alpha="100.0" length="2" tooSmall="True" monospace="0.0">TT</doubt><p>• Cross-entropy, in-domain - The average per-word cross-entropy of the training sentence under a 3-gram model trained on TurkTrain.</p><p>• Cross-entropy, background - The average per­word cross-entropy of the training sentence un­der a 3-gram model trained on a random portion of the training set. The random portion was the same size as TurkTrain.</p><p>We used these values to limit training to only AAC-like sentences. We tried three different selec­tion methods. In <i>WER selection, </i>only sentences be­low a threshold on the word error rate were kept in the training data. This tends to find variants of exist­ing communications in our Turk collection.</p><p>In <i>cross-entropy selection, </i>we used only sen­tences below a threshold on the per-word cross-entropy with respect to a TurkTrain language model. This is equivalent to placing a threshold on the perplexity. Previously this technique has been used to improve language models based on web data (Bulyko et al., 2007; Gao et al., 2002) and to con­struct domain-specific models (Lin et al., 1997).</p><p>In <i>cross-entropy difference selection, </i>a sentence's score is the in-domain cross-entropy minus the back­ground cross-entropy (Moore and Lewis, 2010). This technique has been used to supplement Euro­pean parliamentary text (48M words) with newswire data (3.4B words) (Moore and Lewis, 2010). We were curious how this technique would work given our much smaller in-domain set of 24K words.</p></subsection><subsection number="4.2" title="Data Selection and Pruning"><p>We built models selecting sentences below different thresholds on the WER, in-domain cross-entropy, or cross-entropy difference. For comparison, we also pruned our models using conventional count-cutoff and entropy pruning (Stolcke, 1998). During en­tropy pruning, we used a Good-Turing estimated model for computing the history marginals as the lower-order Kneser-Ney distributions are unsuitable for this purpose (Chelba et al., 2010).</p><p>We calculated the perplexity of each model on three test sets. We also tallied the number of model parameters (all n-gram probabilities plus all backoff weights). On TurkDev, cross-entropy difference selection performed the best for all models sizes and for all training sets (figure 6). We also found cross(a) Twitter (c) Usenet<page local="8" global="707"/></p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">o</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">1</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">0</doubt><doubt alpha="71.4" length="7" tooSmall="False" monospace="0.0">(b)Blog</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">CL</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">ff</doubt><doubt alpha="66.7" length="6" tooSmall="False" monospace="0.0">2-gram</doubt><doubt alpha="66.7" length="6" tooSmall="False" monospace="0.0">3-gram</doubt><doubt alpha="66.7" length="6" tooSmall="False" monospace="0.0">4-gram</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">ml</doubt><doubt alpha="0.0" length="2" tooSmall="False" monospace="0.0">-1</doubt><p>Threshold</p><doubt alpha="0.0" length="15" tooSmall="False" monospace="0.0">1 -1.4 -0.4 0.4</doubt><figure caption="Figure 7: Perplexity onTurkDevvarying the cross-entropy difference threshold."></figure><p>entropy difference was the best on CoMM, reducing perplexity by 10-20% relative compared to cross-entropy selection. Results on Specialists showed that WER and both forms of cross-entropy selection performed similarly. All three data selection meth­ods were superior to count-cutoff or entropy prun­ing. We use cross-entropy difference selection for the remainder of this paper.</p></subsection><subsection number="4.3" title="Model Order and Optimal Thresholds"><p>We created 2-gram, 3-gram and 4-gram models on Twitter, Blog, and Usenet using a range of cross-entropy difference thresholds. 4-gram models slightly outperformed 3-gram models (figure 7). The optimal threshold for 4-gram models were as fol­lows: Twitter 0.0, Blog -0.4, and Usenet -0.7. These thresholds resulted in using 20% of TWIT­TER, 5% of Blog, and 1% of Usenet.</p></subsection><subsection number="4.4" title="Mixture Model"><p>We created a mixture model using linear interpo­lation from the TWITTER, USENET and BLOG 4-gram models created with each set's optimal thresh­old. The mixture weights were optimized with re­spect to TurkDev using SRILM. The final mix­ture weights were: Twitter 0.42, Blog 0.29, and USENET 0.29. Our final 4-gram mixture model had 43M total parameters and a compressed disk size of 316MB.</p></subsection></section><section number="5" title="Evaluation"><p>in this section, we compare our mixture model against baseline models. We show performance with respect to usage in a typical AAC text entry interface based on word prediction.</p><subsection number="5.1" title="Predictive Text Entry"><p>Many AAC communication devices use word pre­dictions. in a word prediction interface users type letters and the interface offers word completions based on the prefix of the current word and often the prior text. By selecting one of the predictions, the user can potentially save keystrokes as compared to typing out every letter of each word.</p><p>We assume a hypothetical predictive keyboard in­terface that displays five word predictions. Our key­board makes predictions based on up to three words of prior context. our keyboard predicts words even before the first letter of a new word is typed. As a user types letters, predictions are limited to words consistent with the typed letters. If the system makes a correct prediction, we assume it takes only one keystroke to enter the word and any following space.</p><p>We only predict words in our 63K word vocab­ulary (empty prediction slots are possible). We dis­play a word even if it was already a proposed predic­tion for a shorter prefix of the current word. The first word in a sentence is conditioned on the sentence-start pseudo-word. if an out-of-vocabulary word is typed, the word is replaced in the language model's context with the unknown pseudo-word.</p><p>We evaluate our predictive keyboard using the common metric of keystroke savings (Ks):</p><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">KS</doubt><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">0-(£))</doubt><doubt alpha="16.7" length="6" tooSmall="False" monospace="0.0">x100%,</doubt><p>where <i>kp</i><i> </i>is the number of keystrokes required with word predictions and <i>ka </i>is the number of keystrokes required without word prediction.</p></subsection><subsection number="5.2" title="Predictive Performance Experiment"><p>We compared our mixture model using cross-entropy difference selection with three baseline models trained on all of Twitter, Switchboard and TurkTrain. The baseline models were un-pruned 4-gram models trained using interpolated modified Kneser-Ney smoothing. They had 72M, 5M, and 129K parameters respectively.</p><p>As shown in table 4, our mixture model per­formed the best on the three most AAC-like test sets (Comm, Specialists, and TurkTest). The<page local="9" global="708"/></p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">1 -1</doubt><p>LM Test set PPL KS</p><p>Table 4: Perplexity (PPL) and keystroke savings (KS) of different language models on four test sets. The bold line shows the best performing language model on each test set.</p><p>mixture model provided substantial increases in keystroke savings compared to a model trained solely on Switchboard. The mixture model also per­formed better than simply training a model on a large amount of Twitter data. The model trained on only 24K words of Turk data did surprisingly well given its extremely limited training data.</p><p>our Switchboard model performed the best on SwitchTest with a keystroke savings of 58.8%. For comparison, past work reported a keystroke sav­ings of 55.7% on SwitchTest using a 3-gram model trained on Switchboard (Trnka et al., 2009). While our mixture model performed less well on SwitchTest (52.8%), it is likely the other three test sets better represent AAC communications.</p></subsection><subsection number="5.3" title="Larger Mixture Model Experiment"><p>our mixture language model used the best thresh­olds with respect to TurkDev. This resulted in throwing away most of the training data. This might be suboptimal in practice if an AAC user's com­munications are somewhat different or more diverse than the language generated by the Turk workers.</p><p>We trained a series of mixture models in which we varied the cross-entropy difference thresholds</p><doubt alpha="0.0" length="16" tooSmall="False" monospace="0.0">-0.5 0.0 0.5 1.0</doubt><p>Change from optimal thresholds</p><p>Figure 8: Keystroke savings on mixture models varying a constant added to the optimal thresholds with respect to TurkDev.</p><p>by adding a constant to all three thresholds. The mixture weights for each new model were opti­mized with respect to TurkDev. Using somewhat larger models did improve keystroke savings for all test sets except for TurkTest (figure 8). How­ever, using too large thresholds eventually hurt per­formance except on SwitchTest. Performance on SwitchTest steadily increased from 52.8% to 56.6%. These gains however came at the cost of big­ger models. The model using +1.0 of the optimal thresholds had 384M parameters and a compressed size of 3.0 GB.</p></subsection></section><section number="6" title="Discussion"><p>Given the ethical implications of collecting mes­sages from actual AAC users, it is unlikely that a large corpus of genuine AAC messages will ever be available to researchers. An important finding in this paper is that crowdsourcing can be an effective way to obtain surrogate data for improving AAC lan­guage models. Another finding is that Twitter pro­vides a continuous stream of large amounts of very AAC-like data. Twitter also has the advantage of al­lowing models to be continually updated to reflect current events, new vocabulary, etc.</p><subsection number="6.1" title="Limitations and Implications"><p>We collected data from a large number of workers, some of whom may have written only a single communication.<page local="10" global="709"/> This may have resulted in more mes­sages about simple situations and perceived needs which could differ from true AAC usage.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>__^-x-*-*-*-*-*-*-»-«</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>0      —A—</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>Specialist</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>Comm</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>B TurkTest</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>SwitchTest</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Mixture</b></p></td><td class="cell"><p><b>Comm</b></p></td><td class="cell"><p><b>47.9</b></p></td><td class="cell"><p><b>62.5%</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Twitter</p></td><td class="cell"><p>Comm</p></td><td class="cell"><p>55.9</p></td><td class="cell"><p>60.9%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Switchboard</p></td><td class="cell"><p>Comm</p></td><td class="cell"><p>151.1</p></td><td class="cell"><p>54.4%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Turk</p></td><td class="cell"><p>Comm</p></td><td class="cell"><p>165.9</p></td><td class="cell"><p>52.7%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Mixture</b></p></td><td class="cell"><p><b>Specialists</b></p></td><td class="cell"><p><b>25.7</b></p></td><td class="cell"><p><b>63.1%</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Twitter</p></td><td class="cell"><p>Specialists</p></td><td class="cell"><p>27.3</p></td><td class="cell"><p>61.9%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Switchboard</p></td><td class="cell"><p>Specialists</p></td><td class="cell"><p>64.5</p></td><td class="cell"><p>57.7%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Turk</p></td><td class="cell"><p>Specialists</p></td><td class="cell"><p>85.9</p></td><td class="cell"><p>52.8%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Mixture</b></p></td><td class="cell"><p><b>TurkTest</b></p></td><td class="cell"><p><b>31.2</b></p></td><td class="cell"><p><b>62.0%</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Twitter</p></td><td class="cell"><p>TurkTest</p></td><td class="cell"><p>42.3</p></td><td class="cell"><p>59.3%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Switchboard</p></td><td class="cell"><p>TurkTest</p></td><td class="cell"><p>172.5</p></td><td class="cell"><p>50.6%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Turk</p></td><td class="cell"><p>TurkTest</p></td><td class="cell"><p>51.0</p></td><td class="cell"><p>57.6%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Mixture</p></td><td class="cell"><p>SwitchTest</p></td><td class="cell"><p>174.3</p></td><td class="cell"><p>52.8%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Twitter</p></td><td class="cell"><p>SwitchTest</p></td><td class="cell"><p>142.6</p></td><td class="cell"><p>54.9%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Switchboard</b></p></td><td class="cell"><p><b>SwitchTest</b></p></td><td class="cell"><p><b>79.2</b></p></td><td class="cell"><p><b>58.8%</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Turk</p></td><td class="cell"><p>SwitchTest</p></td><td class="cell"><p>642.5</p></td><td class="cell"><p>42.9%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>our data does not contain long-term two-sided conversations. Thus it may not be as useful for eval­uating techniques that adapt to past messages or that use the conversation partner's communications.</p><p>We asked workers to imagine they were using a scanning-style AAC device. We believe this led workers to presume they would require assistance in many routine physical tasks. our workers were (presumably) without cognitive or language impair­ments. Thus our collection is more representative of one subgroup of AAC communicators (scanning users with normal cognitive function and language skills). By modifying the situation given to workers, it is likely we can expand our collection to better rep­resent other groups of AAC users, such as those us­ing predictive keyboards or eye-trackers. However, obtaining data representative of users with cognitive or language impairments via crowdsourcing would probably be difficult.</p><p>While we were unable to obtain real AAC mes­sages for testing, we believe the Comm and Spe­CiALiST test sets provide a good indication of the real-world potential for our methods. our collected Turk data was compared with reported data from ac­tual AAC users (though this comparison was neces­sarily coarse-grained). We hope that by releasing our data and models it may be possible for those privy to real AAC communications to validate and report about the techniques described in this paper.</p><p>We evaluated our models in terms of perplexity and keystrokes savings within the auspices of a pre­dictive keyboard. Further work is needed to inves­tigate how our numeric gains translate to real-world benefits to users. However, past work indicates more accurate predictions do in fact yield improvements in human performance (Trnka et al., 2009).</p><p>Finally, while the predictive keyboard is a com­monly studied interface, it is not appropriate for all AAC users. Eye-tracker users may prefer an in­terface such as Dasher (Ward and MacKay, 2002). Single-switch users may prefer an interface such as Nomon (Broderick and MacKay, 2009). Any AAC interface based on word- or letter-based predictions stands to benefit from the methods described in this paper.</p></subsection></section><section number="7" title="Conclusions"><p>in this paper we have shown how workers' creativity on a microtask crowdsourcing market can be used to create fictional but plausible AAC communica­tions. We have demonstrated that these messages model conversational AAC better than the currently used datasets based on telephone conversations or newswire text. We used our new crowdsourced dataset to intelligently select sentences from Twit­ter, blog and Usenet data.</p><p>We compared a variety of different techniques for intelligent training data selection. We found that even for our small amount of in-domain data, the recently proposed cross-entropy difference method was consistently the best (Moore and Lewis, 2010). Finally, compared to a model trained only on Switchboard, our best performing model reduced perplexity by 60-82% relative on three AAC-like test sets. This translated to a potential keystroke savings in a predictive keyboard interface of 5-11%.</p><p>in conclusion, we have shown how to create long­span AAC language models using openly avail­able resources. Our models significantly outperform models trained on the commonly used data sources of telephone transcripts and newswire text. To aid other researchers, we have publicly released our crowdsourced AAC collection, word lists and best-performing models. We hope complementary tech­niques such as topic modeling and language model adaptation will provide additive gains to those ob­tained by training models on large amounts of AAC-like data. We plan to use our models to design and test new interfaces that enable faster communication for AAC users.</p></section><section title="Acknowledgments"><p>We thank Keith Trnka and Horabail Venkatagiri for their assistance. This work was supported by the En­gineering and Physical Sciences Research Council (grant number EP/H027408/1).</p></section><references><p>Bruce Baker, Katya Hill, and Richard Devylder. 2000. Core vocabulary is the same across environments. In <i>California State University at Northridge Conference.</i></p><p>David R. Beukelman, Kathryn M. Yorkston, Miguel Poblete, and Carlos Naranjo.   1984.  Frequency of word occurrence in communication samples produced by adult communication aid users.<page local="11" global="710"/> <i>Journal of Speech and Hearing Disorders, </i>49:360-367. Tamara Broderick and David J. C. MacKay. 2009. Fast and flexible selection with a single switch. <i>PLoS ONE, </i>4(10):e7481.</p><p>Ivan Bulyko, Mari Ostendorf, Manhung Siu, Tim Ng, Andreas Stolcke, and Ozgur CCetin. 2007. Web resources for language modeling in conversational speech recognition. <i>ACM Transactions on Speech and Language Processing, </i>5(1):1-25.</p><p>Kevin Burton, Akshay Java, and Ian Soboroff. 2009. The iCWSM 2009 Spinn3r dataset. in <i>Proceedings ofthe 3rd Annual Conference on Weblogs and Social Media.</i></p><p>Ciprian Chelba, Thorsten Brants, Will Neveitt, and Peng Xu. 2010. Study on interaction between entropy prun­ing and Kneser-Ney smoothing. In <i>Proceedings of the International Conference on Spoken Language Pro­cessing, </i>pages 2422-2425.</p><p>Stanley F. Chen and Joshua T. Goodman. 1998. An empirical study of smoothing techniques for language modeling. Technical report, Computer Science Group, Harvard University.</p><p>Jianfeng Gao, Joshua Goodman, Mingjing Li, and Kai-Fu Lee. 2002. Toward a unified approach to statistical language modeling for chinese. <i>ACM Transactions on Asian Language Information Processing, </i>1:3-33.</p><p>Nestor Garay-Vitoria and Julio Abascal. 2006. Text pre­diction systems: A survey. <i>Universal Access in the Information Society, </i>4:188-203.</p><p>Nestor Garay-Vitoria and Julio Gonzalez-Abascal. 1997. Intelligent word-prediction to enhance text input rate. In <i>Proceedings of the 2nd ACM International Confer­ence on Intelligent User Interfaces, </i>pages 241-244.</p><p>J.J. Godfrey, E.C. Holliman, and J. McDaniel. 1992. SWITCHBOARD: Telephone speech corpus for re­search and development. <i>Proceedings ofthe IEEE Conference on Acoustics, Speech, and Signal Process­ing, </i>pages 517-520.</p><p>David Graff, Roni Rosenfeld, and Doug Pau. 1995. CSR-III text. Linguistic Data Consortium, Philadel­phia, PA, USA.</p><p>David Graff. 2003. English gigaword corpus. Linguistic Data Consortium, Philadelphia, PA, USA.</p><p>Sheri Hunnicutt. 1989. Using syntactic and semantic in­formation in a word prediction aid. In <i>Proceedings of the 1st European Conference on Speech Communica­tion and Technology, </i>pages 1191-1193.</p><p>Reinhard Kneser and Hermann Ney. 1995. Im­proved backing-off for m-gram language modeling.</p><p>In <i>Proceedings ofthe IEEE Conference on Acoustics, Speech, and Signal Processing, </i>pages 181-184.</p><p>Gregory W. Lesher and Gerard J. Rinkus. 2002. Domain-specific word prediction for augmentative communication. In <i>Proceedings of the RESNA 2002 Annual Conference.</i></p><p>Jianhua Li and Graeme Hirst. 2005. Semantic knowl­edge in word completion. In <i>Proceedings of the 7th International ACM SIGACCESS Conference on Com­puters and Accessibility, </i>pages 121-128.</p><p>Sung-Chien Lin, Chi-Lung Tsai, Lee-Feng Chien, Ker-Jiann Chen, and Lin-Shan Lee. 1997. Chinese lan­guage model adaptation based on document classifi­cation and multiple domain-specific language models. In <i>Proceedings of the 5th European Conference on Speech Communication and Technology, </i>pages 1463­1466.</p><p>Robert C. Moore and William Lewis. 2010. Intelligent selection of language model training data. In <i>Proceed­ings of the 48th Annual Meeting of the Association of Computational Linguistics, </i>pages 220-224.</p><p>Alan Ritter, Colin Cherry, and Bill Dolan. 2010. Un-supervised modeling of twitter conversations. In <i>Pro­ceedings of HLT-NAACL 2010, </i>pages 172-180.</p><p>Cyrus Shaoul and Chris Westbury. 2009. A USENET corpus (2005-2009). University of Alberta, Canada.</p><p>Andreas Stolcke. 1998. Entropy-based pruning of backoff language models. In <i>Proceedings ofDARPA Broadcast News Transcription and Understanding Workshop, </i>pages 270-274.</p><p>Andreas Stolcke. 2002. SRILM - an extensible language modeling toolkit. In <i>Proceedings ofthe 7th Annual In­ternational Conference on Spoken Language Process­</i>ing, pages 901-904.</p><p>Keith Trnka, Debra Yarrington, and Christopher Penning­ton. 2006. Topic modeling in fringe word prediction for AAC. In <i>Proceedings of the 11th ACM Interna­tional Conference on Intelligent User Interfaces, </i>pages 276-278.</p><p>Keith Trnka, John McCaw, Debra Yarrington, Kathleen F. McCoy, and Christopher Pennington. 2009. User in­teraction with word prediction: The effects of predic­tion quality. <i>ACM Transactions on Accessible Com­puting, </i>1:17:1-17:34.</p><p>Keith Trnka. 2008. Adaptive language modeling for word prediction. In <i>Proceedings of the 46th Annual Meeting ofthe Association for Computational Linguis­tics on Human Language Technologies: Student Re­search Workshop, </i>pages 61-66.</p><p>Horabail Venkatagiri. 1999. Efficient keyboard layouts for sequential access in augmentative and alternative communication. <i>Augmentative and Alternative Com­munication, </i>15(2):126-134.</p><p>Tonio Wandmacher and Jean-Yves Antoine. 2007. Methods to integrate a language model with semantic information for a word prediction component.<page local="12" global="711"/> <i>Pro­ceedings ofthe Joint Conference on Empirical Meth­ods in Natural Language Processing and Computa­tional Natural Language Learning, </i>pages 506-513. Tonio Wandmacher, Jean-Yves Antoine, Franck Poirier, and Jean-Paul Departe. 2008. SIBYLLE, an assis-tive communication system adapting to the context and its user. <i>ACM Transactions on Accessible Computing, </i>1:6:1-6:30.</p><doubt alpha="58.5" length="53" tooSmall="False" monospace="0.0">D. J. Ward and D. J. C. MacKay. 2002. Fast hands-free</doubt><p>writing by gaze direction. <i>Nature, </i>418(6900):838.</p></references></body></article>