A major (words) task in natural language generation (NLG) is information lexical choice, that is, choosing lexemes to communicate module. From to the reader the selected by the system’s content determination on models of word meanings, a semantic which perspective when lexical word choice algorithms are based of may state a can and cannot be used; course, lexical when choice algorithms words. also consider syntactic constraints and pragmatic Such models features choosing user. However, assume that it is possible to specify what a particular word means to a particular out in S T both project the cognitive University science literature of and of recent which experiments carried the UM IME may at the Aberdeen, suggest this be difficult to do because of the current authors are a More is, part, that precisely, because the same may word may mean variations among people, that rough different things what to different agree level about a word people. may although people word about its at a precise This means definition, it and may in particular, impossible to what in objects means, they disagree or events a can word be applied. meanings that with multiple be even with for texts readers, and indeed principle to specify precise reader, for a our the system has access to an were extremely detailed in which user model. texts single A individuals corpus in unless study (in project also showed words that there were differences used in sense words that were some used only words the orthographically (spelled). by a subset of the authors) This and also how suggests that it may be for NLG systems (and realized risky on interpreting indeed human words authors) to depend for communicative as in intends. This success in the human reader’s turn models that perhaps NLG exactly the system suggests it may systems should be cautious using very detailed in lexical and not also interpret that is in case the in reader which does a word be useful to add some This redundancy to texts only one as expected. if especially users many true applications each NLG texts, then Human over user reads read perhaps time they will generated text; generated the lexical usage. variability also needs learn about and adapt into to natural system’s processing (NLP) to be taken account by language not researchers identical performing corpora analyses; making such analyses should assume that everyone uses rules when linguistic decisions. ∗ Department of Computing Science, University of Aberdeen, Aberdeen AB24 3UE, UK. E-mail: ereiter@csd.abdn.ac.uk † Department of Computing Science, University of Aberdeen, Aberdeen AB24 3UE, UK. E-mail: ssripada@csd.abdn.ac.uk © c 2002 Association for Computational Linguistics Computational Linguistics Volume 28, Number 4

2.1 Previous Linguists Research have word. acknowledged Nunberg (1978, that people 81), may associate different writes: meanings with the same page for example, There is considerable not variation among speakers member of in beliefs category. about what does may and does constitute a includes the not ... Take jazz. may I believe that the opposite. category we ragtime, will but blues; you believe the exact of After all, have been exposed to a very thority, different we must set exemplars. our own And absent a of commonly accepted most prob- au- in of construct theories of musical categories, ably the light varying degrees sophistication. Many modern mental theories of mental categorization (Rosch 1978; prototypes Smith assume categories represented by or and Medin 1981) if that are prototypes exemplars. Therefore, different people are exposed to different category membership. and exemplars, they are likely Parikh to (1994) have made different rules for evaluating category For a similar point and from backed it Munsell up with some simple experimen- tation. example, he showed squares the or blue; chart individuals to participants character- and asked ized them to characterize in the ways. squares squares if In as red different the different were another experiment he showed that differences remained with even participants allowed to associate fuzzy-logic-type truth values In statements. the psychological community, objects. For Malt et al. (1999) investigated what wished names par- ticipants whether gave to real-world would example, these researchers to know container. participants They were describe interested a pump-top in hand lotion across dispenser as a bottle or a primarily in named even objects. within variations linguistic communities, but they also discovered that They a linguistic (page community there only were ences objects how participants state 242) that 2 of differ- in were name of 76 native-English- the 60 their study given the same by all their The 2.2 S S UM T T IME Project UM IME project of at the University of Much Aberdeen of is project researching techniques on for deter- gen- erating mination summaries time-series Sripada data. 1 (see, [2001]), the it is focuses content for example, et al. which is where but also originated. time-series To S summaries, the work examining lexical described in choice algorithms for T on this ticle has primarily focused domains, weather ar- date, of UM IME summaries sensors, although we two forecasts and gas turbine have recently started Reiter and Sripada Human Variation and Lexical Choice work in a third domain as well, summaries of sensor readings in neonatal intensive care units. 2.2.1 Gas Turbine Domain. we In order to develop write a lexicon for describing sensor two experts to short descriptions of patterns gas data, asked 38 in turbine (A produce were with is an interval of signal fragments. signal fragment an average size of a single 8.3 time-series words. In no data channel.) The descriptors small, Many of case did the experts exactly the of same descriptor words for a fragment. the differences simply reflected one usage different to express what the same other underlying concept; expert to describe the called increase. for example, typically used rise cases the reflected different levels of detail. For expert example, one In other differences fragment was described by expert , whereas A as Generally B steady with a slow rising trend. Lots of noise and a steps the shorter phrase Rising trend, with noise . few small noise expert used Both experts only also had personal whereas vocabulary; for example, the terms bathtub B. expert A, the terms square wave and transient were and dome were used by used only by expert Most importantly from the perspective of this article, in there were meanings cases in which words. experts reflected a difference the associated with the differences For between the example, were as oscillations, both experts used the word but two signals, including oscillation . Six by experts the one signals were shown in Figure described both as oscillations only by why expert B. We not 1, described do have to support hypotheses about the agreed on enough examples solidly experts on its application others, of the term oscillation to some signals and disagreed seems fit the available data is application to that the experts agreed on but one explanation that to prototype to [Rosch signals 1978] that of were oscillation), very similar to a sine wave (which applying the term presumably were but sometimes disagreed on its is the an to that less similar to wave, such Figure 1. application signals a sine as 2.2.2 Meteorology of 1,099 Domain. In the meteorology weather domain we offshore accumulated a human-written forecasts for oil and ana- lyzed with corpus (produced numerical weather rigs, together the data when files writing by a The were simulation) written that the forecasters examined from the forecasts. forecasts is in Figure by wind a typical forecast shown occur 2; five different fore- casters. A was issued. in short extract this text describes changes speed from and direction predicted to two is days in after An data file shown Table the 1; forecast extract wind the corresponding scribes predicted speed and direction from the numerical weather it de- the intervals. simulation at three-hourly As in the idiosyncratic gas turbine domain, our corpus only analysis showed For that individual had they used. example, one fore- casters vocabulary or the verb initial freshening to indicate that a moderate increase in wind speed from forecaster used moderate no other There were a low value, but forecaster used this verb. also 547 Figure Signal 1 fragment (gas turbine exhaust temperature): Is this an oscillation ? Computational Linguistics Volume 28, Number 4 FORECAST 00-24 WIND(10M): GMT, WSW WEDNESDAY, BACKING 04-Oct 20-24 VEERING SSW 2000 SW BY 10-14 BY EVENING MIDDAY THEN 24-28 Figure Wind (at 2 10m) extract from five-day weather forecast issued on October 2, 2000. differences in orthography. N, E, S, For W, example, whereas others some forecasters lexicalized the four W’LY. and used the lexicalizations N’LY, basic directions as E’LY, S’LY, and We performed a number words; of semantic invariably analyses to determine when different fore- casters For used different we infer these meaning showed of differences between authors. example, attempted to the matched time phrases such wind as descriptor. by evening by searching The for in the Figure first data 2, file record that the wind corresponding forecast for example, says at time by BY MIDDAY . In that the will change to SSW suggested in with a direction SSW the corresponding data shown in 10–14 the of and a speed in the 10–14 range is 1200; Table 1, the first entry example phrase by midday is with the 1200. hence this the time suggests in associated time this example the time phrase by evening is A associated with similar analysis 0000 (on 5-10-00). that the time We repeated this procedure for every individual forecast in our corpus and statistically More an- alyzed the results to determine procedure how forecasters Reiter used Sripada time phrases. details about in the paper, analysis are given by on and meaning (2002). of As re- ported that the forecasters seemed to agree the some They all on forecasters of predominantly other including used by midday evening. to mean time phrases; for example, however, in Table in use particular, whereas terms, by The 1200. disagreed, is the F3 (the of use of by evening text in Figure shown 2) most often 2; often used 1800. this The phrase to mean 0000, forecaster forecasters F1 author and F4 most the of used this phrase to mean 001 differences between forecasters in (which their usage by evening are significant at p < . one-way under both a of chi-squared (which test treats time mean as a categorical variable) and a we analysis variance the forecaster; for this the hour 0 compares time for each test recoded as 24). 2.2.3 in Knows was Java Experiment. with Some colleagues pointed out to us that meteorology whose particular words might a domain an established sublanguage wondered and usage to terms, and what conventions, correspond technical would hapTable Wind (at 1 10m) extract from October 2, 2000, data file (output of numerical weather model). [BAR] Day Hour Wind Direction Wind Speed [BAR] WSW WSW SW SSW SW SSW SSW SSW SW Reiter and Sripada Human Variation and Lexical Choice Table How 2 often by evening was used to refer to each time, for each forecaster (mode in boldface font). [BAR] Hour F1 F2 F3 F4 F5 Total Total 31 14 61 29 14 149 pen in nology. a We domain in which there was no established sublanguage in a small experiment at the University and technical of termi- out which we therefore performed asked which of students and academic staff members Aberdeen 21 postgraduate individuals would to fill a questionnaire : asking the following they regard as knowing Java 2 • • • • • A programming cannot program in Java, but knows that Java is a popular language. B cannot write a Java program Java program from scratch, (such but can make very simple changes to an existing URL). as changing a string constant that specifies a C can use a tool control such as JBuilder to write a very while simple Java program, but cannot use flow constructs such as loops. D can write Java programs only within that one use while loops, arrays, write and program the Java class libraries, of but class; she cannot a that consists several classes. E occasionally can create complex Java programs and classes, of but needs to for details the Java to refer documentation language and class libraries. Respondents E. They could tick Yes, whether Unsure, or B, No. C, All 21 respondents ticked No for A B, and D could be considered to know Java; and Yes for disagreed ticked for 5 about Yes for C, 13 Yes for D. In other words, 3 Yes ticked group, and ticked was over even what among this relatively homogeneous meant in there the phrase of considerable disagreement knows Java terms actual knowledge of the Java programming language. Computational Linguistics Volume 28, Number 4

The meanings previous section with has argued that people in many oscillation, cases do associate words, as by evening, different lexemes and phrases such freshening, only and knows; of and that some such as bathtub What and impact are used on by a subset authors in a particular In domain. applications in which does this have only one lexical choice? users read we generated text, it may be necessary to restrict lexeme was definitions to those that expect in all users to tially advice given domain earlier project share. Indeed, on essen- this to us by a (Reiter, expert Robertson, an Osman generating personalized in which smoking-cessation many letters users over and made texts a of 2000). In cations generated period of time, however, appli- read an argument could be for using a richer vocabulary and set lexeme definitions and expecting of users it may to adapt appropriate to and learn the system’s vocabulary information and usage over course to add “redundant” to texts (sec- the 3.2) while time; be tion the if user is still learning the system’s lexical usage. This if but is can lead to short and less awkward not, texts. Consistency strategy has some risks, successful, is essential this strategy mean 1800 followed; the system should for use to and sometimes by evening to mean example, 0000. sometimes with We by evening not aware in of meanings previous research in use are lexical choice that focuses Perhaps the that different human readers associate with on dealing differences the research is that which investigates tailoring word words. closest strand Robin, expertise 1993). of the For user (Bateman and Coordinated Paris 1989; Reiter choice Multimedia 1991; and phrasing McKeown, according to the Explanation Testbed and (COMET) Tanenblatt (McKeown, Robin, example, Tanenblatt the and 1993) could generate Check the polarity for skilled users and Make sure the General plus on the battery of lines up with the plus on the battery Litman in compartment NLG for unskilled are given by Stede (1995) users. and Wanner reviews (1996); Zukerman previous lexical choice research (2001) in modeling NLP. and In review community, research user word In other words, Parikh (1994) and the linguistic if we has suggested (1) probability that utility of theory interpreted or misinterpreted know the a word’s be applied to choice. being interpretation correctly of misinterpretation, and (2) we the benefit to the user overall of correct of and the cost word. This then interesting can compute an user seems like an theoretical model, utility to the (at using in the on we have looked at), whereas it may but in practice least the applications of interpretation of word, it be is just probably about possible impossible to get data the likelihood models to of misinterpretation, correct calculate cost will because we a the do not have task that how the the generated (and accurate specify exactly it is probably user impossible use texts models). our domain experts have told us that to construct such is, 3.1.1 words Near with Synonyms. meanings: A related problem is choosing between near synonyms, that when similar when changes in wind for example, speed, The or choosing between easing and decreasing describing most saw-tooth in-depth transient and shark-tooth near turbine was signals. examination of transient describing gas Edmonds (1999), who choosing be- tween synonyms using based on undertaken lexicographic work by essentially suggested (Gove rules 1984). such as Webster’s New Dictionary of Synonyms Edmonds was working we and hence Indeed, was on machine translation, not generating texts looking at larger not in differences than the ones from with nonlin- guistic data, which of are concerned. S T dictionaries ; do we general not give definitions of at the level detail required by UM IME for example, are aware any dictionary that Reiter and Sripada Human Variation and Lexical Choice 3.2 Redundancy Another may implication not interpret of human words variation in word it may usage is that if there is NLG a chance that people include information in as expected, it be useful what is for needed an words generates, beyond intended. Indeed, if system to extra interpret the texts users could be word expected interpretations to exactly as the system with noise, perhaps in be considered (text) to be a type of unexpected could “semantic” noise; and intended as meaning. all redundancy the signal can help the reader recover the For example, the identify referring-expression generation on model of Dale and Reiter (1995) selects interpret attributes to referents based the assumption two in the system , expects. Assume, for instance, that the hearer will the attributes as that there are books focus, B 1 and B 2 and Then, the system’s knowledge Dale base Reiter, records that B 1 color according the red book is has red and B 2 has color blue. identifies to however, that uniquely in B 1 . and a distinguishing Parikh, description which has shown are blue; if that this is people can the case, then it is fact disagree about which red and possible that the hearer will objects in eventuality, identify are able B 1 as the referent after hearing the red book . To not fact be to it might information guard against this be useful to add additional attribute for example, if B is 100 about a different to the referring expression; . This 1 is pages and B 2 is 1,000 pages, the system may could generate the thin red book longer protection take longer to utter interpretations. and comprehend, but its than the red redundancy provides book and thus perhaps against unexpected lexical 3.3 Corpus Analysis A in final point is that differences between individuals should perhaps be (Reiter by performing corpus analyses to derive rules for NLG considered general and Sripada people To one randomly chosen example, Hardt and Rambow systems (2001) 2002). on machine suggest a set of take rules for deciding on verb Penn phrase Treebank (VP) ellipsis that These are based 35% learning author, in wonder error techniques applied to the against a They corpus. rate baseline rule. do not achieve a in reduction we if of consider individual considerable amount the remaining error is variation and in a when VP. It would interesting due to variation with deciding to ellide a analysis author specified one a of be to the features given to the machine perform similar whether improved as learning algorithm and see this performance. Computational Linguistics Volume 28, Number 4

To finish NLG on a more positive note, human variation may potentially consistency. In be an opportunity for systems, we because they can guarantee the which means for example, may problems users receive texts written weather-forecasting domain examined, by evening; they have reliably interpreting all five forecasters, an that NLG An in contrast, could be programmed phrases such as consistently. system, NLG programmed always to use this with be to avoid idiosyncratic phrase which system in which users might not could also familiar ( bathtub, for example) and not terms be (e.g., to use cases about their applicability oscillation for Figure terms in Our people disagree corpus and with domain experts suggest that it is not 1). analyses easy writers discussions follow such consistency rules, especially if always for human of to they have limited amounts Psychologists time. on word believe (Garrod that interaction is a 1987). key aspect of Perhaps the process of who and Anderson a small group of humans agreeing usage which with other over a long time period (presumably people constantly communicate each modern under language evolved) will agree on word meanings. But in the circumstances world it never is met or with for whom human writers to write documents for people whom the common they have of natural they interaction have never mechanism otherwise interacted, which reduce effectiveness the for on word may meanings. the agreeing NLG In summary, dealing will with lexical variation among human readers is for undoubtably require of a challenge systems and But if NLG a considerable amount collection. do good job of thought, research, might and producing data many systems can writers, a which would this, they end up of NLG superior technology. texts to human greatly enhance the appeal by Acknowledgments Our thanks to the many whom this work individuals with us (not who have discussed with our including all of Regina agree Barzilay, Copestake, analysis!), Dale, Phil Edmonds, Ann Jim Hunter, Robert Kilgarriff, Owen Rambow, Graeme Adam Rosemary Stevenson, Sandra Williams, Ritchie, Jin Yu. We and are also grateful to the anonymous Special reviewers for their DARE helpful comments. Joan Hall providing thanks to with editor for Last us the not DARE fieldwork work would data. not but certainly least, this without of our have industrial been possible the help collaborators WNI Oceanroutes. at Intelligent This work Applications was and / supported Physical Sciences by the Research UK Engineering Council and (EPSRC), under grant GR / M76681. References Bateman, Phrasing John and in Cecile Paris. 1989. a text In Proceedings terms the of user 11th can understand. International Joint Conference on the Intelligence (IJCAI-89) , Artificial volume 2, Cassidy, pages 1511–1517. 1996. Dictionary Frederick of and Joan Hall, Regional editors. English. Belknap. American Dale, Computational Robert and Ehud interpretations Reiter. 1995. Gricean maxims in of the the Cognitive generation Science of referring 19:233–263. expressions. , Edmonds, Representations Philip. of 1999. Near-Synonyms Semantic Lexical Choice . Ph.D. for Automatic Computer Science Department, thesis, Garrod, University Simon of Toronto, Toronto. 1987. Saying and what Anthony mean Anderson. in you in dialogue: A study conceptual Cognition and , semantic Gove, co-ordination. Philip, editor. 1984. Webster’s 27:181–218. Dictionary of Synonyms . Merriam-Webster. New Hardt, Generation Daniel of and VP-ellipsis: Owen Rambow. 2001. approach. In Proceedings of A corpus-based of Computation the 39th (ACL-01) Association , for Linguistics Meeting the Malt, Barbara, pages Steven 282–289. Gennari, Meiyi Shi, Sloman, Knowing and Yuan Silvia naming: Similarity Wang. 1999. versus and Reiter and Sripada the linguistic Journal of categorization McKeown, Memory of Kathleen, Language artifacts. and Jacques , 40:230–262. Michael Tanenblatt. 1993. Robin, and Tailoring in lexical choice to the multimedia user’s vocabulary Proceedings explanation of 31st generation. Meeting of In Computational Annual Linguistics the Association (ACL93), for Nunberg, Geoffrey. pages 226–234. Reference . 1978. University of The Pragmatics Indiana of Linguistics Parikh, Rohit. Club, The 1994. Bloomington. of Vagueness nouns. and utility: Linguistics semantics Reiter, Ehud. Philosophy common and 1991. new , 17:521–535. nouns. A model of Computational lexical choice Intelligence for Reiter, Ehud, , Osman. Roma 7(4):240–251. Knowledge Robertson, and Liesl natural 2000. acquisition In for Proceedings language on of the First generation. Conference Natural Language International Generation Reiter, pages Ehud 217–215. Should and Somayajulu Sripada. 2002. corpora texts be gold standards , Human Variation and Lexical Choice for NLG? In International Proceedings Conference of on the Second Generation , Natural Language Rosch, Eleanor. pages 97–104. 1978. In Principles E. Rosch of categorization. Cognition and B. Lloyd, editors, Lawrence and Erlbaum, Categorization Hillsdale, NJ, . Smith, pages Edward 27–48. Categories and Douglas Concepts . Medin. Harvard 1981. University and Sripada, Press, Cambridge. Somayajulu, Hunter, Jin Yu. Ehud Reiter, Jim model and 2001. A two-stage Proceedings for content Stede, of ENLGW-2001 determination. 1995. , In Manfred. Lexicalization pages 3–10. natural in language Intelligence generation: Review , A survey. Wanner, Artificial Leo. 1996. Lexical 8:309–336. machine choice in text generation Machine Translation and , 11:3–35. translation. Zukerman, Natural Ingrid and processing Diane Litman. 2001. modeling: language Synergies and user Modeling User-Adapted and limitations. Interaction , User 11:129–158. and