<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="31"/><title>Exploring Approaches to Discriminating among Near-Synonyms</title><pubinfo>Proceedings of the Australasian Language Technology Workshop 2007, pages 31-39</pubinfo><author surname="Gardiner" givenname="Mary"><org  name="Mie University" country="Japan" city="Mie"/></author><author surname="Dras" givenname="Mark"><org  name="Macquarie University" country="Australia" city="Sydney"/></author></firstpageheader><frontmatter><p><b>Exploring approaches to discriminating among near-synonyms</b></p><p><b>Mary Gardiner</b></p><p>Centre for Language Technology</p><p>Macquarie University gardiner@ics.mq.edu.au</p><p><b>Mark Dras</b></p><p>Centre for Language Technology Macquarie University madras@ics.mq.edu.au</p></frontmatter><abstract>Near-synonyms are words that mean ap­proximately the same thing, and which tend to be assigned to the same leaf in ontolo­gies such as WordNet. However, they can differ from each other subtly in both mean­ing and usage—consider the pair of near-synonyms <i>frugal </i>and <i>stingy</i>—and therefore choosing the appropriate near-synonym for a given context is not a trivial problem. Initial work by Edmonds (1997) suggested that corpus statistics methods would not be particularly effective, and led to subsequent work adopting methods based on specific lexical resources. In earlier work (Gardiner and Dras, 2007) we discussed the hypoth­esis that some kind of corpus statistics ap­proach may still be effective in some situa­tions, particularly if the near-synonyms dif­fer in sentiment from each other, and we presented some preliminary confirmation of the truth of this hypothesis. This suggests that problems involving this type of near-synonym may be particularly amenable to corpus statistics methods. In this paper we investigate whether this re­sult extends to a different corpus statistics method and in addition we analyse the re­sults with respect to a possible confound­ing factor discussed in the previous work: the skewness of the sets of near synonyms. Our results show that the relationship be­tween success in prediction and the nature of the near-synonyms is method dependent and that skewness is a more significant factor. </abstract></header><body><section number="1" title="Introduction"><p>Choosing an appropriate word or phrase from among candidate near-synonyms or paraphrases is a significant language generation problem since even though near-synonyms and paraphrases are close in meaning, they differ in connotation and denotation in ways that may be significant to the desired effect of the generation output: for example, word choice can change a sentence from advice to admonish­ment. Particular applications that have been cited as having a use for modules which make effective word and phrase choices among closely related options are summarisation and rewriting (Barzilay and Lee, 2003). Inkpen and Hirst (2006) extended the genera­tion system HALogen (Langkilde and Knight, 1998; Langkilde, 2000) to include such a module.</p><p>We discuss a particular aspect of choice between closely related words and phrases: choice between words when there is any difference in meaning or at­titude. Typical examples are <i>frugal </i>and <i>stingy; slen­der </i>and <i>skinny; </i>and <i>error </i>and <i>blunder.</i></p><p>In this paper, as in Gardiner and Dras (2007), we explore whether corpus statistics methods have promise in discriminating between near-synonyms with attitude differences, particularly compared to near-synonyms that do not differ in attitude. In our work, we used the work of (Edmonds, 1997), the first to attempt to distinguish among near-synonyms, adopting a corpus statistics approach. Based on that work, we found that there was a significant difference in attitudinal versus non-attitudinal near-synonyms. However, the Edmonds algorithm pro­duced on the whole poor results, only a little above the given baseline, if at all. According to (Inkpen, 2007), the poor results were due to the way the algorithm handled data sparseness; she consequently presented an alternative algorithm with much bet­ter results.<page local="2" global="32"/> We also found that attitudinal versus non-attitudinal near-synonyms differed significantly in their baselines as a consequence of skewness of synset distribution, complicating analysis.</p><p>In this paper then we develop an algorithm based on that of Inkpen, and use a far larger data set and a methodology suited to large data sets, to see whether this alternative method will support our pre­vious findings. In addition we analyse results with regard to a measure of synset skewness. In Sec­tion 2 we outline the near-synonym task descrip­tion; in Section 3 we present our method based on Inkpen; in Section 5 we present out method based on Inkpen, and our experimental method using it; in Section 4 we evaluate its effectiveness in compari­son with Inkpen's own method; in Section 5 we test our hypothesis, present our results and discuss them; and in Section 6 we conclude.</p></section><section number="2" title="Task Description"><p>Our experiment tests a system's ability to fill a gap in a sentence from a given set of near-synonyms. This problem was first described by Edmonds (1997). Edmonds describes an experiment that he designed to test whether or not co-occurrence statistics are sufficient to predict which word in a set of near-synonyms fills a <i>lexical gap. </i>He gives this example of asking the system to choose which of <i>error, mis­take </i>or <i>oversight </i>fits into the gap in this sentence:</p><p>(1) However, such a move also of cutting deeply into U.S. economic growth, which is why some economists think it would be a big_.</p><p>Performance on the task is measured by compar­ing system performance against real word choices: that is, sentences such as example 1 are drawn from real text, a word is removed, and the system is asked to choose between that word and all of its near-synonyms as candidates to fill the gap.</p></section><section number="3" title="An approximation to Inkpen's solution"><p><b>to the near-synonym choice problem</b></p><p>We know of two descriptions of algorithms used to choose between near-synonyms based upon context: that described by Edmonds (1997) and that de­scribed by Inkpen (2007).</p><p>In our previous work we used Edmonds' method for discriminating between near-synonyms as a ba­sis for comparing whether near-synonyms that differ in attitude in predictability from near-synonyms that do not. The more recent work by Inkpen is a more robust and reliable approach to the same problem, and therefore in this paper we develop a methodol­ogy based closely on that of Inkpen, using a differ­ent style of training corpus, in order to test whether the differences between the performance of near-synonyms that differ in sentiment and those that do not persists on the better performing method.</p><p>Edmonds' and Inkpen's approaches to near-synonym prediction have the same underlying hy­pothesis: that the choice between near-synonyms can be predicted to an extent from the words im­mediately surrounding the gap. Returning to exam­ple 1, their approaches use words around the gap, eg <i>big, </i>to predict which of <i>error, mistake </i>or <i>oversight </i>would be used. They do this using some measure of how often <i>big, </i>and other words surrounding the gap, is used in contexts where each of <i>error, mistake </i>and <i>oversight </i>are used. Edmonds uses every word in the sentence containing the gap, whereas Inkpen uses a generally smaller window of words surrounding the gap.</p><p>In this section we briefly describe Edmonds' ap­proach to discriminating between near-synonyms in Section 3.1 and describe Inkpen's approach in more detail in Section 3.2. We then describe our adapta­tion of Inkpen's approach in Section 3.3.</p><subsection number="3.1" title="Edmonds' approach"><p>In Edmonds' approach to the word choice problem, the suitability of any candidate word c for a sentence S can be approximated as a score(c, S) of suitabil­ity, and where score(c,S) is a sum of the association between the candidate c and every other word w in the sentence.</p><doubt alpha="46.2" length="26" tooSmall="False" monospace="0.0">(2) score(c, S) =sig(c, w)</doubt><p>In Edmonds' original method, which we used in Gardiner and Dras (2007), sig(c, w) is computed us­ing either the t-score of c and w or a second degree association: a combination of the t-scores of c with a word wo and the same word wo with w.<page local="3" global="33"/> Edmonds' t-scores were computed using co-occurrence counts in the 1989 Wall Street Journal, and the performance did not improve greatly over a baseline of choosing the most frequent word in the synset to fill all gaps.</p></subsection><subsection number="3.2" title="Inkpen's approach"><p>In Inkpen's method, the suitability of candidate c for a given gap is approximated slightly differently: the entire sentence is not used to measure the suitability of the word. Instead, a certain sized window of k words either side of the gap is used. For example, if k = 3, the word missing from the sentence in exam­ple 3 is predicted using only the six words shown in example 4.</p><p>(3) Visitors to Istanbul often sense a second, _layer beneath the city's tangible beauty.</p><p>(4) sense a second,_layer beneath the</p><p>Given a text fragment f consisting of 2k words, k words either side of a gap g (wi, w2,... <i>,wk </i><i>,g,wk+x,...,w2k</i><i> </i>), the suit­ability s(c,g) of any given candidate word c to fill the gap g is given by:</p><doubt alpha="50.0" length="4" tooSmall="False" monospace="0.0">k 2k</doubt><doubt alpha="48.6" length="35" tooSmall="False" monospace="0.0">(5)s(c,g) =]TPMI(c,wj) +J2PMI(wj,c)</doubt><doubt alpha="33.3" length="9" tooSmall="False" monospace="0.0">j=1 j=k+1</doubt><p>PMI(x,y) is the pointwise mutual information score of two words x and y, and is given by (Church</p><doubt alpha="47.1" length="17" tooSmall="False" monospace="0.0">and Hanks, 1991):</doubt><doubt alpha="47.4" length="19" tooSmall="False" monospace="0.0">(6)PMI(x,y)='°g2ira</doubt><p>C (x), <i>C</i><i> </i><i>(y)</i><i> </i>and <i>C</i><i> </i><i>(x,y)</i><i> </i>are estimated using to­ken counts in a corpus: C(x,y) is the number of times that x and y are found together, C(x) is the total number of occurrences of x in the corpus and C(y) the total numberofoccurrences ofy in the cor­pus. N is the total number of words in the text.</p><p>Inkpen estimated C(x), C(y) and C(x, y) by issuing queries to the Waterloo MultiText Sys­tem (Clarke and Terra, 2003). She defined C(x, y) the number of times where x is followed by y within a certain <i>query frame </i>of length q within a corpus, so that, for example, if q = 3, example 7 would count as a co-occurrence of <i>fresh </i>and <i>mango, </i>but exam­ple 8 would not:</p><p>(7) He likes <i>fresh </i>cold <i>mango.</i></p><p>(8) I like <i>fresh </i>fruits in general, particularly <i>mango.</i></p><p>She also experimented with document counts where C(x) is the number of documents that x is found in and C (x, y) is the number of documents in which both x and y are found, called PMI-IR (Tur-ney, 2001); but found that this method did not per­form as well, although the difference was not statis­tically significant.</p><p>Inkpen's method outperformed both the baseline and Edmonds' method by 22 and 10 percentage points respectively.</p></subsection><subsection number="3.3" title="Our variation of Inkpen's approach"><p>Our variation on Inkpen's approach is designed to estimate PMI(x, y), the pointwise mutual informa­tion of words x and y, using the Web 1T 5-gram corpus Version 1 (Brants and Franz, 2006).</p><p>Web 1T contains n-gram frequency counts, up to and including 5-grams, as they occur in a trillion words of World Wide Web text. There is no con­text information beyond the n-gram boundaries. Ex­amples of a 3-gram and a 5-gram and their respec­tive counts from Web 1T are shown in examples 9</p><doubt alpha="42.9" length="7" tooSmall="False" monospace="0.0">and 10:</doubt><doubt alpha="66.7" length="24" tooSmall="False" monospace="0.0">(9)means official and 41</doubt><doubt alpha="58.3" length="36" tooSmall="False" monospace="0.0">(10)Valley National Park 1948 Art 51</doubt><p>These n-gram counts allow us to estimate C(x, y) for a given window width k by summing the Web 1T counts of k-grams in which words x and y occur and x is followed by y.</p><p>Counts are computed using a an especially de­veloped version of the Web 1T processing software "Get 1T"<footnote anchor="1"/> originally described in Hawker (2007) and detailed in Hawker et. al (2007). The Get 1T software allows n-gram queries of the form in the following examples, where &lt;*&gt; is a wildcard which</p><p>Available at http://get1t.sf.net/<page local="4" global="34"/></p><p>will match any token in that place in the n-gram. In order to find the number of n-grams with <i>fresh </i>and <i>mango </i>we need to construct three queries:</p><doubt alpha="52.6" length="19" tooSmall="False" monospace="0.0">(11)&lt;*&gt; fresh mango</doubt><doubt alpha="52.6" length="19" tooSmall="False" monospace="0.0">(12)fresh &lt;*&gt; mango</doubt><doubt alpha="52.6" length="19" tooSmall="False" monospace="0.0">(13)fresh mango &lt;*&gt;</doubt><p>However, in order to find <i>fresh </i>and <i>mango </i>within 4 grams we need multiple wildcards as in exam­ple 14, and added the embedded query hashing func­tionality described in Hawker et. al (2007).</p><doubt alpha="43.5" length="23" tooSmall="False" monospace="0.0">(14)fresh &lt;*&gt; &lt;*&gt; mango</doubt><p>Queries are matched case-insensitively, but no stemming takes place, and there is no deeper anal­ysis (such as part of speech matching).</p><p>This gives us the following methodology for a given lexical gap g and a window of k words either side of the gap:</p></subsection></section><section number="1." title="for every candidate near-synonym c:"><p>(a) for every word wj in the set of words proceeding the gap, w1 , . . . , w<i>k,</i><i> </i>calculate PMI(wj, c) as in equation 6, given counts for C(wj), C(c) and C(w^c) from Web 1T<footnote anchor="2"/></p><p>(b) for every word <i>wj</i><i> </i>in the set of words fol­lowing the gap, wk+1,..., w2k, calculate PMI(c, wj) as in equation 6, given counts for C(c), C(wj) and C(c, wj) from Web</p><doubt alpha="50.0" length="2" tooSmall="False" monospace="0.0">1T</doubt><p>(c) compute the suitability score s(c,g) of candidate c as given by equation 5</p><p>2. select the candidate near-synonym with the highest suitability score for the gap where a sin­gle such candidate exists</p><p>3. where there is no single candidate with a high­est suitability score, select the most frequent candidate for the gap (that is, fall back to the baseline described in Section 3.4)<footnote anchor="3"/></p><footnote label="2">The result of equation 6 is undefined when any of C(x) = 0, C(y) = 0 or C (x,y) = 0 hold, that is, x or y or at least one n-gram containing x and y cannot be found in the Web 1T counts. For the purpose of computing s(c,g), we define PMI(x, y) = 0 when C(x) = 0, C (y) = 0 or C (x, y) = 0, so that it has no influence on the score s(c, g) given by equation 5.</footnote><footnote label="3">Typically, in this case, all candidates have scored 0.</footnote><p>Since Web 1T contains 5-gram counts, we can use query frame sizes from q = 1 (words x and y must be adjacent, that is, occur in the 2-gram counts) to q = 4.</p><subsection number="3.4" title="Baseline method"><p>The baseline method that our method is compared to uses the most frequent word from a given synset as the chosen candidate for any gap requiring a mem­ber of that synset. Frequency is measured using fre­quency counts of the combined part of speech and word token in the 1989 Wall Street Journal.</p></subsection></section><section number="4" title="Effectiveness of the approximation to Inkpen's method"><p>In this section we compare our approximation of Inkpen's method described in Section 3.3 with her method described in Section 3.2. This will allow us to determine whether our approximation is effective enough to allow us to compare attitudinal and non-attitudinal near-synonyms.</p><subsection number="4.1" title="Test sets"><p>In order to compare the two methods, we use five sets of near-synonyms, also used as test sets by both Edmonds and Inkpen:</p><p>• the adjectives <i>difficult, hard </i>and <i>tough;</i></p><p>• the nouns <i>error, mistake </i>and <i>oversight;</i></p><p>• the nouns <i>job, task </i>and <i>duty;</i></p><p>• the nouns <i>responsibility, burden, obligation </i>and <i>commitment; </i>and</p><p>• the nouns <i>material, stuff</i>and <i>substance.</i></p><p>Inkpen compared her method to Edmonds' using these five sets and two more, both sets of verbs, which we have not tested on, as our attitudinal and non-attitudinal data does not included annotated verbs. We are therefore interested in the predictive power of our method compared to Inkpen's and Ed-mond's on adjectives and nouns.</p></subsection><subsection number="4.2" title="Test contexts"><p>We performed this experiment, as Edmonds and Inkpen did, using the 1987 Wall Street Journal as a source of test sentences.<page local="5" global="35"/><footnote anchor="4"/> Where ever one of the words in a test set is found, it is removed from the context in which it occurs to generate a gap for the algorithm to fill.</p><p>So, for example, when sentence 15 is found in the test data, the word <i>error </i>is removed from it and the system is asked to predict which of <i>error, mistake </i>or <i>oversight </i>fills the gap at 16:</p><p>(15) .. .his adversarys' characterization of that minor sideshow as somehow a colossal <i>er­ror </i>on the order of a World War. . ..</p><doubt alpha="63.6" length="22" tooSmall="False" monospace="0.0">(16) a colossal_on the</doubt></subsection><subsection number="4.3" title="Parameter settings"><p>Recall from Section 3.2 these two parameters used by Inkpen: k and q.</p><p>Parameter k is the size of the 'window' of context on either side of a lexical gap in <i>the test set: </i>the k words on either side of a gap are used to predict which of the candidate words best fills the gap.</p><p>Parameter q is the query size used when querying <i>the corpus </i>to find out how often words x and y occur together in order to compute the value of C(x, y). In order to be counted as occurring together, x and y must occur within a window of length at most q.</p><p>Inkpen found, using Edmonds' near-synonym set <i>difficult </i>and <i>hard </i>as a development set, that results are best for a small window (k G {1,2}) but that the query frame had to be somewhat longer to get the best results. Her results were reported using k = 2 and q = 5, chosen via tuning on the development set.</p><p>We have retained the setting k = 2 and explored results where q = 2 and q = 4: due to Web 1T containing 5-grams but no higher order n-grams we cannot measure the frequency of two words occur­ring together with any more than three intervening words, so q = 4 is the highest value q can have.</p></subsection><subsection number="4.4" title="Results and Discussion"><table caption="Table 1 shows the performance of Edmonds' method and Inkpen's method as given in Inkpen (2007)5and"></table><footnote label="4">All references to the Wall Street Journal data used in this paper refer to Charniak et. al (2000).</footnote><footnote label="5">Inkpen actually gives two methods, one using PMI esti­mates from document counts, one using PMI estimates using word counts. Here we are discussing her word count method and use those values in our table.</footnote><p>our modified method on each of the test sets de­scribed in Section 4.1. Note that Inkpen reports dif­ferent baseline results from us—we have not been able to reproduce her baselines. This may be due to choosing different part of speech tags: we simply used JJ for adjectives and NN for nouns.</p><p>Inkpen's improvements for the test synsets given in Section 4.1 were between +3.2% and 30.6%. Our performance is roughly comparable, with improve­ments as high as 31.2%. Further, we tend to improve especially largely over the baseline where Inkpen also does so: on the two sets <i>error </i>etc and <i>responsi­bility </i>etc..</p><p>The major anomaly when compared to Inkpen's performance is the set <i>job, task </i>and <i>duty, </i>where our method performs very badly compared to both Edmonds' and Inkpen's methods and the baseline (which perform similarly). We also perform under both methods on <i>material, stuff</i>and <i>substance, </i>al­though not as dramatically.</p><p>Overall, the fact that we tend to improve over Ed­monds where Inkpen also does so suggests that our algorithm based on Inkpen's takes advantage of the same aspects as hers to gain improvements over Ed­monds, and thus that the method is a good candidate for use in our main experiment.</p></subsection></section><section number="5" title="Comparing attitudinal and non-attitudinal synsets"><p>Having determined in Section 4 that our modified version of Inkpen's method performs as a passable approximation of hers, and particularly that where her method improved dramatically over the baseline and Edmonds' method that ours improves likewise, we then tested our central hypothesis: that attitu-dinal synsets respond better to statistical prediction techniques than non-attitudinal synsets.</p><subsection number="5.1" title="Test set"><p>In order to test our hypothesis, we use synsets di­vided into near-synonym sets that differ in attitudi-nal and sets that do not.</p><p>This test set is drawn from our set of annotated <i>attitudinal </i>and <i>non-attitudinal </i>near-synonyms de­scribed in Gardiner and Dras (2007). These are WordNet2.0 (Fellbaum, 1998) synsets whose mem­bers occur particularly frequently in the 1989 Wall<page local="6" global="36"/></p><doubt alpha="14.3" length="7" tooSmall="False" monospace="0.0">(k = 2)</doubt><p>Street Journal. The synsets were annotated as <i>at-titudinal </i>and <i>non-attitudinal </i>by the authors of this paper. Synsets were chosen where both annotators are certain of their label, and where both annotators have the same label. This results in 60 synsets in to­tal: 8 where the annotators agreed that there was def­initely an attitude difference between words in the synset, and 52 where the annotators agreed that there were definitely not attitude differences between the words in the synset.</p><p>An example of a synset agreed to have attitudinal differences was:</p><p>(17) <i>bad, insecure, risky, high-risk, speculative</i></p><p>An example of synsets agreed to not have attitu-dinal differences was:</p><p>(18) <i>sphere, domain, area, orbit, field, arena</i></p><p>The synsets are not used in their entirety, due to the differences in the number of words in each synset (compare <i>{violence, force} </i>with two members to { <i>arduous, backbreaking, grueling, gruelling, hard, heavy, laborious, punishing, toilsome} </i>with nine, for example). Instead, a certain number n of words are selected from each synset (where n G {3,4}) based on the frequency count in the 1989 Wall Street Journal corpus. For example <i>hard, heavy, gruelling </i>and <i>punishing </i>are the four most frequent words in the { <i>arduous, backbreaking, grueling, gruelling, hard, heavy, laborious, punishing, toilsome} </i>synset, so when n = 4 those four words would be selected. When the synset's length is less than or equal to n, for example when n = 4 but the synset is { <i>violence, force} </i>, the entire synset is used.</p><p>These test sets are referred to as <i>top3 </i>(synsets re­duced to 3 or less members) and <i>top4 </i>(synsets re­duced to 4 or less members).</p></subsection><subsection number="5.2" title="Test contexts"><p>Exactly as in Section 4.2, our lexical gaps and their surrounding contexts are drawn from sentences in the 1987 Wall Street Journal containing one of the words in the test synsets.</p></subsection><subsection number="5.3" title="Parameter settings"><p>As described in Sections 3.2 and 4.3, there are two parameters that can be varied regarding the context around a lexical gap (k), and the nearness of two words x and y in the corpus in order for them to be considered to occur together (q).</p><p>As per Inkpen's results on her development set, and as in Section 4 we use the setting k = 2 and vary q such that q = 2 on some test runs and q = 4 on others. We cannot test with Inkpen's suggested q = 5, as that would require 6-grams.</p></subsection><subsection number="5.4" title="Results and Discussion"><p>The overall performance of our method on our sets of attitudinal and non-attitudinal near-synonyms is shown in Table 2.</p><p>We did four test runs in total, two each on sets <i>top3 </i>and <i>top4 </i>varying q between q = 2 and q = 4. The baseline result does not depend on q and there­fore is the same for both tests of <i>top3 </i>and of <i>top4.</i></p><table caption="Table 1: Performance of Inkpen's test sentences on Edmond's method, Inkpen's method and our method" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Set</p></td><td class="cell"><p>Inkpen's</p></td><td class="cell"><p>Edmonds'</p></td><td class="cell"><p>Inkpen's</p></td><td class="cell"><p>No. test</p></td><td class="cell"><p>Our base-</p></td><td class="cell"><p>Our</p></td><td class="cell"><p>in-</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>baseline</p></td><td class="cell"><p>increase</p></td><td class="cell"><p>increase</p></td><td class="cell"><p>sentences</p></td><td class="cell"><p>line value</p></td><td class="cell"><p>crease</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>value %</p></td><td class="cell"><p>over</p></td><td class="cell"><p>over</p></td><td class="cell"><p>we found</p></td><td class="cell"><p>%</p></td><td class="cell"><p>over</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>baseline</p></td><td class="cell"><p>baseline</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>baseline</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>%</p></td><td class="cell"><p>(q = 5) %</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>%</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>q = 2</p></td><td class="cell"><p>q = 4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>difficult etc.</p></td><td class="cell"><p>41-7</p></td><td class="cell"><p>+6-2</p></td><td class="cell"><p>+17-4</p></td><td class="cell"><p>5959</p></td><td class="cell"><p>44-3</p></td><td class="cell"><p>+15-3</p></td><td class="cell"><p>+12-3</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>error etc.</p></td><td class="cell"><p>30-9</p></td><td class="cell"><p>+18-9</p></td><td class="cell"><p>+30-6</p></td><td class="cell"><p>1026</p></td><td class="cell"><p>46-8</p></td><td class="cell"><p>+25-5</p></td><td class="cell"><p>+20-4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>job etc.</p></td><td class="cell"><p>70-2</p></td><td class="cell"><p>-1-3</p></td><td class="cell"><p>+3-2</p></td><td class="cell"><p>4020</p></td><td class="cell"><p>74-2</p></td><td class="cell"><p>-14-4</p></td><td class="cell"><p>-23-0</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>responsibility etc.</p></td><td class="cell"><p>38-0</p></td><td class="cell"><p>+7-3</p></td><td class="cell"><p>+28-0</p></td><td class="cell"><p>1119</p></td><td class="cell"><p>36-7</p></td><td class="cell"><p>+31-2</p></td><td class="cell"><p>+24-9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>material etc.</p></td><td class="cell"><p>59-5</p></td><td class="cell"><p>+5-1</p></td><td class="cell"><p>+12-7</p></td><td class="cell"><p>934</p></td><td class="cell"><p>57-8</p></td><td class="cell"><p>+5-5</p></td><td class="cell"><p>-1-1</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="7" global="37"/><table caption="Table 2: Performance of the baseline and our method on all test sentences(k= 2)"></table><table caption="Table 3: Distribution of improvements on baseline fortop3,k= 2,q= 2"></table><p>As in our previous paper (Gardiner and Dras, 2007), the baselines behave noticeably differently for attitudinal and non-attitudinal synsets. Calculat­ing the z-statistic as is standard for comparing two proportions (Moore and McCabe, 2003) we find that the difference between the pair of attitudinal and non-attitudinal results for each test are all statisti­cally significant (p &lt; 0.01). Thus, again, it is dif­ficult from the data in Table 2 alone to determine whether the better performance of non-attitudinal synsets is due to the higher baseline performance for those same synsets.</p><p>There are two major aspects of this result requir­ing further investigation. The first is that our method performs very similarly to the baseline according to these aggregate numbers, which wasn't anticipated based on the results in Section 4, which showed that on a limited set of synsets our method performed well above the baseline, although not as well as Inkpen's original method.</p><p>Secondly, inspection of individual synsets and their performance reveals that this aggregate is not representative of the performance as a whole: it is simply an average of approximately equal numbers of good and bad predictions by our method. Table 3 shows that for one test run <i>(top3, </i>k = 2, q = 2) there were a number of synsets on which our method per­formed very well with an improvement of more than 20 percentage points over the baseline but also a sub­stantial number where it performed very badly, los­ing more than 20 percentage points from the base­line.</p><p>In our previous work we expressed a suspicion that the success of Edmonds' prediction method might be being influenced by the evenness of dis­tribution of frequencies within a synset. That is, if a synset contains a very dominant member (which will cause the baseline to perform well) then the Ed­monds method may perform worse against the base­line than it would for a synset in which the word choices were distributed fairly evenly among the members of the set.</p><p>Given the results of the test runs shown in Table 2, and the wide distribution of prediction successes shown in Table 3, we decided to test this hypothe­sis that the distribution of words in the synsets influ­ence the performance of prediction methods that use context. This is described in Section 5.4.1.</p><subsubsection number="5.4.1" title="Entropy analysis"><p>In this section, we describe an analysis of the re­sults in Section 5.4 in terms of whether the balance of frequencies among words in the synset contribute to the quality of our prediction result.</p><p>In order to measure a correlation between the bal­ance of frequencies of words and the prediction re­sult, we need a measure of 'balance'. In this case we have chosen information entropy (Shannon, 1948), the measure of bits of information required to con­vey a particular result. The entropy of a synset's frequencies here is measured using the proportion<page local="8" global="38"/></p><table caption="Table 2: Performance of the baseline and our method on all test sentences (k = 2)" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Contexts containing a test</p></td><td class="cell"><p>Baseline correctness (%)</p></td><td class="cell"><p>q</p></td><td class="cell"><p>Our method's correctness</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>word</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>(%)</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Synsets</p></td><td class="cell"><p>Att.</p></td><td class="cell"><p>Non-att.</p></td><td class="cell"><p>Att.</p></td><td class="cell"><p>Non-att.</p></td><td class="cell"><p></p></td><td class="cell"><p>Att.</p></td><td class="cell"><p>Non-att.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>top3</p></td><td class="cell"><p>45953</p></td><td class="cell"><p>353155</p></td><td class="cell"><p>59-52</p></td><td class="cell"><p>69-71</p></td><td class="cell"><p>2</p></td><td class="cell"><p>59-51</p></td><td class="cell"><p>69-95</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>4</p></td><td class="cell"><p>56-93</p></td><td class="cell"><p>69-96</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>top4</p></td><td class="cell"><p>48515</p></td><td class="cell"><p>357290</p></td><td class="cell"><p>56-37</p></td><td class="cell"><p>68-91</p></td><td class="cell"><p>2</p></td><td class="cell"><p>52-26</p></td><td class="cell"><p>67-60</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p>4</p></td><td class="cell"><p>50-82</p></td><td class="cell"><p>67-59</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 3: Distribution of improvements on baseline fortop3,k = 2, q = 2" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Improvement over</p></td><td class="cell"><p>Number of synsets</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>baseline</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Att.</p></td><td class="cell"><p>Non-att.</p></td><td class="cell"><p>Total</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>&gt; +20%</p></td><td class="cell"><p>0</p></td><td class="cell"><p>16</p></td><td class="cell"><p>16</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>&gt; +10% and &lt; +20%</p></td><td class="cell"><p>1</p></td><td class="cell"><p>7</p></td><td class="cell"><p>8</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>&gt; +5% and &lt; +1%</p></td><td class="cell"><p>2</p></td><td class="cell"><p>2</p></td><td class="cell"><p>4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>&gt; -5% and &lt; -5%</p></td><td class="cell"><p>2</p></td><td class="cell"><p>10</p></td><td class="cell"><p>12</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>&lt; -5% and &gt; -10%</p></td><td class="cell"><p>0</p></td><td class="cell"><p>6</p></td><td class="cell"><p>6</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>&lt; -10% and &gt; -20%</p></td><td class="cell"><p>1</p></td><td class="cell"><p>3</p></td><td class="cell"><p>4</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>&lt; -20%</p></td><td class="cell"><p>1</p></td><td class="cell"><p>8</p></td><td class="cell"><p>9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Table 4: Regression co-efficients between inde­pendent variables synset category and synset en­tropy, and dependent variable prediction improve­ment over baseline (statistically significant results p &lt; 0.05 marked *) of total uses of the synset that each particular word represents. A synset in which frequencies are rea­sonably evenly distributed has high information en­tropy and a synset in which one or more words are very frequent as a proportion of use of that synset as a whole have low entropy.</p><p>We then carried out multiple regression analysis using the category of the synset (attitudinal or not attitudinal, coded as 1 and 0 for this analysis) and the entropy of the synset's members' frequencies as our two independent variables; this allows us to sep­arate out the two effects of synset skewness and at-titudinality. Regression co-efficients are shown in</p><p>Table 4.</p><p>Table 4 shows that in general, performance is neg­atively correlated with both category but positively with entropy, although the correlation with category is not always significant. The positive relationship with entropy confirms our suspicion in Gardiner and Dras (2007) that statistical techniques perform bet­ter when the synset does not have a highly dominant member. The negative correlation with category im­plies that the reverse of our main hypothesis holds: that our statistical method works better for predict­ing the use of non-attitudinal near-synonyms.</p><p>There are two questions that arise from the result that our Inkpen-based method gives a different result from the Edmonds-based one.</p><p>First, is our approximation to Inkpen's method in­herently faulty or can it be improved in some way? We know from Section 4 that it tends to perform well where her method performs well. An obvious second test is to compare our results to another test described in Inkpen (2007) which used a larger set of near-synonyms and tested the predictive power using the British National Corpus as a source of test contexts. This test will test our system's per­formance in genres quite different from news-wire text, and allow us to make a further comparison with Inkpen's method.</p><p>Second, why do we perform significantly better for near-synonyms without attitude difference? One possible explanation that we intend to explore is that attitude differences are predicted by attitude differ­ences exhibited in a very large context; perhaps an entire document or section thereof. Sentiment anal­ysis techniques may be able to be used to detect the attitude bearing parts of a document and these may serve as more useful features for predicting attitudi-nal word choice than surrounding words.</p></subsubsection></subsection></section><section number="6" title="Conclusion and Future Work"><p>In this paper we have developed a modification to Inkpen's method of making a near-synonym choice that on a set of her test data performs reasonably promisingly; however, when tested on a larger set of near-synonyms on average it does not perform very differently to the baseline. We have also shown that, contrary to our hypothesis that near-synonyms with attitude differences would perform better us­ing statistical methods, on this method the near-synonyms without attitude differences are predicted better when there's a difference in predictive power.</p><p>Ultimately, we plan to develop a system that will acquire and predict usage of attitudinal near-synonyms, drawing on statistical methods and meth­ods from sentiment analysis. In order to achieve this we will need a comprehensive understanding of why this method's performance was not adequate for the task.</p></section><section title="Acknowledgements"><p>Thank you to: Diana Inkpen for sending us a copy of Inkpen (2007) while it was under review; and Tobias Hawker for providing a copy of his Web 1T process­ing software, Get 1T, before its public release.</p><p>This work has been supported by the Aus­tralian Research Council under Discovery Project</p><p>DP0558852.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Test set</p></td><td class="cell"><p>q</p></td><td class="cell"><p>Category</p></td><td class="cell"><p>Entropy</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>top3</p></td><td class="cell"><p>2</p></td><td class="cell"><p>-0-11</p></td><td class="cell"><p>0-41*</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>top3</p></td><td class="cell"><p>4</p></td><td class="cell"><p>-0-10</p></td><td class="cell"><p>0-36*</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>top4</p></td><td class="cell"><p>2</p></td><td class="cell"><p>-0-17*</p></td><td class="cell"><p>0-38*</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>top4</p></td><td class="cell"><p>4</p></td><td class="cell"><p>-0-15*</p></td><td class="cell"><p>0-34*</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="9" global="39"/></section><references><p>Regina Barzilay and Lillian Lee. 2003. Learning to paraphrase: An unsupervised approach using multiple-sequence alignment. In <i>HLT-NAACL 2003: Main Pro­ceedings, </i>pages 16-23.</p><p>Thorsten Brants and Alex Franz. 2006. Web 1T 5-gram Version  1. http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId= LDC2006T13.</p><p>Eugene Charniak, Don Blaheta, Niyu Ge, Keith Hall, 1987-89 WSJ Corpus Release 1. http://www.</p><doubt alpha="62.8" length="43" tooSmall="False" monospace="0.0">John Hale, , and Mark Johnson.  2000. BLLIP</doubt><p>ldc.upenn.edu/Catalog/CatalogEntry. jsp?catalogId=LDC2 0 0 0T4 3.</p><p>Kenneth Church and Patrick Hanks. 1991. Word asso­ciation norms and mutual information, lexicography. <i>Computational Linguistics, </i>16(1):22-29.</p><p>Charles L. A. Clarke and Egidio L. Terra. 2003. Passage retrieval vs. document retrieval for factoid question an­swering. In <i>Proceedings of the 26th Annual Interna­tional ACM SIGIR Conference on Research and De­velopment in Information Retrieval, </i>pages 427-428, Toronto, Canada.</p><p>Philip Edmonds. 1997. Choosing the word most typical in context using a lexical co-occurrence network. In <i>Proceedings of the 35th Annual Meeting of the Associ­ation for Computational Linguistics and the 8th Con­ference ofthe European Chapter ofthe Association for Computational Linguistics, </i>pages 507-509, July.</p><p>Christiane Fellbaum, editor. 1998. <i>WordNet: An Elec­tronic Lexical Database. </i>The MIT Press, May.</p><p>Mary Gardiner and Mark Dras. 2007. Corpus statistics approaches to discriminating among near-synonyms. In <i>Proceedings of the 10th Conference of the Pacific Association for Computational Linguistics (PACLING 2007), </i>pages 31-39, Melbourne, Australia, September.</p><p>Tobias Hawker, Mary Gardiner, and Andrew Bennetts. 2007. Practical queries of a massive n-gram database. In <i>Proceedings ofthe Australasian Language Technol­ogy Workshop 2007 (ALTW 2007), </i>Melbourne, Aus­tralia. To appear.</p><p>Tobias Hawker. 2007. USYD: WSD and lexical substi­tution using the Web 1T corpus. In <i>Proceedings of SemEval-2007: the 4th International Workshop on Se­mantic Evaluations, </i>Prague, Czech Republic.</p><p>Diana Inkpen and Graeme Hirst. 2006. Building and using a lexical knowledge-base of near-synonym dif­ferences. <i>Computational Linguistics, </i>32(2):223-262, June.</p><p>Diana Inkpen. 2007. A statistical model of near-synonym choice. <i>ACM Transactions ofSpeech and Language Processing, </i>4(1):1-17, January.</p><p>Irene Langkilde and Kevin Knight. 1998. The practical value of N-grams in generation. In <i>Proceedings ofthe 9th International Natural Language Generation Work­shop, </i>pages 248-255, Niagra-on-the-Lake, Canada.</p><p>Irene Langkilde. 2000. Forest-based statistical sentence generation. In <i>Proceedings of the 1st Conference of the North American Chapter of the Association for Computational Linguistics and the 6th Conference on Applied Natural Language Processing (NAACL-ANLP 2000), </i>pages 170-177, Seattle, USA.</p><p>David S. Moore and George P. McCabe. 2003. <i>Introduc­tion to the Practice ofStatistics. </i>W. H. Freeman and Company, 4 edition.</p><p>Claude E. Shannon. 1948. A mathematical theory of communication. <i>Bell System Technical Journal, </i>27:379-423 and 623-656, July and October.</p><p>Peter Turney. 2001. Mining the web for synonyms: PMI-IR versus LSA on TOEFL. In <i>Proceedings of the Twelfth European Conference on Machine Learn­ing (ECML 2001), </i>pages 491-502, Freiburg and Ger­many.</p></references></body></article>