<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="49"/><title>Augmenting Approximate Similarity Searching with Lexical Information</title><pubinfo>Proceedings of the Australasian Language Technology Workshop 2005,pages 49-56, Sydney, Australia, December 2005.</pubinfo><author surname="Gorman" givenname="James"><org  name="University of Sydney" country="Australia" city="Sydney"/></author><author surname="Curran" givenname="James R."><org  name="University of Sydney" country="Australia" city="Sydney"/></author></firstpageheader><frontmatter><p><b>Augmenting Approximate Similarity Searching with Lexical Information</b></p><p><b>James Gorman </b>and <b>James R. Curran</b></p><p>School of Information Technologies University of Sydney NSW 2006, Australia (jgorman2,james}@it.usyd.edu.au</p></frontmatter><abstract>Accurately representing synonymy using distribu­tional similarity requires large volumes of data to reliably represent infrequent words. However, the naive nearest-neighbour approach to compare con­text vectors extracted from large corpora scales poorly. The Spatial Approximation Sample Hier­archy (sash) is a data-structure for performing ap­proximate nearest-neighbour queries, and has been previously used to improve the scalability of dis­tributional similarity searches. We add lexical se­mantic information from WordNet to the sas h in an attempt to improve the accuracy and efficiency of similarity searches. </abstract></header><body><section number="1" title="Introduction"><p>Lexical semantic resources and electronic thesauri are regularly used to solve nlp problems, includ­ing collocation discovery (Pearce, 2001), smooth­ing and estimation (Brown et al., 1992; Clark and Weir, 2001) and question answering (Pasca and Harabagiu, 2001). These use similarity relation­ships between words, as given in the resources, to enhance corpus-based statistics.</p><p>It is difficult to account for the needs of the many domains in which nlp techniques are now being ap­plied and for rapid change in language use. Manual creation is expensive and time consuming, and open to the problems of bias, inconsistency and limited coverage. The assisted or automatic creation and maintenance of these resources would be of great advantage.</p><p>Much of the existing work on automatically ex­tracting lexical semantic resources is based on the <i>distributional hypothesis </i>that <i>similar words appear in similar contexts. </i>Terms are described by collat­ing information about their contexts in a corpus into a vector. These <i>context vectors </i>are then compared for similarity. Existing approaches differ primarily in their definition of "context", e.g. the surround­ing words or the entire document, and their choice of distance metric for calculating similarity between the context vectors representing each term.</p><p>Finding synonyms using distributional similarity requires a nearest-neighbour search over the context vectors of each term. This is computationally in­tensive, scaling to the number of terms and the size of their context vectors. Curran and Moens (2002) have demonstrated that dramatically increasing the volume of raw input text used to extract context in­formation significantly improves the quality of ex­tracted synonyms. This will increase the size of the vocabulary, decreasing the efficiency of a naive nearest-neighbour approach.</p><p>Using a data-structure such as the Spatial Ap­proximation Sample Hierarchy (sash; Houle and Sakuma, 2005) allows us to reduce the original <i>0(n) </i>complexity (for an <i>n </i>term vocabulary) to O(logn) (Gorman and Curran, 2005).</p><p>The sash represents the distributional space as a hierarchical directed graph in which each node is connected to several near-neighbour children, deriv­ing its structure from the distribution of the space it represents. The sash is searched by traversing these edges.</p><p>WordNet (Fellbaum, 1998) is an electronic lexi­cal database. The main unit of organisation within WordNet is the synset, which is a collection ofsyn-onymous words. In the case ofnouns, there is a sec­ondary organisation based on hyponymy. The struc­ture of WordNet was derived from a model of how humans understand language.</p><p>WordNet has been used successfully to solve nlp problems. Clark and Weir (2001) use the WordNet hierarchy to improve probability models of noun-predicate relationships. Pearce (2001) uses Word-Net's synsets to improve collocation discovery. We investigate whether using WordNet can improve the accuracy or the efficiency of the sash algorithm by informing the internal representation with gold­standard lexical semantic knowledge.</p></section><section number="2" title="Measuring Distributional Similarity"><p>We are measuring two classes of semantic relation using distributional similarity: synonymy and hyponymy/hypernymy (Curran, 2004).<page local="2" global="50"/> It is hard to distinguish between these two classes using distri­butional similarity.</p><p>Synonymy relates to the nearness of word mean­ing. Very few cases of true synonymy exist. Instead what exists is near-synonymy, where two words are not directly substitutable, but share some close com­mon meaning. The distinction between loud and noisy is an example of this. They both represent the idea of high volume sound, but noisy also has a negative connotation not present in loud.</p><p>Measuring distributional similarity first requires the extraction of context information for each of the vocabulary terms from raw text. These terms are then compared for similarity using a nearest-neighbour search or clustering based on distance calculations between the statistical descriptions of their contexts.</p><subsection number="2.1" title="Extraction Method"><p>A <i>context relation </i>is defined as a tuple <i>(w,r,wr) </i>where is a term, which occurs in some grammat­ical relation with another word in some sen­tence. We refer to the tuple <i>(r,wr) </i>as an <i>attribute </i>of w. For example, (dog, direct-obj, walk) indicates that dog was the direct object of walk in a sentence.</p><p>Context extraction begins with a Maximum Entropy pos tagger and chunker (Ratnaparkhi, 1996). The Sextant relation extractor (Grefen-stette, 1994) produces context relations that are then lemmatised using the Minnen et al. (2000) morpho­logical analyser. The relations for each term are col­lected together and counted, producing a vector of attributes and their frequencies in the corpus.</p><p>The syntactic contexts that are extracted by Sex­tant are:</p></subsection></section><section number="1." title="term is the subject of a verb"></section><section number="2." title="term is the (direct/indirect) object ofa verb"></section><section number="3." title="term is modified by a noun or adjective"></section><section number="4." title="term is modified by a prepositional phrase"><subsection number="2.2" title="Measures and Weights"><p>Both nearest-neighbour and cluster analysis meth­ods require a distance measure to calculate the sim­ilarity between context vectors. Curran (2004) de­composes this into <i>measure </i>and <i>weight </i>functions. The <i>measure </i>function calculates the similarity be­tween two weighted context vectors and the <i>weight </i>function calculates a weight from the raw frequency information for each context relation.</p><p>For these experiments we use the Jaccard (1) measure and the TTest (2) weight functions, as Curran (2004) found them to have the best performance in his comparison of many distance mea­sures.</p><doubt alpha="0.0" length="6" tooSmall="False" monospace="0.0">(1)(1)</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(2)</doubt><doubt alpha="28.6" length="21" tooSmall="False" monospace="0.0">\jp(*, r, w')p(w,*.*)</doubt></subsection><subsection number="2.3" title="Nearest-neighbour search"><p>The simplest algorithm for finding synonyms is a ^-nearest-neighbour (fc-nn) search, which involves pair-wise vector comparison of the target term with every term in the vocabulary. Given an term vo­cabulary and up to attributes for each term, the asymptotic time complexity of nearest-neighbour search is .   This is very expensive, with even a moderate vocabulary making the use of huge datasets infeasible. It is for this reason that the sash data-structure is used to reduce the time complexity.</p></subsection></section><section number="3" title="The sash"><p>The sash approximates a fc-nn search by precom-puting some near neighbours for each node (terms in our case). This produces multiple paths between terms, allowing the sash to shape itself to the data set (Houle, 2003). The following description is adapted from Houle and Sakuma (2005).</p><p>The sash is a directed, edge-weighted graph with the following properties (see Figure 1):</p><p>• Each term corresponds to a unique node.</p><p>• The nodes are arranged into a hierarchy of lev­els, with the bottom level containing | nodes and the top containing a single root node. Each level, except the top, will contain half as many nodes as the level below. These are numbered from 1 (top) to .</p><p>• Edges between nodes are linked from consecu­tive levels. Each node will have at most /?<i>par­ent </i>nodes in the level above, and e <i>child </i>nodes in the level below.</p><p>• Every node must have at least one parent so that all nodes are reachable from the root.</p><p>Construction begins with the nodes being ran­domly distributed between the levels. The sash is then constructed iteratively by each node finding its closest parents in the level above. The parent will keep the closest of these children, forming edges in the graph, and reject the rest. Any nodes with­out parents after being rejected are then assigned as children of the nearest node in the previous level with fewer than children.</p><page local="3" global="51"/><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">5</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">K L</doubt><figure caption="Figure 1: Asash,wherep= 2,c= 3 andk = 2"></figure><p>Searching is performed by finding the £ nearest nodes at each level, which are added to a set of near nodes. To limit the search, only those nodes whose parents were found to be nearest at the pre­vious level are searched. The closest nodes from the set of near nodes are then returned. The search complexity is .</p><p>In Figure 1, the filled nodes demonstrate a search for the near-neighbours of some node g, using <i>k = </i>2. Our search begins with the root node . As we are using <i>k </i>= 2, we must find the two nearest children of A using our similarity measure. In this case, C and <i>D </i>are closer than <i>B. </i>We now find the closest two children of and . is not checked as it is only a child of 5. All other nodes are checked, including F and <i>G,</i><i> </i>which are shared as children by</p><p>and . From this level we chose and . We then consider the fourth and fifth levels similarly</p><p>At this point we now have the list of near nodes /l CJX üJ/, <i>I, J, K </i>and <i>L.</i><i> </i>From this we chose the two nodes nearest : and marked in black. These are returned as the near-neighbours of .</p><p>can be varied at each level to force a larger num­ber of elements to be tested at the base of the sash using, for instance, the equation:</p><doubt alpha="16.7" length="6" tooSmall="False" monospace="0.0">i*-(•1</doubt><doubt alpha="36.4" length="22" tooSmall="False" monospace="0.0">ki =max{k]0&amp;2",-pc\(3)</doubt><p>This changes our search complexity to:</p><doubt alpha="50.0" length="10" tooSmall="False" monospace="0.0">ku^k~"pc2,</doubt><doubt alpha="20.0" length="15" tooSmall="False" monospace="0.0">- + ^-log2« (4)</doubt><p>(Houle and Sakuma, 2005). We use this geometric function in our experiments.</p></section><section number="4" title="WordNet"><p>Within WordNet (Fellbaum, 1998), words are di­vided into four syntactic categories: noun, verb, ad­jective and adverb. Each of these categories has a different structure, representing their use. We are only concerned with nouns in these experiments and, when referring to WordNet, we only refer to this part of it.</p><p>The key building block of WordNet is the <i>synset: </i>a set of synonymous terms. Words in a synset may not be fully interchangeable, but are in at least some contexts. Because words are organised by concept, polysemous words will appear in several synsets.</p><p>Synsets are arranged in a hierarchy based on hy-ponymic relations. Those near the root are more general, and those near the leaves are more specific.</p><p>WordNet 2.1 consists of 117,097 unique terms in 81,426 synsets. Of these terms 15,776 are polyse-mous, yielding a total of 145,104 word-sense pairs. Our experimental corpus consists of 246,067 unique terms, of which 88,925 remain after a frequency cut­off of 5 is applied. 22,537 terms occur in both Word-Net and our corpus, yielding 32,057 senses.</p><p>A coarse-grained sense distinction is made by 25 lexicographer files (see Table 1). Each of these rep­resent distinct conceptual and lexical domains and were selected to cover all possible English nouns. These map to the top most synsets in the WordNet hierarchy, either uniquely or as hyponyms.</p><p>Table 1 also show the proportion of WordNet cov­ered by each domain (by type), and the proportion of the terms in both the BNC and WordNet in each domain (by token from the BNC). We represent our corpus statistics by token as this is indicative of how reliable the context information is for each domain. Where a term appears in several domains, its count is divided by the number of domains and spread evenly between them, following the Resnik (1995) uniform mass splitting strategy.</p><p>WordNet itself can be used to measure semantic similarity. Budanitsky and Hirst (2001) found the method proposed by Jiang and Conrath (1997) to be the most successful in malapropism detection. They used information content to measure the conditional probability of finding a child synset given a parent synset.</p><p>Leacock and Chodorow (1998) measure the log of the path distance between two synsets, scaled by the overall depth of the hierarchy This performed nearly as well as Jiang and Conrath's method.<page local="4" global="52"/></p><doubt alpha="64.6" length="48" tooSmall="False" monospace="0.0">Table 1: 25 lexicographer files (Fellbaum, 1998)</doubt></section><section number="5" title="Evaluation"><p>Our evaluation uses a combination of three elec­tronic thesauri: the Macquarie (Bernard, 1990), Ro-get's (Roget, 1911) and Moby (Ward, 1996) the­sauri. It is possible to use precision and recall mea­sures to evaluate the quality of the extracted the­saurus. To help overcome the problems of direct comparisons we use several measures of system per­formance: direct matches (Direct), inverse rank (InvR), and precision of the top n synonyms (P(n)), for       , 5 and 10.</p><p>InvR is the sum of the inverse rank of each matching synonym, e.g. matches at ranks 3, 5 and 28 give an inverse rank score of | + ^ + ^g. With at most 100 synonyms, the maximum InvR score is 5.187. P(n) is the percentage of matching synonyms in the top   extracted synonyms.</p><p>The same 300 single-word nouns were used for the evaluation as used by Curran (2004) for his large scale evaluation. These were chosen randomly from WordNet such that they covered a range over the following properties:</p><p><b>frequency </b>Penn Treebank and bnc frequencies <b>number of senses </b>WordNet and Macquarie senses <b>specificity </b>depth in the WordNet hierarchy <b>concreteness </b>distribution across WordNet subtrees</p><p>For each of these terms, the closest 100 terms and their similarity score were extracted.</p></section><section number="6" title="Experiments"><p>The contexts were extracted from the non-speech portion of the British National Corpus (Burnard, 1995). All experiments used the Jaccard measure function, the TTest weight function and a cut-off frequency of 5. The sash was constructed using the geometric equation for   described in section 3.</p><p>The values 1-4, 2-8, 4-16, 8-32 and 16-64 were chosen for number of parents <i>(p)</i><i> </i>and children (e) in the sash, giving are range of branching factors to test the balance between <i>sparseness </i>and <i>bushiness.</i></p><p>As in Gorman and Curran (2005), we use the brute force fc-nn search (Naive) as our base-line for all our experiments. We also reproduce the re­sults for the fully random distribution (Random), when ordered by frequency (Sort) and when <i>folded </i>about some number of relations (Fold AI).</p><p>Random is consistent with the original design of the sash. In accordance with Zipf's law (Zipf, 1949), the majority of the terms have low frequen­cies, and comparisons with these low frequency terms are unreliable (Curran and Moens, 2002), Sort forces high frequency terms towards the root, producing more accurate results by providing more reliable initial search paths.</p><p>Unfortunately, these more reliable search paths are also more expensive to calculate. To mitigate this, FoldAI chooses <i>more </i>accurate initial paths, rather than <i>most </i>accurate paths. For each term, if its number of relations is greater than some cho­sen number of relations , it is given a new rank­ing based on the score Otherwise its ranking based on its number of relations. This has the ef­fect of pushing very high and very low frequency terms away form the root. The folding points this was tested for were 500, 1000 and 1500.</p></section><section number="7" title="Integrating WordNet"><p>Integrating information from WordNet produces much more complicated sorting schemes. The most direct method of using WordNet would be to <i>use </i>the WordNet hierarchy as the top levels of the sash. Those terms present in our vocabulary and in Word­Net would be inserted into the sash in the same order and with the same linkages as given by Word-<page local="5" global="53"/></p><table class="main" frame="box" rules="all" border="0" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>act, activity</p></td><td class="cell"><p>7.0%</p></td><td class="cell"><p>11.4%</p></td><td class="cell"><p>natural object</p></td><td class="cell"><p>1.8%</p></td><td class="cell"><p>1.7%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>animal, fauna</p></td><td class="cell"><p>10.9%</p></td><td class="cell"><p>4.5%</p></td><td class="cell"><p>natural phenomenon</p></td><td class="cell"><p>0.8%</p></td><td class="cell"><p>0.8%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>artifact</p></td><td class="cell"><p>12.3%</p></td><td class="cell"><p>16.1%</p></td><td class="cell"><p>person, human being</p></td><td class="cell"><p>13.8%</p></td><td class="cell"><p>14.7%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>attribute</p></td><td class="cell"><p>3.4%</p></td><td class="cell"><p>6.5%</p></td><td class="cell"><p>plant, flora</p></td><td class="cell"><p>12.6%</p></td><td class="cell"><p>2.6%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>body</p></td><td class="cell"><p>2.8%</p></td><td class="cell"><p>2.5%</p></td><td class="cell"><p>possession</p></td><td class="cell"><p>1.1%</p></td><td class="cell"><p>1.0%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>cognition, knowledge</p></td><td class="cell"><p>3.3%</p></td><td class="cell"><p>4.9%</p></td><td class="cell"><p>process</p></td><td class="cell"><p>0.9%</p></td><td class="cell"><p>1.3%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>communication</p></td><td class="cell"><p>6.3%</p></td><td class="cell"><p>7.8%</p></td><td class="cell"><p>quantity, amount</p></td><td class="cell"><p>1.4%</p></td><td class="cell"><p>2.0%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>event, happening</p></td><td class="cell"><p>1.2%</p></td><td class="cell"><p>2.2%</p></td><td class="cell"><p>relation</p></td><td class="cell"><p>0.5%</p></td><td class="cell"><p>0.6%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>feeling, emotion</p></td><td class="cell"><p>0.6%</p></td><td class="cell"><p>1.3%</p></td><td class="cell"><p>shape</p></td><td class="cell"><p>0.4%</p></td><td class="cell"><p>0.6%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>food</p></td><td class="cell"><p>2.8%</p></td><td class="cell"><p>2.7%</p></td><td class="cell"><p>state</p></td><td class="cell"><p>4.4%</p></td><td class="cell"><p>5.2%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>group, grouping</p></td><td class="cell"><p>3.0%</p></td><td class="cell"><p>2.2%</p></td><td class="cell"><p>substance</p></td><td class="cell"><p>3.7%</p></td><td class="cell"><p>4.7%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>location</p></td><td class="cell"><p>3.8%</p></td><td class="cell"><p>1.1%</p></td><td class="cell"><p>time</p></td><td class="cell"><p>1.3%</p></td><td class="cell"><p>1.1%</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>motivation, motive</p></td><td class="cell"><p>0.1%</p></td><td class="cell"><p>0.1%</p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Net. Those terms in our vocabulary and not in Wordnet would then be inserted into levels below the already linked terms, and then normal SaSH building process would link them.</p><p>This method is very different to the original de­sign of the sash. Even when we order by frequency or number of relations, the ordering of semantic re­lations is still random because synonymy is not a function of frequency. The sash relies on this ran­domness to cluster the terms successfully.</p><p>despite the paths in WordNet being between se-mantically similar terms, the success ofthis method is doubtful. many terms at the top of the hierar­chy, where searches begin, will not produce reliable measurements. Some, such as thing, are too general to narrow a search. Others, such as psychological feature, will occur with such a low frequency as to make measurement unreliable.</p><p>The most specific terms at the bottom of the WordNet hierarchy will have those terms not in WordNet as children. These specific terms are likely to have a lower frequency than terms in the middle of the hierarchy. The low frequency WordNet terms will produce less accurate paths when they find their children during construction, resulting in unreliable searches for terms not in WordNet. The fixed struc­ture of the WordNet paths will also reduce the abil­ity to find new similarities within WordNet as the paths to these will not exist.</p><p>Rather than using the knowledge provided by the synsets and hyponymy relations directly, we use the knowledge that both WordNet and the sash arrange terms as a graph. From WordNet, use additional knowledge from the 25 lexicographer files covering distinct conceptual domains (Table 1).</p><p>an analysis of the terms occuring in both Word­Net and our corpus shows an uneven distribution. The act, artifact and person domains each represent 10-15% of these terms, while the motivation, re­lation and shape domains represent less than 1%. When randomly distributed, there will be many more high frequency domains represented at the top of the sash. The initial paths formed at the top of a sash determine the accruacy of searches. If</p><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">il</doubt><doubt alpha="56.9" length="58" tooSmall="False" monospace="0.0">naive fold1000 fold1000-wn1 fold1000-wn2 sort-wn1 sort-wn2</doubt><doubt alpha="50.0" length="2" tooSmall="False" monospace="0.0">a:</doubt><doubt alpha="0.0" length="7" tooSmall="False" monospace="0.0">1.2 1.1</doubt><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">1</doubt><doubt alpha="0.0" length="49" tooSmall="False" monospace="0.0">0   200 400 600 800 1000 1200 1400 1600 1800 2000</doubt><p>Avg Search Time (ms)</p><p>Figure 2: InvR against average search time the initial path is innaccurate, then there is little chance of finding correct near-neighbours. If a do­main is not represented at the top of the sash, any searches for terms in that domain will first have to pass through domains with which they have little similarity. These paths are likely to be inaccurate, reducing the accuracy for the whole search.</p><p>We want all conceptual domains represented evenly at the top levels of the sash, without overly affecting the distribution of the terms themselves. Both Sort and Fold At improve the performance, preserving the distribution of terms, but do not guar­antee the even distribution of the domains. We want to ensure this even distribution.</p><p>To combine information from WordNet and our existing sorting techniques, we split our vocabulary according to membership ofdomains. This provides us with 25 lists of terms that appear in WordNet, and single a list of those that do not.</p><p>Each of these lists are then sorted by one of the sorting schemes (Random, Sort or Fold At). The lists are then merged by taking the current top­most term from each list and inserting it into a single list that will be used to create the sash. For poly-semous terms appearing in several lists, the list with the highest sorting is used.</p><p>Those terms not appearing in WordNet are treated in two ways. The first (wn1) is to treat them as<page local="6" global="54"/></p><doubt alpha="0.0" length="7" tooSmall="False" monospace="0.0">1.8 1.7</doubt><doubt alpha="0.0" length="7" tooSmall="False" monospace="0.0">1.6 1.5</doubt><doubt alpha="0.0" length="5" tooSmall="False" monospace="0.0">&gt; 1.4</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">1.3</doubt><table caption="Table 2: Evaluation of random and fully sorted distributions" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>DlST</b></p></td><td class="cell"><p><i>c</i></p></td><td class="cell"><p>Direct</p></td><td class="cell"><p>P(l)</p></td><td class="cell"><p>P(5)</p></td><td class="cell"><p>P(10)</p></td><td class="cell"><p>InvR</p></td><td class="cell"><p>Time</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Naive</p></td><td class="cell"><p>5.29</p></td><td class="cell"><p>60%</p></td><td class="cell"><p>47%</p></td><td class="cell"><p>39%</p></td><td class="cell"><p>1.72</p></td><td class="cell"><p>12217ms</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Random</p></td><td class="cell"><p>8</p></td><td class="cell"><p>4.93</p></td><td class="cell"><p>61%</p></td><td class="cell"><p>47%</p></td><td class="cell"><p>39%</p></td><td class="cell"><p>1.71</p></td><td class="cell"><p>520ms</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Random</p></td><td class="cell"><p>16</p></td><td class="cell"><p>5.23</p></td><td class="cell"><p>60%</p></td><td class="cell"><p>48%</p></td><td class="cell"><p>39%</p></td><td class="cell"><p>1.73</p></td><td class="cell"><p>872ms</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Random</p></td><td class="cell"><p>32</p></td><td class="cell"><p>5.30</p></td><td class="cell"><p>60%</p></td><td class="cell"><p>47%</p></td><td class="cell"><p>39%</p></td><td class="cell"><p>1.74</p></td><td class="cell"><p>1899ms</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Sort</p></td><td class="cell"><p>8</p></td><td class="cell"><p>4.89</p></td><td class="cell"><p>62%</p></td><td class="cell"><p>47%</p></td><td class="cell"><p>39%</p></td><td class="cell"><p>1.71</p></td><td class="cell"><p>317ms</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Sort</p></td><td class="cell"><p>16</p></td><td class="cell"><p>5.30</p></td><td class="cell"><p>61%</p></td><td class="cell"><p>48%</p></td><td class="cell"><p>39%</p></td><td class="cell"><p>1.75</p></td><td class="cell"><p>677ms</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Sort</p></td><td class="cell"><p>32</p></td><td class="cell"><p>5.32</p></td><td class="cell"><p>60%</p></td><td class="cell"><p>48%</p></td><td class="cell"><p>39%</p></td><td class="cell"><p>1.74</p></td><td class="cell"><p>1709ms</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Table 3 : Evaluation of folded distributions</p><doubt alpha="9.5" length="21" tooSmall="False" monospace="0.0">18I-1-1-1-1-1-1-1-1-r</doubt><doubt alpha="26.3" length="19" tooSmall="False" monospace="0.0">(//!      / naive -</doubt><doubt alpha="19.4" length="36" tooSmall="False" monospace="0.0">1.6"    t ; ■      ;random ------- ~</doubt><doubt alpha="33.3" length="21" tooSmall="False" monospace="0.0">iii/ fold500 --------</doubt><doubt alpha="13.3" length="30" tooSmall="False" monospace="0.0">1.5 -    ; )';    / fold1000 -</doubt><doubt alpha="22.7" length="22" tooSmall="False" monospace="0.0">j://!fold1500 --------</doubt><doubt alpha="14.3" length="35" tooSmall="False" monospace="0.0">o;1.4 -   » /;'    / sort ------- -</doubt><footnote label="1">1.3 " ff  /  "</footnote><doubt alpha="0.0" length="9" tooSmall="False" monospace="0.0">1.2 "1/ -</doubt><doubt alpha="0.0" length="7" tooSmall="False" monospace="0.0">1.1 - -</doubt><doubt alpha="0.0" length="7" tooSmall="False" monospace="0.0">1 . : -</doubt><doubt alpha="4.3" length="23" tooSmall="False" monospace="0.0">0 9I_1_1_1_1_1_1_1_1_1_</doubt><doubt alpha="20.3" length="74" tooSmall="False" monospace="0.0">0    200  400  600  800 1000 1200 1400 1600 1800 2000 Avg Search Time (ms)</doubt><p>Figure 3: InvR against average search time a twenty-sixth lexical category and merge them as such. The second (wn2) is to place these terms after those that appear in WordNet. This is more in the spirit of the method, as those terms not in WordNet are not all of a single domain.</p><p>Although it would seem that only having one quarter of the terms in the sash arranged by domain would be too few to have an effect, this represents the top thirteen levels of our sash, from at total of fifteen. As noise is more significant in initial path formation, the effect of changing the distribution at the top of the sas h has more affect than changing it at the bottom.</p></section><section number="8" title="Results"><p>Figure 2 plots the trade-off between accuracy and efficiency after we have introduced WordNet infor­mation into the sash, using values of e between 4 and 64. The initial sharp increase in efficiency is for values of from 4 to 8. We see knee points be­tween 400 and 600ms for the WordNet distributions, and 300ms for Fold 1000, when e is between 8 and 16. After the InvR exceeds Naive, we have a long tail where InvR converges on Naive as the search time increases. What is most interesting in the <i>sharp </i>knee of Fold 1 000-wn 1. From performing worse than Fold1000, it increases sharply to an equiva­lent performance, then converges to a an equivalent InvR to Naive.</p><p>Figure 3 plots the trade-off between accuracy and efficiency for Random, Sort and Fold1000 us­ing InvR and search time, again using the values of between 4 and 64. This can be contrasted with Figure 2. Again we have an initial sharp increase in efficiency, and a long tail converging to Naive. All the sash distributions have their knee points at around 300-500ms, when e is between 8 and 16.</p><p>Table 2 presents the results for the original Naive, Random and Sort experiments. These have been run using an improved implementation of the sash from that used in Gorman and Curran (2005). Only the results for e = 8, 16 and 32 are shown, as these span the knee point. Sort consis­tently outperformed Random in efficiency and out­performed Random in accuracy for <i>c &gt; </i>16. Both sash solutions outperformed Naive in efficiency by more than 14 times when <i>c </i>= 16. At <i>c </i>= 16, Sort produced similar results for Direct and out­performed in InvR by 1%. Random produced a similar InvR and was outperformed in Direct by 1%.</p><p>Table 3 presents the results for the folded dis­tributions. At <i>c </i>= 16, these produced accuracies equivalent to Random, at twice the speed of Sort and 33 times the speed of Naive. Fold1500 was the slowest, although only by 30ms, which cannot be considered significant. Its accuracy was 98% of Direct and equivalent In vR of Naive. Fold500 has the highest InvR at 1.75, but the lowest Di­rect at 97% of Naive. Fold1000 provided the best balance with the accuracy of Fold1500 and the speed of Fold500.</p><p>Table 4 presents the results when WordNet in­formation is used. Random-wn1 similar accu­racy to, but is nearly time as fast as Random. Random-wn2 produces similar accuracy, but with only a minor increase in efficiency. Sort-wn1 produces a similar accuracy Sort, but is slower.<page local="7" global="55"/> Sort-wn2 is also slower and suffer a minor accu­racy penalty. Fold1000-wn1 produces a similar accuracy to Fold 1000 and a similar search time. Fold1000-wn2 produces a similar accuracy and is nearly twice as slow.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>DlST</b></p></td><td class="cell"><p><i>c</i></p></td><td class="cell"><p>Direct</p></td><td class="cell"><p>P(l)</p></td><td class="cell"><p>P(5)</p></td><td class="cell"><p>P(10)</p></td><td class="cell"><p>InvR</p></td><td class="cell"><p>Time</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Naive</p></td><td class="cell"><p>5.29</p></td><td class="cell"><p>60%</p></td><td class="cell"><p>47%</p></td><td class="cell"><p>39%</p></td><td class="cell"><p>1.72</p></td><td class="cell"><p>12217ms</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Fold500</p></td><td class="cell"><p>8</p></td><td class="cell"><p>4.24</p></td><td class="cell"><p>60%</p></td><td class="cell"><p>45%</p></td><td class="cell"><p>35%</p></td><td class="cell"><p>1.60</p></td><td class="cell"><p>185ms</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Fold500</p></td><td class="cell"><p>16</p></td><td class="cell"><p>5.15</p></td><td class="cell"><p>62%</p></td><td class="cell"><p>48%</p></td><td class="cell"><p>39%</p></td><td class="cell"><p>1.75</p></td><td class="cell"><p>336ms</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>FOLD500</p></td><td class="cell"><p>32</p></td><td class="cell"><p>5.30</p></td><td class="cell"><p>60%</p></td><td class="cell"><p>48%</p></td><td class="cell"><p>39%</p></td><td class="cell"><p>1.74</p></td><td class="cell"><p>961ms</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>FOLDlOOO</p></td><td class="cell"><p>8</p></td><td class="cell"><p>4.43</p></td><td class="cell"><p>60%</p></td><td class="cell"><p>46%</p></td><td class="cell"><p>37%</p></td><td class="cell"><p>1.64</p></td><td class="cell"><p>180ms</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>F</b><b>OLD</b><b>1000</b></p></td><td class="cell"><p><b>16</b></p></td><td class="cell"><p><b>5.21</b></p></td><td class="cell"><p><b>61%</b></p></td><td class="cell"><p><b>48%</b></p></td><td class="cell"><p><b>39%</b></p></td><td class="cell"><p><b>1.73</b></p></td><td class="cell"><p><b>331ms</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>FOLDlOOO</p></td><td class="cell"><p>32</p></td><td class="cell"><p>5.31</p></td><td class="cell"><p>60%</p></td><td class="cell"><p>48%</p></td><td class="cell"><p>39%</p></td><td class="cell"><p>1.74</p></td><td class="cell"><p>1015ms</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>FOLD1500</p></td><td class="cell"><p>8</p></td><td class="cell"><p>4.43</p></td><td class="cell"><p>59%</p></td><td class="cell"><p>45%</p></td><td class="cell"><p>37%</p></td><td class="cell"><p>1.62</p></td><td class="cell"><p>236ms</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Fold1500</p></td><td class="cell"><p>16</p></td><td class="cell"><p>5.21</p></td><td class="cell"><p>61%</p></td><td class="cell"><p>48%</p></td><td class="cell"><p>39%</p></td><td class="cell"><p>1.74</p></td><td class="cell"><p>366ms</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>FOLD1500</p></td><td class="cell"><p>32</p></td><td class="cell"><p>5.31</p></td><td class="cell"><p>60%</p></td><td class="cell"><p>48%</p></td><td class="cell"><p>39%</p></td><td class="cell"><p>1.74</p></td><td class="cell"><p>1157ms</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>The consistent pattern in the results is that once we order by frequency or relations, any improve­ments in accuracy are not significant. In addition, any improvements from using WordNet information are inconsistent.</p></section><section number="9" title="Analysis"><p>The results for using the sash without WordNet show that it provides a significant improvement over a naive search. It is less clear whether adding the WordNet information brings further improvement.</p><p>Fold 1 000-wn 1 produces a result that is similar to the best results for Fold 1000. Random-wn1 is much faster than Random without a loss in ac­curacy All other results using WordNet are worse.</p><p>What we see most here is that there is no obvious pattern to the effects of adding WordNet informa­tion to the sash. In most cases it simply degrades performance, but sometime it improves aspects of it. This occurs for both wn1 and wn2, using different base distributions. A deeper analysis is needed.</p><p>There was no general pattern where a distribution of the sash was more accurate for some term than others except for approximately 25 terms which scored consistently lower or higher when WordNet information was used. These words were compared for polysemy, lexical file membership, depth in the hierarchy, distance, corpus frequency and number of relations. None of these provided any pattern as to identifying either high or low scoring terms.</p><p>The analysis of the sash covered both the con­struction and the searches. The construction con­sidered the distribution of terms and the number of children of each term. Term distribution was mea­sured by calculating the proportion of terms shared between two distributions for a certain number of terms a the top the distribution. Random distributions share an average of 1% of the top 1000 terms with any other distribution. Between 57% and 69% of terms were shared between distributions with and without WordNet information. Although there was a pattern following the distribution, there was none that indicated its success.</p><p>Children were counted to determine if there was a change in the bushiness of the sash as the dis­tribution changed. This was considered both glob­ally and for each level of the sash. Again trends were only indicative of the distribution. This was extended to consider the average distance to and the number of relations of each child without yielding further information.</p><p>Searches were analysed by measuring the propor­tion of searching done at each level. This considered the number of terms compared, the number num­ber of relations compared and distance to the search term. This showed no trends.</p><p>Given that no trends were found indicating which broad structural and distributional changes had a positive influence, we are left to conclude that the problem lies in the way the sash clusters particular distributions.</p><p>When used as designed the sash is robust. Our initial distribution functions all produce stable re­sults for various values of e and <i>p. </i>WordNetinfor­mation can improve the performance of the Ran­dom distribution, but our Sort and FoldAI order­ing functions increase the stability of the data at the top of the sash, improving results without needing additional lexical information.</p></section><section number="10" title="Conclusion"><p>We have used lexical semantic information from WordNet to inform the internal structure of the Spa-cial Approximation Sample Hierarchy (sash). The sash has shown the current methods of improving performance to be stable enough that adding this in­formation does not provide any benefit.</p><p>That we had some positive results using Word­Net, albeit inconsistently, indicates that using lex­ical information may still provide some improvement in accuracy or efficiency.<page local="8" global="56"/> What this informa­tion is and how it should be combined are questions that are yet to be answered. Although dismissed in its simplest form, using WordNet more directly in the SASH presents one possible direction.</p><table caption="Table 4: Evaluation of WordNet distributions" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>DlST</b></p></td><td class="cell"><p>Direct</p></td><td class="cell"><p>P(l)</p></td><td class="cell"><p>P(5)</p></td><td class="cell"><p>P(10)</p></td><td class="cell"><p>InvR</p></td><td class="cell"><p>Time</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Naive</p></td><td class="cell"><p>5.29</p></td><td class="cell"><p>60%</p></td><td class="cell"><p>47%</p></td><td class="cell"><p>39%</p></td><td class="cell"><p>1.72</p></td><td class="cell"><p>12217ms</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>FOLDlOOO</b></p></td><td class="cell"><p><b>5.21</b></p></td><td class="cell"><p><b>61%</b></p></td><td class="cell"><p><b>48%</b></p></td><td class="cell"><p><b>39%</b></p></td><td class="cell"><p><b>1.73</b></p></td><td class="cell"><p><b>331ms</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Random-wnI</p></td><td class="cell"><p>5.24</p></td><td class="cell"><p>59%</p></td><td class="cell"><p>47%</p></td><td class="cell"><p>39%</p></td><td class="cell"><p>1.72</p></td><td class="cell"><p>488ms</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Random-wn2</p></td><td class="cell"><p>5.25</p></td><td class="cell"><p>59%</p></td><td class="cell"><p>48%</p></td><td class="cell"><p>39%</p></td><td class="cell"><p>1.73</p></td><td class="cell"><p>773ms</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>sort-WNl</p></td><td class="cell"><p>5.26</p></td><td class="cell"><p>59%</p></td><td class="cell"><p>48%</p></td><td class="cell"><p>39%</p></td><td class="cell"><p>1.73</p></td><td class="cell"><p>759ms</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Sort-wn2</p></td><td class="cell"><p>5.30</p></td><td class="cell"><p>59%</p></td><td class="cell"><p>48%</p></td><td class="cell"><p>39%</p></td><td class="cell"><p>1.74</p></td><td class="cell"><p>737ms</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Fold1000-wn1</p></td><td class="cell"><p>5.23</p></td><td class="cell"><p>59%</p></td><td class="cell"><p>48%</p></td><td class="cell"><p>39%</p></td><td class="cell"><p>1.73</p></td><td class="cell"><p>686ms</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Fold1000-wn2</p></td><td class="cell"><p>5.23</p></td><td class="cell"><p>59%</p></td><td class="cell"><p>48%</p></td><td class="cell"><p>39%</p></td><td class="cell"><p>1.73</p></td><td class="cell"><p>686ms</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>We intend to further investigate using lexical se­mantic information to improve performance, imple­ment other term ordering strategies, as well as fur­ther investigating the canonical vector heuristic pre­sented in Gorman and Curran (2005).</p><p>Having set out with the aim of applying lexical knowledge to approximate distributional similarity searches, we have found that the existing methods for improving the performance of the SASH are suf­ficiently robust that this is unnecessary.</p></section><section title="Acknowledgements"><p>We would like to thank the anonymous reviewers for their helpful feedback and corrections. This work has been supported by the Australian Research Council under Discovery Project DP0453131.</p></section><references><p>John R. L. Bernard, editor. 1990. <i>The Macquarie Ency­clopedic Thesaurus. </i>The Macquarie Library, Sydney, Australia.</p><p>Peter F. Brown, Vincent J. DellaPietra, Peter V deSouza, Jennifer C. Lai, and Robert L. Mercer. 1992. Class-based n-gram models of natural language. <i>Computa­tional Linguistics, </i>18(4):467-479, December.</p><p>Alexander Budanitsky and Graeme Hirst. 2001. Se­mantic distance in WordNet: An experimental, application-oriented evaluation of five measures. In <i>Workshop on WordNet and Other Lexical Resources, </i>Pittsburgh, PA, USA, 2-7 June.</p><p>Lou Burnard, editor. 1995. <i>Users Reference Guide British National Corpus Version 1.0. </i>Oxford Univer­sity Computing Services, Oxford, UK.</p><p>Stephen Clark and David Weir. 2001. Class-based prob­ability estimation using a semantic hierarchy. In <i>Pro­ceedings of the 2nd Meeting of the North American Chapter ofthe Association for Computational Lin­guistics, </i>pages 95-102, Pittsburgh, PA, USA, 2-7 June.</p><p>James Curran and Marc Moens. 2002. Scaling context space. In <i>Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, </i>pages 231-238, Philadelphia, PA, USA, 7-12 July.</p><p>James Curran. 2004. <i>From Distributional to Semantic Similarity. </i>Ph.D. thesis, University of Edinburgh, Ed­inburgh, UK.</p><p>Christiane Fellbaum, editor. 1998. <i>WordNet: an elec­tronic lexical database. </i>The MIT Press, Cambridge,</p><p>MA, USA.</p><p>James Gorman and James Curran. 2005. Approxi­mate searching for distributional similarity. In <i>ACL-SIGLEX2005 Workshop on Deep Lexical Acquisition,</i></p><p>Ann Arbor, MI, USA, 30 June.</p><p>Gregory Grefenstette. 1994. <i>Explorations in Automatic Thesaurus Discovery. </i>Kluwer Academic Publishers,</p><p>Boston, MA, USA.</p><p>Michael E. Houle and Jun Sakuma. 2005. Fast approxi­mate similarity search in extremely high-dimensional data sets. In <i>Proceedings of the 21st International Conference on Data Engineering, </i>pages 619-630, Tokyo, Japan, 5-8 April. Michael E. Houle. 2003. Navigating massive data sets via local clustering. In <i>Proceedings of the 9th ACM SIGKDD International Conference on Knowl­edge Discovery and Data Mining, </i>pages 547-552,</p><p>Washington, DC, USA, 24-27 August.</p><p>Jay J. Jiang and David W. Conrath. 1997. Semantic sim­ilarity based on corpus statistics and lexical taxonomy.</p><p>In <i>Proceedings of10th International Conference on Research in Computational Linguistics, </i>pages 19-33, Taipei, Taiwan, 22-24 August. Claudia Leacock and Martin Chodorow. 1998. Com­bining local context and WordNet similarity for word sense identification. In <i>Fellbaum, 1998, </i>pages 265­283.</p><p>Guido Minnen, John Carroll, and Darren Pearce. 2000. Robust applied morphological generation. In <i>Pro­ceedings ofthe 1st International Natural Language Generation Conference, </i>pages 201-208, Mitzpe Ra­mon, Israel, 12-16 June. Marius Pasca and Sanda Harabagiu. 2001. The infor­mative role of WordNet in open-domain question an­swering. In <i>Proceedings ofthe Workshopon WordNet and Other Lexical Resources: Applications, Exten­sions and Customizations, </i>pages 138-143, Pittsburgh,</p><p>PA, USA, 2-7 June.</p><p>Darren Pearce. 2001. Synonymy in collocation extrac­tion. In <i>Proceedings ofthe Workshop on WordNet and Other Lexical Resources: Applications, Exten­sions and Customizations, </i>pages 41-46, Pittsburgh,</p><p>Adwait Ratnaparkhi. 1996. A maximum entropy part-of-speech tagger. In <i>Proceedings ofthe Conference on Empirical Methods in Natural Language Processing, </i>pages 133-142, 17-18 May. Philip Resnik. 1995. Using information content to eval­uate semantic similarity. In <i>Proceedings of the 14th International Joint Conference on Artificial Intelli­gence, </i>pages 448-453, Montreal, Canada, 20-25 Au­gust.</p><p>Peter Roget.  1911.  <i>Thesaurus ofEnglish words and</i></p><p><i>phrases. </i>Longmans, Green and Co., London, UK. Grady Ward. 1996. <i>MobyThesaurus. </i>Moby Project. George K. Zipf. 1949. <i>Human Behaviour and the Prin­ciple ofLeast Effort. </i>Addison-Wesley, Cambridge,</p></references></body></article>