<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="315"/><title>Classifier Combination for Contextual Idiom Detection Without Labelled Data</title><author surname="Li" givenname="Linlin"><org  name="Saarland University" country="Germany" city="Saarbrucken"/></author><author surname="Sporleder" givenname="Caroline"><org  name="Lund University" country="Sweden"/></author></firstpageheader><frontmatter><p><b>Classifier Combination for Contextual Idiom Detection Without Labelled Data</b></p><p><b>Liniin Li </b>and <b>Caroline Sporleder</b></p><p>Saarland University Postfach 15 11 50 66041 Saarbrücken Germany</p><p>{liniin,csporled}@coli.uni-saarland.de</p></frontmatter><abstract>We propose a novel unsupervised approach for distinguishing literal and non-literal use of idiomatic expressions. Our model com­bines an unsupervised and a supervised classifier. The former bases its decision on the cohesive structure of the context and labels training data for the latter, which can then take a larger feature space into account. We show that a combination of both classi­fiers leads to significant improvements over using the unsupervised classifier alone. </abstract></header><body><section number="1" title="Introduction"><p>Idiomatic expressions are abundant in natural lan­guage. They also often behave idiosyncratically and are therefore a significant challenge for natural language processing systems. For example, idioms can violate selectional restrictions (as in <i>push one's luck), </i>disobey typical subcategorisation constraints (e.g., <i>in line </i>without a determiner before <i>line), </i>or change the default assignments of semantic roles to syntactic categories (e.g., in <i>break sth with X the </i>argument <i>X </i>would typically be an instrument but for the idiom <i>break the ice </i>it is more likely to fill a patient role, as in <i>break the ice with Russia).</i></p><p>In order to deal with such idiosyncracies and as­sign the correct analyses, NLP systems need to be able to recognise idiomatic expressions. Much pre­vious research on idioms has been concerned with <i>type-based classification, </i>i.e., dividing expressions into 'idiom' or 'not idiom' irrespective of their ac­tual use in a given context. However, while some expressions, such as <i>by and large, </i>always have an idiomatic meaning, several other expressions, such as <i>break the ice </i>or <i>spill the beans, </i>can be used liter­ally as well as idiomatically (see examples (1) and (2), respectively). Sometimes the literal usage can even dominate in a domain, as for <i>drop the ball,</i> which occurs fairly frequently in a literal sense in the sports section of news texts.</p><p>(1) Dad had to <u>break the ice</u> on the chicken troughs so that they could get water.</p><p>(2) Somehow I always end up spilling the beans all over the floor and looking foolish when the clerk comes to sweep them up.</p><p>Hence, whether a particular occurrence of a po­tentially ambiguous expression has literal or non-literal meaning has to be inferred from the context <i>{token-based idiom classification). </i>Recently, there has been increasing interest in this classification task and both supervised and unsupervised tech­niques have been proposed. The work we present here builds on previous research by Sporleder and Li (2009), who describe an unsupervised method that exploits the presence or absence of cohesive ties between the component words of a potential idiom and its context to distinguish between literal and non-literal use. If strong ties can be found the expression is classified as literal otherwise as non-literal. While this approach often works fairly well, it has the disadvantage that it focuses exclu­sively on lexical cohesion, other linguistic cues that might influence the classification decision are disregarded.</p><p>We show that it is possible to improve on Sporleder and Li's (2009) results by employing a two-level strategy, in which a cohesion-based unsupervised classifier is combined with a super­vised classifier. We use the unsupervised classifier to label a sub-set of the test data with high confi­dence. This sub-set is then passed on as training data to the supervised classifier, which then labels the remainder of the data set. Compared to a fully unsupervised approach, this two-stage method has the advantage that a larger feature set can be ex­ploited. This is beneficial for examples, in which the cohesive ties are relatively weak but which con­tain other linguistic cues for literal or non-literal use.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">315</doubt><p><i>Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, </i>pages 315-323, Singapore, 6-7 August 2009. ©2009 ACL and AFNLP<page local="2" global="316"/></p></section><section number="2" title="Related Work"><p>Most studies on idiom classification focus on type-based classification; few researchers have worked on token-based approaches (i.e., classification of an expression in a given context). Type-based meth­ods frequently exploit the fact that idioms have a number of properties which differentiate them from other expressions. For example, they often exhibit a degree of syntactic and lexical fixedness. Some idioms, for instance, do not allow internal modi­fiers ( <i>*shoot the long breeze) </i>or passivisation ( <i>*the bucket was kicked). </i>They also typically only al­low very limited lexical variation <i>(*kick the vessel, * strike the bucket).</i></p><p>Many approaches for identifying idioms focus on one of these two aspects. For instance, measures that compute the association strength between the elements of an expression have been employed to determine its degree of compositionality (Lin, 1999; Fazly and Stevenson, 2006) (see also Villav-icencio et al. (2007) for an overview and a com­parison of different measures). Other approaches use Latent Semantic Analysis (LSA) to determine the similarity between a potential idiom and its components (Baldwin et al., 2003). Low similar­ity is supposed to indicate low compositionality. Bannard (2007) looks at the syntactic fixedness of idiomatic expressions, i.e., how likely they are to take modifiers or be passivised, and compares this to what would be expected based on the ob­served behaviour of the component words. Fazly and Stevenson (2006) combine information about syntactic and lexical fixedness (i.e., estimated de­gree of compositionality) into one measure.</p><p>The few token-based approaches include a study by Katz and Giesbrecht (2006), who devise a super­vised method in which they compute the meaning vectors for the literal and non-literal usages of a given expression in the training data. An unseen test instance of the same expression is then labelled by performing a nearest neighbour classification.</p><p>Birke and Sarkar (2006) model literal vs. non-literal classification as a word sense disambiguation task and use a clustering algorithm which compares test instances to two automatically constructed seed sets (one with literal and one with non-literal ex­pressions), assigning the label of the closest set. While the seed sets are created without immediate human intervention they do rely on manually cre­ated resources such as databases of known idioms.</p><doubt alpha="48.9" length="47" tooSmall="False" monospace="0.0">Cook et al. (2007) and Fazly et al. (2009) pro-</doubt><p>pose an alternative method which crucially relies on the concept of <i>canonical form, </i>which is a fixed form (or a small set of those) corresponding to the syntactic pattern(s) in which the idiom normally occurs (Riehemann, 2001).<footnote anchor="1"/> The canonical form allows for inflectional variation of the head verb but not for other variations (such as nominal inflection, choice of determiner etc.). It has been observed that if an expression is used idiomatically, it typically occurs in its canonical form. For example, Riehe-mann (2001, p. 34) found that for decomposable idioms 75% of the occurrences are in canonical form, rising to 97% for non-decomposable idioms.<footnote anchor="2"/>Cook et al. exploit this behaviour and propose an unsupervised method which classifies an expres­sion as idiomatic if it occurs in canonical form and literal otherwise.</p><p>Finally, in earlier work, we proposed an unsu­pervised method which detects the presence or ab­sence of cohesive links between the component words of the idiom and the surrounding discourse (Sporleder and Li, 2009). If such links can be found the expression is classified as 'literal' otherwise as 'non-literal'. In this paper we show that the per­formance of such a classifier can be significantly improved by complementing it with a second-stage supervised classifier.</p></section><section number="3" title="First Stage: Unsupervised Classifier"><p>As our first-stage classifier, we use the unsuper­vised model proposed by Sporleder and Li (2009). This model exploits the fact that words in a co­herent discourse exhibit <i>lexical cohesion </i>(Halliday and Hasan, 1976), i.e. concepts referred to in sen­tences are typically related to other concepts men­tioned elsewhere in the discourse. Given a suitable measure of semantic relatedness, it is possible to compute the strength of such cohesive ties between pairs of words. While the component words of literally used expressions tend to exhibit lexical co­hesion with their context, the words of non-literally used expressions do not. For example, in (3) the ex­pression <i>play with fire </i>is used literally and the word <i>fire </i>is related to surrounding words like <i>grilling, dry-heat, cooking, </i>and <i>coals. </i>In (4), however <i>play with fire </i>is used non-literally and cohesive ties be-</p><footnote label="1">This is also the form in which an idiom is usually listed in a dictionary.</footnote><p>decomposable idioms are expressions such as <i>spill the beans </i>which have a composite meaning whose parts can be mapped to the words of the expression (e.g., <i>spills </i>'reveal', <i>beans^ </i>'secret').</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">316</doubt><page local="3" global="317"/><p>tween <i>play </i>or <i>fire </i>and the context are absent.</p><p>(3) <b>Grilling </b>outdoors is much more than just another <b>dry-heat cooking </b>method. It's the chance to play with <b>fire, </b>satisfying a primal urge to stir around in <b>coals .</b></p><p>(4) And PLO chairman Yasser Arafat has accused Israel of playing with Are by supporting HAMAS in its infancy.</p><p>To determine the strength of cohesive links, the unsupervised model builds a graph structure (called <i>cohesion graph) </i>in which all pairs of content words in the context are connected by an edge which is weighted by the pair's semantic relatedness. Then the <i>connectivity </i>of the graph is computed, defined as the average edge weight. If the connectivity increases when the component words of the idiom are removed, then there are no strong cohesive ties between the expression and the context and the example is labelled as 'non-literal', otherwise it is labelled as 'literal'.</p><p>To model semantic distance, we use the <i>Nor­malized Google Distance </i>(NGD, see Cilibrasi and Vitanyi (2007)), which computes relatedness on the basis of page counts returned by a search engine.<footnote anchor="3"/>It is defined as follows:</p><doubt alpha="52.5" length="40" tooSmall="False" monospace="0.0">NGD(xy)=maxil°9f(x)&gt;l°9f(y)}- log f(x,y)</doubt><doubt alpha="63.0" length="27" tooSmall="False" monospace="0.0">log M—min{log f(x),logf(y)}</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(5)</doubt><p>where <i>x </i>and <i>y </i>are the two words whose association strength is computed <i>(e.g., fire </i>and <i>coal), </i><i>f(x)</i><i> </i>is the page count returned by the search engine for <i>x </i>(and likewise for <i>f(y)</i><i> </i>and <i>y), </i><i>f(x,y)</i><i> </i>is the page count returned when querying for "x AND y", and <i>M </i>is the number of web pages indexed by the search engine. The basic idea is that the more often two terms occur together, relative to their overall occurrence, the more closely they are related.</p><p>We hypothesise that the unsupervised classifier will give us relatively good results for some exam­ples. For instance, in (3) there are several strong cues which suggest that <i>play with fire </i>is used liter­ally. However, because the unsupervised classifier only looks at lexical cohesion, it misses many other clues which could help distinguish literal and non-literal usages. For example, if <i>break the ice </i>is followed by the prepositions <i>between </i>or <i>over </i>as in example (6), it is more likely to be used idiomati­cally (at least in the news domain).</p><p>(6) "Gujral will meet Sharif on Monday and discuss bilateral relations," the Press Trust of India added.</p><footnote label="3">We employ Yahoo! rather than Google since we found that it returns more stable counts.</footnote><p>The minister said Sharif and Gujral would be able to <u>break the ice</u> over Kashmir.</p><p>Furthermore, idiomatic usages also exhibit co­hesion with their context but the cohesive ties are with the <i>non-literal </i>meaning of the expression. For example, in news texts, <i>break the ice </i>in its figu­rative meaning often co-occurs with <i>discuss, rela­tions, talks </i>or <i>diplomacy </i>(see (6)). At the moment we do not have any way to model these cohesive links, as we do not know the non-literal meaning of the idiom.<footnote anchor="4"/> However if we had labelled data we could train a supervised classifier to learn these and other contextual clues. The trained classifier might then be able to correctly classify examples which were misclassified by the unsupervised classifier, i.e., examples in which the cohesive ties are weak but where other clues exist which indicate how the expression is used.</p><p>For example, in (7) there is weak cohesive evi­dence for a literal use of <i>break the ice, </i>due to the semantic relatedness between <i>ice </i>and <i>water. </i>How­ever, there are stronger cues for non-literal usage, such as the preposition <i>between </i>and the presence of words like <i>diplomats </i>and <i>talks, </i>which are in­dicative of idiomatic usage. Examples like this are likely to be misclassified by the unsupervised model; a supervised classifier, on the other hand, has a better chance to pick up on such additional cues and predict the correct label.</p><p>(7) Next week the two diplomats will meet in an attempt to <u>break the ice</u> between the two nations. A crucial issue in the talks will be the long-running water dispute.</p></section><section number="4" title="Second Stage: Supervised Classifier"><p>For the supervised classifier, we used Support Vec­tor Machines as implemented by the LIBSVM package.<footnote anchor="5"/> We implemented four types of features, which encode both cohesive information and word co-occurrence more generally.<footnote anchor="6"/> <b>Salient Words (salW) </b>This feature aims to iden­tify words which are particularly <i>salient </i>for literal usage.<page local="4" global="318"/> We used a frequency-based definition of salience and computed the <i>literal saliency score </i>for each word in a five-paragraph context around the target expression:</p><footnote label="4">It might be possible to compute the Normalized Google Distance between the whole expression and the words in the context, assuming that whenever the whole expression occurs it is much more likely to be used figuratively than literally. For expressions in canonical form this is indeed often the case (Riehemann, 2001), however there are exceptions (see Section 6.1) for which such an approach would not work.</footnote><footnote label="5">Available from: http : / /www. csie . ntu . edu . tw/ ~c jlin/libsvm/ We used the default parameters.</footnote><footnote label="6">We also experimented with linguistically more informed features, such as the presence of named entities in the local context of the expression, and properties of the subject or co-ordinated verbs, but we found that these features did not lead to a better performance of the supervised classifier. This is probably partly due to data sparseness.</footnote><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">317</doubt><p>where <i>salut(w) </i>is the saliency score of the word <i>w </i>for the class <i>lit; fut(w) </i>is the token frequency of the word <i>w </i>for literally used expressions; <i>iut(w) </i>is the number of instances of the target expressions classified as <i>lit </i>which co-occur with word <i>w </i>(and mutatis mutandis <i>nonlit </i>for target expressions la­belled as non-literal).<footnote anchor="7"/></p><p>Words with a high <i>s aim </i>occur much more fre­quently with literal usages than with non-literal ones. Conversely, words with a low <i>salm </i>should be more indicative of the non-literal class. How­ever, we found that, in practice, the measure is better at picking out indicative words for the literal class; non-literal usages tend to co-occur with a wide range of words. For example, among the high­est scoring words for <i>break the ice </i>we find <i>thick, bucket, cold, water, reservoir </i>etc. While we do find words like <i>relations, diplomacy, discussions </i>among the lowest scoring terms (i.e., terms indicative of the non-literal class), we also find a lot of noise <i>(ask, month). </i>The effect is even more pronounced for other expressions (like <i>drop the ball) </i>which tend to be used idiomatically in a wider variety of situations <i>(drop the ball on a ban of chemical weapons, drop the ball on debt reduction </i>etc.).</p><p>We implement the saliency score in our model by encoding for the 300 highest scoring words whether the word is present in the context of a given exam­ple and how frequently it occurs.<footnote anchor="8"/> Note that this feature (as well as the next one) can be computed in a per-idiom or a generic fashion. In the former case, we would encode the top 300 words separately for each idiom in the training set, in the latter across all idioms (with the consequence that more frequent idioms in the training set contribute to more po­sitions in the feature vector). We found that, in practice, it does not make a big difference which variant is used. Moreover, in our bootstrapping scenario, we cannot ensure that we have sufficient examples of each idiom in the training set to train separate classifiers, so we opted for generic models throughout all experiments.</p><footnote label="7">Our definition of salm bears similarities with the well known tf.idf  score. We include both the term frequencies (fut)  and the instance frequencies (iut)  in the formula because we believe both are important. However, the instance fre­quency is more informative and less sensitive to noise because it indicates that expression classified as 'literal' consistently co-occurs with the word in question. Therefore we weight down the effect of the term frequency by taking its log.</footnote><footnote label="8">We also experimented with different feature dimensions besides 300 but did not find a big difference in performance.</footnote><p><b>Related Words (relW) </b>This feature set is a vari­ant of the previous one. Here we score the words not based on their saliency but we determine the semantic relatedness between the noun in the id­iomatic expression and each word in the global context, using the <i>Normalized Google Distance </i>mentioned in Section 3. Again we encode the 300 top-scoring words.</p><p>While the <i>related words </i>feature is less prone to overestimation of accidental co-occurrence than the saliency feature, it has the disadvantage of conflat­ing different word senses. For example, among the highest scoring words for <i>ice </i>are <i>cold, melt, snow, skate, hockey </i>but also <i>cream, vanilla, dessert.</i></p><p><b>Relatedness Score (relS) </b>The fourth feature set implements the <i>relatedness score </i>which encodes the scores for the 100 most highly weighted edges in the cohesion graph of an instance.<footnote anchor="9"/> If these scores are high, there are many cohesive ties with the surrounding discourse and the target expression is likely to be used literally.</p><p><b>Discourse Connectivity (connect.) </b>Finally, we implemented two features which look at the cohe­sion graph of an instance. We encode the connec­tivity of the graph (i) when the target expression is included and (ii) when it is excluded. The un­supervised classifier uses the difference between these two values to make its prediction. By encod­ing the absolute connectivity values as features we enable the supervised classifier to make use of this information as well.</p></section><section number="5" title="Combining the Classifiers"><p>As mentioned before, we use the unsupervised clas­sifier to label an initial training set for the super­vised one. To ensure that the training set does not contain too much noise, we only add those ex­amples about which the unsupervised classifier is most confident.<page local="5" global="319"/> We thus need to address two ques­tions: (i) how to define a <i>confidence function </i>for the unsupervised classifier, and (ii) how to set the <i>confidence threshold </i>governing what proportion of the data set is used for training the second classifier.</p><footnote label="9">We only used the 100 highest ranked edges because we are looking at a specific context here rather than the contexts of the literal or non-literal class overall. Since the contexts we use are only five paragraphs long, recording the 100 strongest edges seems sufficient.</footnote><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">318</doubt><p>The first question is relatively easy to answer: as the unsupervised classifier bases its decision on the difference in connectivity between including or excluding the component words of the idiom in the cohesion graph, an obvious choice for a confidence function is the difference in connectivity; i.e., the higher the difference, the higher the confidence of the classifier in the predicted label.</p><p>The confidence threshold could be selected on the basis of the unsupervised classifier's perfor­mance on a development set. Note that when choos­ing such a threshold there is usually a trade-off be­tween the size of the training set and the amount of noise in it: the lower the threshold, the larger and the noisier the training set. Ideally we would like a reasonably-sized training set which is also rela­tively noise-free, i.e., does not contain too many wrongly labelled examples. One way to achieve this is to start with a relatively small training set and then expand it gradually.</p><p>A potential problem for the supervised classifier is that our data set is relatively imbalanced, with the non-literal class being four times as frequent as the literal class. Supervised classifiers often have problems with imbalanced data and tend to be overly biased towards the majority class (see, e.g., Japkowicz and Stephen (2002)). To overcome this problem, we experimented with boosting the literal class with additional examples.<footnote anchor="10"/> We describe our methods for training set enlargement and boosting the literal class in the remainder of this section.</p><p><b>Iteratively Enlarging the Training Set </b>A typi­cal method for increasing the training set is to go through several iterations of enlargement and re­training.<footnote anchor="11"/> We adopt a conservative enlargement strategy: we only consider instances on whose la­bels both classifiers agree and we use the confi­dence function of the unsupervised classifier to determine which of these examples to add to the training set. The motivation for this is that we hy­pothesise that the supervised classifier will not have</p><p>'"Throughout this paper, we use the term 'boosting' in a non-technical sense.</p><footnote label="11">In our case re-training also involves re-computing the ranked lists of salient and related words. As the process goes on the classifier will be able to discover more and more useful cue words and encode them in the feature vector.</footnote><p>a very good performance initially, as it is trained on a very small data set. As a consequence its con­fidence function may also not be very accurate. On the other hand, we know from Sporleder and Li (2009) that the unsupervised classifier has a rea­sonably good performance. So while we give the supervised classifier a veto-right, we do not allow it to select new training data by itself or overturn classifications made by the unsupervised classifier.</p><p>A similar strategy was employed by Ng and Cardie (2003) in a self-training set-up. However, while they use an ensemble of supervised classi­fiers, which they re-train after each iteration, we can only re-train the second classifier; the first one, being unsupervised, will never change its predic­tion. Hence it does not make sense to go through a large number of iterations; the more iterations we go through, the closer the performance of the combined classifier will be to that of the unsuper­vised one because that classifier will label a larger and larger proportion of the data. However, going through one or two iterations allows us to slowly enlarge the training set and thereby gradually im­prove the performance of the supervised classifier.</p><p>In each iteration, we select 10% of the remain­ing examples to be added to the training set.<footnote anchor="12"/>We could simply add those 10% of the data about which the unsupervised classifier is most confident, but if the classifier was more confident about one class than about the other, we would risk obtain­ing a severely imbalanced training set. Hence, we decided to separate examples classified as 'literal' from those classified as 'non-literal' and add the top 10% from each set. Provided the automatic classification is reasonably accurate, this will en­sure that the distribution of classes in the training set is roughly similar to that in the overall data set at least at the early stages of the bootstrapping.</p><p><b>Boosting the Literal Class </b>As the process goes on, we are still likely to introduce more and more imbalance in the training set. This is due to the fact that the supervised classifier is likely to have some bias towards the majority class (and our ex­periments in Section 6.2 suggest that this is indeed the case). Hence, as the bootstrapping process goes on, potentially more and more examples will be labelled as 'non-literal' and if we always select the top 10% of these, our training set will gradually become more imbalanced.<page local="6" global="320"/> This is a well-known problem for bootstrapping approaches (Blum and Mitchell, 1998; Le et al., 2006). We could coun­teract this by selecting a higher proportion of ex­amples labelled as 'literal'. However given that the number of literal examples in our data set is relatively small, we would soon deplete our literal instance pool and moreover, because we would be forced to add less confidently labelled examples for the literal class, we are likely to introduce more noise in the training set.</p><footnote label="12">Since we do not have a separate development set, we chose the value of 10% intuitively as it seemed a reasonably good threshold.</footnote><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">319</doubt><p>A better option is to boost the literal class with external examples. To do this we exploit the fact that non-canonical forms of idioms are highly likely to be used literally. Given that our data set only contains canonical forms (see Section 6.1), we automatically extract non-canonical form variants and label them as 'literal'. To generate possible variants, we either (i) change the number of the noun (e.g., <i>rock the boat </i>becomes <i>rock the boats), </i>(ii) change the determiner (e.g., <i>rock a boat), </i>or (iii) replace the verb or noun by one of its synonyms, hypernyms, or siblings from WordNet (e.g., <i>rock the ship). </i>While this strategy does not give us addi­tional literal examples for all idioms, for example we were not able to find non-canonical form occur­rences of <i>sweep under the carpet </i>in the Gigaword corpus, for most idioms we were able to gener­ate additional examples. Note that this data set is potentially noisy as not all non-canonical form ex­amples are used literally. However, when checking a small sample manually, we found that only very small percentage (&lt;&lt; 1%) was mis-labelled.</p><p>To reduce the classifier bias when enlarging the training set, we add additional literal examples dur­ing each iteration to ensure that the class distri­bution does not deviate too much from the dis­tribution originally predicted by the unsupervised classifier.<footnote anchor="13"/> The examples to be added are selected randomly but we try to ensure that each idiom is represented. When reporting the results, we disre­gard these additional external examples.</p></section><section number="6" title="Experiments and Results"><p>We carried out a number of different experiments. In Section 6.2 we investigate the performance of the different features of the supervised classifier and in Section 6.3 we look more closely at the behaviour of the combined classifier. We start by describing the data set.</p><footnote label="13">We are assuming that the true distribution is not known and use the predictions of the unsupervised classifier to ap­proximate the true distribution.</footnote><doubt alpha="50.0" length="8" tooSmall="False" monospace="0.0">6.1 Data</doubt><p>We used the data from Sporleder and Li (2009), which consist of 17 idioms that can be used both literally and non-literally (see Table 1). For each expression, all canonical form occurrences were extracted from the Gigaword corpus together with five paragraphs of context and labelled as 'literal' or 'non-literal'.<footnote anchor="14"/> The inter-annotator agreement on a small sample of doubly annotated examples was 97% and the kappa score 0.7 (Cohen, 1960).</p><p>Table 1: Idiom statistics (* indicates expressions for which the literal usage is more common than the non-literal one)</p><subsection number="6.2" title="Feature Analysis for the Supervised Classifier"><p>In a first experiment, we tested the contribution of the different features (Table 2). For each set, we trained a separate classifier and tested it in 10-fold cross-validation mode. We also tested the perfor­mance of the first three features combined (salient and related words and relatedness score) as we wanted to know whether their combination leads to performance gains over the individual classifiers. Moreover, testing these three features in combi­nation allows us to assess the contribution of the connectivity feature, which is most closely related to the unsupervised classifier. We report the accu­racy, and because our data are fairly imbalanced, also the F-Score for the minority class ('literal').<page local="7" global="321"/></p><footnote label="14">The restriction to canonical forms was motivated by the fact that for the mostly non-decomposable idioms in the set, the vast majority (97%) of non-canonical form occurrences will be used literally (see Section 2).</footnote><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">320</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>expression</p></td><td class="cell"><p>literal</p></td><td class="cell"><p>non-literal</p></td><td class="cell"><p>all</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>back the wrong horse</p></td><td class="cell"><p>0</p></td><td class="cell"><p>25</p></td><td class="cell"><p>25</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>bite off more than one can chew</p></td><td class="cell"><p>2</p></td><td class="cell"><p>142</p></td><td class="cell"><p>144</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>bite one's tongue</p></td><td class="cell"><p>16</p></td><td class="cell"><p>150</p></td><td class="cell"><p>166</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>blow one's own trumpet</p></td><td class="cell"><p>0</p></td><td class="cell"><p>9</p></td><td class="cell"><p>9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>bounce off the wall*</p></td><td class="cell"><p>39</p></td><td class="cell"><p>7</p></td><td class="cell"><p>46</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>break the ice</p></td><td class="cell"><p>20</p></td><td class="cell"><p>521</p></td><td class="cell"><p>541</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>drop the ball*</p></td><td class="cell"><p>688</p></td><td class="cell"><p>215</p></td><td class="cell"><p>903</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>get one's feet wet</p></td><td class="cell"><p>17</p></td><td class="cell"><p>140</p></td><td class="cell"><p>157</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>pass the buck</p></td><td class="cell"><p>7</p></td><td class="cell"><p>255</p></td><td class="cell"><p>262</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>play with Are</p></td><td class="cell"><p>34</p></td><td class="cell"><p>532</p></td><td class="cell"><p>566</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>pull the trigger*</p></td><td class="cell"><p>11</p></td><td class="cell"><p>4</p></td><td class="cell"><p>15</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>rock the boat</p></td><td class="cell"><p>8</p></td><td class="cell"><p>470</p></td><td class="cell"><p>478</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>set in stone</p></td><td class="cell"><p>9</p></td><td class="cell"><p>272</p></td><td class="cell"><p>281</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>spill the beans</p></td><td class="cell"><p>3</p></td><td class="cell"><p>172</p></td><td class="cell"><p>175</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>sweep under the carpet</p></td><td class="cell"><p>0</p></td><td class="cell"><p>9</p></td><td class="cell"><p>9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>swim against the tide</p></td><td class="cell"><p>1</p></td><td class="cell"><p>125</p></td><td class="cell"><p>126</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>tear one's hair out</p></td><td class="cell"><p>7</p></td><td class="cell"><p>54</p></td><td class="cell"><p>61</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>all</p></td><td class="cell"><p>862</p></td><td class="cell"><p>3102</p></td><td class="cell"><p>3964</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>It can be seen that the <i>salient words (salW) </i>fea­ture has the highest performance of the individual features, both in terms of accuracy and in terms of literal F-Score, followed by <i>related words (relW), </i>and <i>relatedness score (relS). </i>Intuitively, it is plausi­ble that the saliency feature performs quite well as it can also pick up on linguistic indicators of idiom usage that do not have anything to do with lexical cohesion. However, a combination of the first three features leads to an even better performance, sug­gesting that the features do indeed model somewhat different aspects of the data.</p><p>The performance of the connectivity feature is also interesting: while it does not perform very well on its own, as it over-predicts the non-literal class, it noticeably increases the performance of the model when combined with the other features, suggesting that it picks up on complementary information.</p></subsection><subsection number="6.3" title="Testing the Combined Classifier"><p>We experimented with different variants of the combined classifier. The results are shown in Ta­ble 3. In particular, we looked at: (i) combining the two classifiers without training set enlargement or boosting of the literal class <i>{combined), </i>(ii) boost­ing the literal class with 200 automatically labelled non-canonical form examples <i>(combined+boost), </i>(iii) enlarging the training set by iteration <i>(com-bined+it), </i>and (iv) enlarging the training set by iteration and boosting the literal class after each iteration <i>(combined+boost+it). </i>The table shows the literal precision, recall and F-Score of the com­bined model (both classifiers) on the complete data set (excluding the extra literal examples). Note that the results for the set-ups involving iterative train­ing set enlargement are optimistic: since we do not have a separate development set, we report the op­timal performance achieved during the first seven iterations. In a real set-up, when the optimal num­ber of iterations is chosen on the basis of a separate data set, the results may be lower. The table also shows the majority class baseline <i>(Basemaj), </i>and the overall performance of the unsupervised model <i>(unsup) </i>and the supervised model when trained in 10-fold cross-validation mode <i>(super 10CV).</i></p><p>It can be seen that the combined classifier is 8% more accurate than both the majority baseline and the unsupervised classifier. This amounts to an error reduction of over 35% (the difference is sta­tistically significant, %<footnote anchor="2"/> test, <i>p « </i>0.01). While the F-Score of the unboosted combined classifier is comparable to that of the unsupervised one, boost­ing the literal class leads to a 7% increase, due to a significantly increased recall, with no signif­icant drop in accuracy. These results show that complementing the unsupervised classifier with a supervised one, can lead to tangible performance gains. Note that the accuracy of the combined clas­sifier, which uses no manually labelled training data, is only 4% below that of a fully supervised classifier; in other words, we do not lose much by starting with an automatically labelled data set. It­erative enlargement of the training set can lead to further improvements, especially when combined with boosting to reduce the classifier bias.</p><p>To get a better idea of the effect of training set enlargement, we plotted the accuracy and F-Score of the combined classifier for a given number of iterations with boosting (Figure 1) and without (Fig­ure 2). It can be seen that enlargement has a notice­able positive effect if combined with boosting. If the literal class is not boosted, the increasing bias of the classifier seems to outweigh most of the pos­itive effects from the enlarged training set. Figure 1 also shows that the best performance is obtained af­ter a relatively small number of iterations (namely two), as expected.<footnote anchor="15"/> With more iterations the per­formance decreases again. However, it decays relatively gracefully and even after seven iterations, when more than 40% of the data are classified by the unsupervised classifier, the combined classifier still achieves an overall performance that is sig­nificantly above that of the unsupervised classifier (84.<page local="8" global="322"/>28% accuracy compared to 78.38%, significant at <i>p « </i>0.01). Hence, the combined classifier seems not to be very sensitive to the exact number of iterations and performs reasonably well even if the number of iterations is sub-optimal.</p><footnote label="15">Note that this also depends on the confidence threshold. For example, if a threshold of 5% is chosen, more iterations may be required for optimal performance.</footnote><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">321</doubt><table caption="Table 2: Performance of different feature sets, 10-fold cross-validation" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Feature</p></td><td class="cell"><p><i>A\ </i>Prec.</p></td><td class="cell"><p><b><i>ig. </i></b>litera Ree.</p></td><td class="cell"><p>(%)</p><p>F-Score</p></td><td class="cell"><p>Avg. (%) Acc.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>salW</p></td><td class="cell"><p>77.10</p></td><td class="cell"><p>56.10</p></td><td class="cell"><p>65.00</p></td><td class="cell"><p>86.83</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>relW</p></td><td class="cell"><p>78.00</p></td><td class="cell"><p>43.20</p></td><td class="cell"><p>55.60</p></td><td class="cell"><p>84.99</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>relS</p></td><td class="cell"><p>74.90</p></td><td class="cell"><p>37.50</p></td><td class="cell"><p>50.00</p></td><td class="cell"><p>83.68</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>connectivity</p></td><td class="cell"><p>78.30</p></td><td class="cell"><p>2.10</p></td><td class="cell"><p>4.10</p></td><td class="cell"><p>78.58</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>salW+relW+relS</p></td><td class="cell"><p>82.90</p></td><td class="cell"><p>63.50</p></td><td class="cell"><p>71.90</p></td><td class="cell"><p>89.20</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>all</p></td><td class="cell"><p>85.80</p></td><td class="cell"><p>66.60</p></td><td class="cell"><p>75.00</p></td><td class="cell"><p>90.34</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 3: Results for different classifiers; * indicates best performance (optimistic)" class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Model</p></td><td class="cell"><p>Prec;</p></td><td class="cell"><p>Ree;</p></td><td class="cell"><p>F-Score;</p></td><td class="cell"><p>Acc.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Base<b>maj</b></p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>-</p></td><td class="cell"><p>78.25</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>unsup.</p></td><td class="cell"><p>50.04</p></td><td class="cell"><p>69.72</p></td><td class="cell"><p>58.26</p></td><td class="cell"><p>78.38</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>combined</p></td><td class="cell"><p>83.86</p></td><td class="cell"><p>45.82</p></td><td class="cell"><p>59.26</p></td><td class="cell"><p>86.30</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>combined+boost</p></td><td class="cell"><p>70.26</p></td><td class="cell"><p>62.76</p></td><td class="cell"><p>66.30</p></td><td class="cell"><p>86.13</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>combined+it*</p></td><td class="cell"><p>85.68</p></td><td class="cell"><p>46.52</p></td><td class="cell"><p>60.30</p></td><td class="cell"><p>86.68</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>combined+boost+it*</p></td><td class="cell"><p>71.86</p></td><td class="cell"><p>66.36</p></td><td class="cell"><p>69.00</p></td><td class="cell"><p>87.03</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>super. 10CV</p></td><td class="cell"><p>85.80</p></td><td class="cell"><p>66.60</p></td><td class="cell"><p>75.00</p></td><td class="cell"><p>90.34</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p><b>— Acc. combined </b>9 0 <b>Acc. unsupervised </b>—|- F<b>-Score combined </b><b>A A </b>F<b>-Score unsupervised</b></p><p>Figure 1 : Accuracy and literal F-Score on complete data set after different iterations with boosting of the literal class</p><doubt alpha="77.2" length="79" tooSmall="True" monospace="0.0">— Acc. combined • • Acc. unsupervised—h F-Score combinedA AF-Score unsupervised</doubt><p>Figure 2: Accuracy and literal F-Score on complete data set after different iterations without boosting of the literal class</p><p>Figure 3 shows how the training set increases as the process goes on<footnote anchor="16"/> and how the number of mis-classifications in the training set develops. In­terestingly, when going from the first to the second iteration the training set nearly doubles (from 396 to 669 instances), while the proportion of errors is also reduced by a third (from 7% to 5%). Hence, the training set does not only grow but the pro­portion of noise in it decreases, too. This shows that our conservative enlargement strategy is fairly successful in selecting correctly labelled examples. Only at later stages, when the classifier bias takes over, does the proportion of noise increase again.</p><doubt alpha="0.0" length="13" tooSmall="True" monospace="0.0">800 1000 1200</doubt><doubt alpha="85.7" length="21" tooSmall="True" monospace="0.0">Items in Training Set</doubt><p>Figure 3 : Training set size and error in training set at different iterations</p></subsection></section><section number="7" title="Conclusion"><p>We presented a two-stage classification approach for distinguishing literal and non-literal use of id­iomatic expressions. Our approach complements an unsupervised classifier, which exploits informa­tion about the cohesive structure of the discourse, with a supervised classifier. The latter can make use of a range of features and therefore base its classification decision on additional properties of the discourse, besides lexical cohesion. We showed that such a combined classifier can lead to a sig­nificant reduction of classification errors. Its per­formance can be improved further by iteratively increasing the training set in a bootstrapping loop and by adding additional examples of the literal class, which is typically the minority class. We found that such examples can be obtained automat­ically by extracting non-canonical variants of the target idioms from an unlabelled corpus.</p><p>Future work should look at improving the su­pervised classifier, which so far has an accuracy of 90%. While this is already pretty good, a more sophisticated model might lead to further improve­ments. For example, one could experiment with linguistically more informed features. While our initial studies in this direction were negative, care­ful feature engineering might lead to better results.</p></section><section title="Acknowledgements"><p>This work was funded by the Cluster of Excellence "Multi­modal Computing and Interaction".</p><p>Again, we disregard the extra literal examples here.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">322</doubt><page local="9" global="323"/></section><references><p>Timothy Baldwin, Colin Bannard, Takaaki Tanaka, and Dominic Widdows. 2003. An empirical model of multiword expression decomposability. In <i>Proceed­ings of the ACL 2003 Workshop on Multiword Ex­pressions: Analysis, Acquisition and Treatment.</i></p><p>Colin Bannard. 2007. A measure of syntactic flex-ibility for automatically identifying multiword ex­pressions in corpora. In <i>Proceedings of the ACL-07 Workshop on A Broader Perspective on Multiword Expressions.</i></p><p>Julia Birke and Anoop Sarkar. 2006. A clustering approach for the nearly unsupervised recognition of nonliteral fanguage. In <i>Proceedings ofEACL-06.</i></p><p>Avrim Blum and Tom Mitcheif. f998. Combining la-befed and unfabefed data with co-training. In <i>Pro­ceedings ofCOLT-98.</i></p><p>Rudi L. Cifibrasi and Paui M.B. Vitanyi. 2007. The Googie simiiarity distance. <i>IEEE Trans. Knowledge and Data Engineering, </i>f9(3):370-383.</p><p>Jacob Cohen. f960. A coefficient of agreement for nominai scafes. <i>Educational and Psychological Measurements, </i>20:37^f6.</p><p>Pauf Cook, Afsaneh Fazly, and Suzanne Stevenson. 2007. Pulling their weight: Exploiting syntactic forms for the automatic identification of idiomatic expressions in context. In <i>Proceedings of the ACL-07 Workshop on A Broader Perspective on Multi­word Expressions.</i></p><p>Afsaneh Fazly and Suzanne Stevenson. 2006. Auto­matically constructing a lexicon of verb phrase id­iomatic combinations. In <i>Proceedings ofEACL-06.</i></p><p>Afsaneh Fazly, Paul Cook, and Suzanne Stevenson. 2009. Unsupervised type and token identification of idiomatic expressions. <i>Computational Linguistics, </i>35(1):61-103.</p><p>M.A.K. Halliday and R. Hasan. 1976. <i>Cohesion in English. </i>Longman House, New York.</p><p>Nathalie Japkowicz and Shaju Stephen. 2002. The class imbalance problem: A systematic study. <i>In­telligent Data Analysis Journal, </i>6(5):429^f50.</p><p>Graham Katz and Eugenie Giesbrecht. 2006. Au­tomatic identification of non-compositionai mufti-word expressions using latent semantic analysis. In <i>Proceedings of the ACL/COLING-06 Workshop on Multiword Expressions: Identifying and Exploiting Underlying Properties.</i></p><p>Anh-Cuong Le, Akira Shimazu, and Le-Minh Nguyen. 2006. Investigating problems of semi-supervised learning for word sense disambiguation. In <i>Proc. ICCPOL-06.</i></p><p>Dekang Lin. 1999. Automatic identification of non-compositionai phrases. In <i>Proceedings of ACL-99, </i>pages 317-324.</p><p>Vincent Ng and Claire Cardie. 2003. Weakly super­vised natural language learning without redundant views. In <i>Proc. of HLT-NAACL-03.</i></p><p>Susanne Riehemann. 2001. <i>A Constructional Ap­proach to Idioms and Word Formation. </i>Ph.D. thesis, Stanford University.</p><p>Caroline Sporleder and Linlin Li. 2009. Unsupervised recognition of literal and non-literal use of idiomatic expressions. In <i>Proceedings ofEACL-09.</i></p><p>Aline Villavicencio, Valia Kordoni, Yi Zhang, Marco Idiart, and Carlos Ramisch. 2007. Validation and evaluation of automatically acquired multiword ex­pressions for grammar engineering. In <i>Proceedings ofEMNLP-07.</i></p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">323</doubt></references></body></article>