The Sprâkdata-ML System as Used for SENSEVAL-2
Dimitrios KOKKINAKIS
Sprâkdata, Göteborg University Box 200, SE-405 30 Göteborg, Sweden Dimitrios.Kokkinakis @svenska.gu.se
The methodology followed for sense disambiguation of the Swedish data by the Sprâkdata-ML system is supervised, based on Machine Learning (ML) techniques, particularly Memory Based Learning (MBL). The MBL implementation we used originates from the university of Tilburg in a system called TiMBL; details can be found in Daelemans et al (1999). Thus, our main contribution in this task has been the effort to try and isolate a set of features that could maximize the performance of the MBL software. However, it is rather difficult to give the exact number of features and examples required for an adequate description of a word's sense or which algorithm performs best. We think that there is space for improvement of our system's performance by better modeling of the available resources (e.g. context, annotations), choice of parameters and algorithms, a claim that we have not explored to its full potential, further exploration is required. Intelligent example selection for supervised learning is an important issue in ML, an issue that we have not fully explored. In previous experiments for a similar problem for Swedish, the algorithm that performed best in TiMBL was a variant of the k-nearest neighbor (Mitchell, 1997) called IB1, an algorithm that we also used in the exercise; (Kokkinakis & Johansson Kokkinakis, 1999).
To enhance the lexical disambiguation results using the available resources, we perform preprocessing in both the dictionary and the text to be sense-disambiguated. This is motivated by the fact that by making certain normalizations and simplifications in the resources we (hopefully) contribute to the production of qualitatively better results.
Initially, a text to be disambiguated is pre-processed by a tokeniser, a sentence boundary identifier, an idiom
The vectors consisted of: (i) selected information gathered from the dictionary entries (5 features); (ii) near-context (5 features); (iii) annotations applied on the training corpus (5 features); and (iv) information acquired from the lemmatised training corpus (85 features).
The corpus instances and dictionary were in XML format. An example of a corpus instance (1) for the first sense of the noun barn 'child' and a fragment of its dictionary description (2) are:
(1) <instance id="barn.114"><answer instance="barn. 114" senseid="barn_1 J " /> <context>... försöken sâ att spädbarnen själva künde styra de retningar som de utsattes för under försöket. Inom sprâkforskningen betyder det att <head>barnen</head> kan pâverka hur olika talljud présenteras. När de far ... </context> </instance>
(2) demma-entry id="barn_1" form= "barn" pos="n" inflection="~et = ><lexeme id= "barn_1_1"xdefinition> människa som ej vuxit färdigk/definition> <definition-ext>till kropp och själ; under ngn aldersgräns som beror pâ samman-hanget</definition-ext> <synt-example>kvinnor och ~ släpptes fria </synt-examplexsynt-exampie>~ under 6 âr kommer in gratis</synt-examplexcompound>spädbarn</compoun d>...<cycle id=" barn„1_1_a"xtrans>spec. om människa som ej nâtt pubertetsâlder, straff-myndighetsâlder etc.</transxsynt-example> ännu nâgot âr är hon ett ~</synt-examplexcompound> barnarbete </compoundxcompound>barnavärdsnämn d</compoundx/cyc/e>.. .</lexeme><lexe me>...<cycle id=" barn_1_2_a"> <trans> äv. utvidgat, spec, om foster </transxsynt-example>hon är med ~ </synt-examplexvalency>med ~ </valency> <lcycle>.. .<llexeme></lemma-entry>
The modeling of the vectors was performed in stages. The first stage of the processing uses the information from the dictionary. For every sense and sub-sense we extracted five representative nouns from the definition (and the definition extension) by applying part-of-speech tagging, lemmatization and exclusion of a number of generic nouns from a stop-list e.g. människa 'human' (a). If the number of nouns were less than five, we completed the list with compounds (if available).
Furthermore, the syntactic examples were used as training corpus and were added to the training instances (b). The valency information (if any) was also used in the same way (c). Consequently the amount of training material increased with 1,296 "new" disambiguated instances. A "dummy" XXX instance-number was given in these cases.
We did not put much effort on a more complex processing of the definitions since these are very short. The representations given below use the dictionary and corpus sample provided in (1) and (2).
(a) <definition>människa som ej vuxit färdigt</definitionxdefinition-ext>till
kropp och själ; under ngn aldersgräns som beror pâ sammanhanget </definition-ext>
become: bam_1_1: kropp, själ, aldersgräns
(b) <synt-exampIe>kvinnor och ~ släpptes fria</synt-examp!e> become: <instance ids'barn.XXX'^ <answer instance=,,barn.XXXH senseid= "barn_1_17> <context> kvinnor och <head>barn</head> släpptes fria </contextx/instance> become: instance id="barn.XXX"> <answer instance="barn.XXX" senseid= "barn_1_2_a,7xcontext> med <head>barn </head> </contextx/instance>
The second stage involved the use of the near-context. Punctuation, auxiliary verbs and a number of other stop-words were removed and the surrounding tokens (±2) of each headword in the corpus were extracted (d). Only the lemma form of the headwords was used, and the context was not lemmatized:
(d) instance id="barn.114"><answer instance=,,barn.114" senseid= "barnJJ" /xcontexb... sprâkforskningen betyder det att <head>barnefh</head> km pâverka im olika ...</context> </instance> became: instance id=,,barn.114"> <answer instance="barn.114n senseid=" barn_1_17xcontext>sprâkforskningen betyder <head>barn</head> pâverka olika <contextx/instance>
During the third stage, the training corpus was processed by a name-entity recognizer (e.g. human, time), an idiom identifier (idiom) and a semantic tagger (e.g. Bio, ethnos, phenomenon). The annotations produced by these tools were gathered in the form of a list of labels, and the five most frequent in the respective set of instances for each sense and sub-sense were used in the vectors. For example, for the sense barn_1_1 the five most frequent annotations found in all training instances were: Bio, Organization-Agency, Location, Situ and Occupation-Agent.
Often, near-context cannot distinguish between different senses. In such cases it is useful to look at a larger context and extract keywords representative for each sense. We made a frequency list of all noun and verb occurrences for all corpus instances for each sense. From the produced lists, 85 keywords per sense were extracted by eliminating high frequency (a word occurred in more than X percent of the cases with the sense) and low frequency words (a word occurred at least Z times in the list). For the sense barn_1_1 the 85 keywords included:
ansikte, ansvar, apparatur, arm, awikelse, barnmorska, barnomsorg, beredskap, betala, bild, detaij, dialog, djur, docka, erfarenhet, fel, forestalling, förslag,...
After the collection and combination of the 95 features common to a sense (stages i, iii, iv in Section 2, el), a complete case for a sense was produced (e2):
(el) Lemma_SENSE: 5 words from the dictionary information, 5 "semantic" labels, 85 representative words from the global context
(e2) barn_1_1 : kropp, själ, smâbarn, spädbarn, äldersgräns, Bio, Organization-Agency, Location, Situ, Occupation-Agent, ansikte, ansvar, apparatur, arm, awikelse, barnmorska, barnomsorg,...
We assume then, that for each training instance the above list is "true" and we convert the training instances into vectors of 102 features, where the 95 positions of the features in each vector were substituted with T keeping intact the near context. Thus, the truncated training instance in (f) was re-formatted to (g):
(f) instance id="barn.114"><answer instance^'barn-IM'' senseid= "barn J J
(g) sprâkforskningen, betyder,<head>barn </head>, pâverka, olika, 1,1,1,1,1,1, 1,..., barn.114, barn_l__l.
The test material consisted of 1,525 corpus instances in the same format as the previous training example, but without any designation of the correct senseid. The material was processed in a similar manner as the training one. The major difference lies in the fact that at the vector-creation stage we used the feature-vectors representative for a sense, example (e) previously, and we compared them with the features produced for each test instance. A feature at a specific position then was assigned T if the feature in the test occurred in the representative feature vector or '0' otherwise. For instance, the test instance in (h) was transformed, after processing, to a 102-feature-vector.
(h) instance id=,,barn.114,,xanswer instance="barn.114" senseid= "??????n/><context>l jungfrukammaren innanför köket bodde en kokerska och en husa. [ Ett hus fyllt av minnen ] Huset är fyllt av minnen. I fotoalbumen kan vi se farmor omgiven av sina smâ vitklädda <head>barn</head> och pappa i sjömanskostym lutad mot en björk. I faiiars svarta, snidade skrivbord ... </context> </instance>
The class of the representative sense-vector that produced more Ts for the test instance was chosen as the class of that instance. In (i) there are four Ts which means that the specific test instance had four common features with the representative vector for sense barn_1_2_a, and less than four for all the other representative vectors for the rest of the senses for barn. Thus, the class for the test instance is assigned that sense (which may be altered by the MBL software during the nearest-neighbor calculation).
The training and test feature vectors were then fed to the TiMBL software, where the IB1 algorithm (nearest neighbor search) was used.
Table 1 shows the evaluation of the test material. Since answers were provided for the whole material, precision and recall obtain the same value. Coarse-grain evaluation was not used, however coarse-grained is considered the least interesting of the three measures.
Conclusion
The existence of sense ambiguity (polysemy and homonymy) is one of the major problems affecting the usefulness of basic corpus exploration tools. In this respect, we regard sense disambiguation as a very important process and component when it is seen in the context of a wider and deeper text-processing architecture. In this paper we have described a simple feature-vector extraction approach to sense disambiguation that was utilized in a MBL software. We do not believe that we have fully exploited the capabilities of either the software or the way we can model the available resources. These issues will be investigated in the future, as well as the evaluation of the sense-tagger on an even larger scale.
Instances | Fine | Mixed | |||
Adjectives | 191 | 48,2% | 54,4% | ||
Nouns | 616 | 71,3% | 74,9% | ||
Verbs | 718 | 57,8% | 66,1% | ||
Most Freq. Baseline | 45,3% | ||||
Whole Sample | 1,525 | 62,0% | 68,2% | ||
Daelemans W., Zavrel J., van der Sloot K. and van den Bosch A. (1999). TiMBL: Tilburg Memory Based Learner, version 2.0, Reference Guide. ELK Technical Report 99-01, Paper available from: http://ilk.kub.nl/~ilk/papers/ilk9901.ps.gz.
Kokkinakis D. and Johansson Kokkinakis S. (1999). Sense Tagging at the Cycle-Level Using GLDB. Nordiska Studier i Lexikografi, vol. 27:146-167. Gellerstam M., Jöhannesson K., Ralph B. and Rogström L. (eds). Nordiska Föreningen for Lexikografi & Meijerbergs Institut for Svensk Etymologisk Forskning.
Kokkinakis D., Toporowska Gronostaj M. and Warmenius K. (2000). Annotating, Disambiguating & Automatically Extending the Coverage of the Swedish SIMPLE Lexicon. Proceedings of the 2nd Languages Resources and Evaluation Conference (LREC), vol. IÏÏ: 1397-1404. Athens, Hellas.
Mitchell T. M. (1997). Machine Learning. McGraw-Hill Series on Computer Science.