<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1"/><title>A Lemmatization Method for Modern Mongolian and its Application to Information Retrieval</title><author surname="Khaltar" givenname="Badam-Osor"><org  name="University of Tsukuba" country="Japan" city="Tsukuba"/></author><author surname="Fujii" givenname="Atsushi"><org  name="University of Tsukuba" country="Japan" city="Tsukuba"/></author></firstpageheader><frontmatter><p><b>A Lemmatization Method for Modern Mongolian and its Application to Information Retrieval</b></p><p><b>Badam-Osor Khaltar        Atsushi Fujii</b></p><p>Graduate School of Library, Information and Media Studies University of Tsukuba 1-2 Kasuga Tsukuba, 305-8550, Japan {khab23, fujii}@slis.tsukuba.ac.jp</p></frontmatter><abstract>In Modern Mongolian, a content word can be inflected when concatenated with suf­fixes. Identifying the original forms of content words is crucial for natural lan­guage processing and information retrieval. We propose a lemmatization method for Modern Mongolian and apply our method to indexing for information retrieval. We use technical abstracts to show the effec­tiveness of our method experimentally. </abstract></header><body><section number="1" title="Introduction"><p>The Mongolian language is divided into Tradition­al Mongolian, which uses the Mongolian alphabet, and Modern Mongolian, which uses the Cyrillic alphabet. In this paper, we focus solely on the lat­ter and use the word "Mongolian" to refer to Mod­ern Mongolian.</p><p>In Mongolian, which is an agglutinative lan­guage, each sentence is segmented on a phrase-by-phrase basis. A phrase consists of a content word, such as a noun or a verb, and one or more suffixes, such as postpositional participles. A content word can potentially be inflected when concatenated with suffixes.</p><p>Identifying the original forms of content words in Mongolian text is crucial for natural language processing and information retrieval. In informa­tion retrieval, the process of normalizing index terms is important, and can be divided into lemma-tization and stemming. Lemmatization identifies the original form of an inflected word, whereas stemming identifies a stem, which is not necessari­ly a word.</p><p>Existing search engines, such as Google and Yahoo!, do not perform lemmatization or stem­ming for indexing Web pages in Mongolian. Therefore, Web pages that include only inflected forms of a query cannot be retrieved.</p><p>In this paper, we propose a lemmatization me­thod for Mongolian and apply our method to in­dexing for information retrieval.</p></section><section number="2" title="Inflection types in Mongolian phrases"><p>Nouns, adjectives, numerals, and verbs can be concatenated with suffixes. Nouns and adjectives are usually concatenated with a sequence of a plural suffix, case suffix, and reflexive possessive suffix. Numerals are concatenated with either a case suffix or a reflexive possessive suffix. Verbs are concatenated with various suffixes, such as an aspect suffix, a participle suffix, and a mood suffix.</p><p>Figure 1 shows the inflection types of content words in Mongolian phrases. In (a), there is no in­flection in the content word <b>"hom </b>(book)", conca­tenated with the suffix <b>"hh </b>(the genitive case)". The content words are inflected in (b)-(e).</p><p>types of content words in<page local="2" global="2"/></p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">l</doubt><table caption="Figure 1: Inflection Mongolian phrases." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Type</p></td><td class="cell"><p>Example</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(a) No inflection</p></td><td class="cell"><p><b>HOM </b><b>+ </b><b>MH </b><b>—&gt;</b><b> HOMMH</b></p><p>book + genitive case</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(b) Vowel insertion</p></td><td class="cell"><p><b>ax </b><b>+ </b><b>g — axag</b></p><p>brother + dative case</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(c) Consonant insertion</p></td><td class="cell"><p><b>oaiiiniiii </b><b>+ </b><b>hhh— </b><b>SaiiiiiHHrHHH</b></p><p>building + genitive case</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(d) The letters "<b>b</b>" or "<b>h</b>" are eliminated, and the vowel converts to "<b>h</b>"</p></td><td class="cell"><p><b>aHra </b><b>+ </b><b>aac — aHraac</b></p><p>return + ablative case</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(e) Vowel elimination</p></td><td class="cell"><p><b>avh'ii.i </b><b>+ </b><b>aac — avk.iaac</b></p><p>work + ablative case</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Loanwords, which can be nouns, adjectives, or verbs in Mongolian, can also be concatenated with suffixes. In this paper, we define a loanword as a word imported from a Western language.</p><p>Because loanwords are linguistically different from conventional Mongolian words, the suffix concatenation is also different from that for con­ventional Mongolian words. Thus, exception rules are required for loanwords.</p><p>For example, if the loanword <b>"cTamj </b>(station)" is to be concatenated with a genitive case suffix, <b>"bm" </b>should be selected from the five genitive case suffixes (i.e., <b>hh</b><b>, </b><b>höh</b><b>, </b><b>h</b><b>, </b><b>hh</b><b>, </b>and <b>h</b><b>) </b>based on the Mongolian grammar. However, because <b>"cTaHij </b>(station)" is a loanword, the genitive case <b>"</b><b>höh</b><b>" </b>is selected instead of <b>"</b><b>hh</b><b>", </b>resulting in the noun phrase <b>"cTaHUHÖH </b>(station's)".</p><p>Additionally, the inflection (e) in Figure 1 never occurs for noun and adjective loanwords.</p></section><section number="3" title="Related work"><p>Sanduijav et al. (2005) proposed a lemmatization method for noun and verb phrases in Mongolian. They manually produced inflection rules and con­catenation rules for nouns and verbs. Then, they automatically produced a dictionary by aligning nouns or verbs with suffixes. Lemmatization for phrases is performed by consulting this dictionary.</p><p>Ehara et al. (2004) proposed a morphological analysis method for Mongolian, for which they manually produced rules for inflections and conca­tenations. However, because the lemmatization methods proposed by Sanduijav et al. (2005) and Ehara et al. (2004) rely on dictionaries, these me­thods cannot lemmatize new words that are not in dictionaries, such as loanwords and technical terms.</p><p>Khaltar et al. (2006) proposed a lemmatization method for Mongolian noun phrases that does not use a noun dictionary. Their method can be used for nouns, adjectives, and numerals, because the suffixes that are concatenated with these are almost the same and the inflection types are also the same. However, they were not aware of the applicability of their method to adjectives and numerals.</p><p>The method proposed by Khaltar et al. (2006) mistakenly extracts loanwords with endings that are different from conventional Mongolian words. For example, if the phrase <b>"</b><b>skojiofhhh </b>(ecology's)" is lemmatized, the resulting content word will be <b>"aKOJior", </b>which is incorrect. The correct word is <b>"aKOJioru </b>(ecology)". This error occurs because the ending <b>"-ojiorH </b>(-ology)" does not appear in conventional Mongolian words.</p><p>In addition, Khaltar et al. (2006)'s method applies (e) in Figure 1 to loanwords, whereas in­flection (e) never occurs in noun and adjective loanwords.</p><p>Lemmatization and stemming are arguably ef­fective for indexing in information retrieval (Hull, 1996; Porter, 1980). Stemmers have been devel­oped for a number of agglutinative languages, in­cluding Malay (Tai et al., 2000), Indonesian (Ber-lian Vega and Bressan, 2001), Finnish (Korenius et al., 2004), Arabic (Larkey et al., 2002), Swedish (Carlberger et al., 2001), Slovene (Popovic and Willett, 1992) and Turkish (Ekmekcioglu et al., 1996).</p><p>Xu and Croft (1998) and Melucci and Orio (2003) independently proposed a language-independent method for stemming, which analyzes a corpus in a target language and identifies an equivalent class consisting of an original form, inflected forms, and derivations. However, their method, which cannot identify the original form in each class, cannot be used for natural language applications where word occurrences must be stan­dardized by their original forms.</p><p>Finite State Transducers (FSTs) have been ap­plied to lemmatization. Although Karttunen and Beesley (2003) suggested the applicability of FSTs to various languages, no rule has actually been proposed for Mongolian. The rules proposed in this paper can potentially be used for FSTs.</p><p>To the best of our knowledge, no attempt has been made to apply lemmatization or stemming to information retrieval for Mongolian. Our research is the first serious effort to address this problem.</p></section><section number="4" title="Methodology 4.1 Overview"><p>In view of the discussion in Section 3, we en­hanced the lemmatization method proposed by Khaltar et al. (2006). The strength of this method is that noun dictionaries are not required.</p><p>Figure 2 shows the overview of our lemmatiza-tion method for Mongolian. Our method consists of two segments, which are identified with dashed lines in Figure 2: "lemmatization for verb phrases" and "lemmatization for noun phrases".</p><page local="3" global="3"/><p>Le^imatiza^on_forverb_ phrases</p><p>Input a phrase</p><p>Output the <b><i>\+j </i></b>content word</p><p>Detect a suffix in the phrase</p><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">W-H</doubt><p>Verb suffix dictionary</p><p>Remove suffixes and extract a content word</p><doubt alpha="100.0" length="3" tooSmall="False" monospace="0.0">Yes</doubt><p>Insert an eliminated vowel</p><p>Verb suffix segmentation rule</p><p>Vowel insertion rule</p><p>Verb dictionary</p><doubt alpha="0.0" length="1" tooSmall="False" monospace="0.0">3</doubt><p>Lemmatization for noun phrases</p><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">H-M</doubt><p>Noun suffix dictionary</p><p>Identify loanword</p><p>Loanword identification rule</p><p><b>U_J   </b>Noun suffix segmentation rule</p><figure caption="Figure 2: Overview of our lemmatization method for Mongolian."></figure><p>In Figure 2, we enhanced the method proposed by Khaltar et al. (2006) from three perspectives.</p><p>First, we introduced "lemmatization for verb phrases". There is a problem to be solved when we target both noun and verb phrases. There are a number of suffixes that can concatenate with both verbs and nouns, but the inflection type can be dif­ferent depending on the part of speech. As a result, verb phrases can incorrectly be lemmatized as noun phrases and vice versa.</p><p>Because new verbs are not created as frequently as nouns, we predefine a verb dictionary, but do not use a noun dictionary. We first lemmatize an entered phrase as a verb phrase and then check whether the extracted content word is defined in our verb dictionary. If the content word is not de­fined in our verb dictionary, we lemmatize the in­put phrase as a noun phrase.</p><p>Second, we introduced a "loanword identifica­tion rule" in "lemmatization for noun phrases". We identify a loanword phrase before applying a "noun suffix segmentation rule" and "vowel inser­tion rule". Because segmentation rules are different for conventional Mongolian words and loanwords, we enhance the noun suffix segmentation rule that was originally proposed by Khaltar et al. (2006). Additionally, we do not use the vowel insertion rule, if the entered phrase is detected as a loanword phrase. The reason is that vowel elimination never occurs in noun loanwords.</p><p>Third, unlike Khaltar et al. (2006), we targeted adjective and numeral phrases. Because the suffix­es concatenated with nouns, adjectives, and num­erals are almost the same, the lemmatization me­thod for noun phrases can also be used for adjec­tive and numeral phrases without any modifica­tions. We use "lemmatization for noun phrases" to refer to the lemmatization for noun, adjective, and numeral phrases.</p><p>We briefly explain our lemmatization process using Figure 2.</p><p>We consult a "verb suffix dictionary" and per­form backward partial matching to determine whether a suffix is concatenated at the end of a phrase. If a suffix is detected, we use a "verb suffix segmentation rule" to remove the suffix and extract the content word. This process will be repeated until the residue of the phrase does not match any of the entries in the verb suffix dictionary.</p><p>We use a "vowel insertion rule" to check wheth­er vowel elimination occurred in the content word and insert the eliminated vowel.</p><p>If the content word is defined in a "verb dictio­nary", we output the content word as a verb and terminate the lemmatization process. If not, we use the entered phrase and perform lemmatization for noun phrases. We consult a "noun suffix dictio­nary" to determine whether one or more suffixes are concatenated at the end of the target phrase.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>+ -</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Insert an eliminated vowel</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Check if the content word is a verb</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>No</p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Process the inputted phrase as a noun phrase</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="4" global="4"/><p>We use a "loanword identification rule" to iden­tify whether the phrase is a loanword phrase. We use a "noun suffix segmentation rule" to remove the suffixes and extract the content word. If the phrase is identified as a loanword phrase we use different segmentation rules.</p><p>We use the "vowel insertion rule" which is also used for verb phrases to check whether vowel eli­mination occurred in the content word and insert the eliminated vowel. However, if the phrase is identified as a loanword phrase, we do not use the vowel insertion rule.</p><p>If the target phrase does not match any of the entries in the noun suffix dictionary, we determine that a suffix is not concatenated and we output the phrase as it is.</p><p>The inflection types (b)-(d) in Figure 1 are processed by the verb suffix segmentation rule and noun suffix segmentation rule. The inflection (e) in Figure 1 is processed by the vowel insertion rule.</p><p>We elaborate on the dictionaries and rules in Sections 4.2-4.8.</p><subsection number="4.2" title="Verb suffix dictionary"><p>We produced a verb suffix dictionary, which con­sists of 126 suffixes that can concatenate with verbs. These suffixes include aspect suffixes, parti­ciple suffixes, and mood suffixes.</p><p>Figure 3 shows a fragment of our verb suffix dictionary, in which inflected forms of suffixes are shown in parentheses. All suffixes corresponding to the same suffix type represent the same meaning.</p></subsection><subsection number="4.3" title="Verb suffix segmentation rule"><p>For the verb suffix segmentation rule, we produced 179 rules. There are one or more segmentation rules for each of the 126 verb suffixes mentioned in Section 4.2.</p><p>Figure 4 shows a fragment of the verb suffix segmentation rule for suffix <b>"</b><b>b </b>(past)". In the column "Segmentation rule", the condition of each "if sentence is a phrase ending. "V" refers to a vowel and "*" refers to any strings. "C9" refers to any of the nine consonants <b>"</b><b>u</b><b>", "</b><b>jk</b><b>", </b><b>"3", "c", "g", </b><b>"</b><b>t</b><b>", </b><b>"m", </b><b>"</b><b>h</b><b>", </b>or <b>"x", </b>and "C7" refers to any of the seven consonants <b>"</b><b>m</b><b>", </b><b>"r", </b><b>"</b><b>h</b><b>", "</b><b>ji</b><b>", </b><b>"6", </b><b>"</b><b>b</b><b>", </b>or <b>"p". </b>If a condition is satisfied, we remove one or more corresponding characters.</p><p>For example, because the verb phrase <b>"</b><b>iuhh3hji3b </b>(renew + past)" satisfies condition (ii),</p><p><u>Suffix type</u>__<u>Suffix</u>_</p><p>Appeal <b>rTyH, </b><b>r</b><b>ryH</b></p><p>Complete <b>hhx</b></p><p>Perfect <b>aag (nag), oog (nog), 33g, eeg </b><u>Progressive-perfect      </u><b><u>caap, coop, C33p, ceep</u>_</b></p><figure caption="Figure 3: Fragment of verb suffix dictionary."></figure><p>we remove the suffix <b>"</b><b>b</b><b>" </b>and the preceding vowel <b>"3" </b>to extract <b>"</b><b>iuhh3hji</b><b>".</b></p></subsection><subsection number="4.4" title="Verb dictionary"><p>We use the verb dictionary produced by Sanduijav et al. (2005), which includes 1254 verbs.</p></subsection><subsection number="4.5" title="Noun suffix dictionary"><p>We use the noun suffix dictionary produced by Khaltar et al. (2006), which contains 35 suffixes that can be concatenated with nouns. These suffix­es are postpositional particles. Figure 5 shows a fragment of the dictionary, in which inflected forms of suffixes are shown in parentheses.</p></subsection><subsection number="4.6" title="Noun suffix segmentation rule"><p>There are 196 noun suffix segmentation rules, of which 173 were proposed by Khaltar et al. (2006). As we explained in Section 3, these 173 rules often incorrectly lemmatize loanwords with different endings from conventional Mongolian words.</p><p>We analyzed the list of English suffixes and found that English suffixes "-ation" and "-ology" are incorrectly lemmatized by Khaltar et al. (2006). In Mongolian, "-ation" is transliterated into <b>"auu" </b>or <b>"</b><b>huh</b><b>" </b>and "-ology" is transliterated into <b>"oj</b><b>i</b><b>oru". </b>Thus, we produced 23 rules for loanwords that end with <b>"auu", </b><b>"</b><b>huh</b><b>", </b>or <b>"oj</b><b>i</b><b>oru".</b></p><p>Figure 6 shows a fragment of our suffix segmen­tation rule for loanwords. For example, for the loanword phrase <b>'bKoj</b><b>i</b><b>oruiiH </b>(ecology + geni­tive)", we use the segmentation rule for suffix <b>"</b><b>hhh </b>(genitive)" in Figure 6. We remove the suffix <b>"</b><b>hhh </b>(genitive)" and add <b>"</b><b>h</b><b>" </b>to the end of the content word. As a result, the noun <b>'^Koj</b><b>i</b><b>oru </b>(ecology)" is correctly extracted.</p><table caption="Figure 4: Fragment of verb suffix segmentation rule." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Suffix</p></td><td class="cell"><p>Segmentation rule</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>b</b></p><p>Past</p></td><td class="cell"><p>(i) If ( *+ V + V + <b>b </b>) Remove <b>b</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(ii) If ( * + C9 + C7 + V + <b>b </b>) Remove V + <b>b</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="5" global="5"/></subsection><subsection number="4.7" title="Vowel insertion rule"><p>To insert an eliminated vowel and extract the orig­inal form of a content word, we check the last two characters of the content word. If they are both consonants, we determine that a vowel was elimi­nated. However, a number of Mongolian words end with two consonants inherently and, therefore, Khaltar et al. (2006) referred to a textbook on the Mongolian grammar (Ts, 2002) to produce 12 rules to determine when to insert a vowel between two consecutive consonants. We also use these rules as our vowel insertion rule.</p></subsection><subsection number="4.8" title="Loanword identification rule"><p>Khaltar et al. (2006) proposed rules for extracting loanwords from Mongolian corpora. Words that satisfy one of seven conditions are extracted as loanwords. Of the seven conditions, we do not use the condition that extracts a word ending with "consonants + <b>h</b><b>" </b>as a loanword because it was not effective for lemmatization purposes in prelimi­nary study.</p></subsection></section><section number="5" title="Experiments"><subsection number="5.1" title="Evaluation method"><p>We collected 1102 technical abstracts from the "Mongolian IT Park" <footnote anchor="1"/> and used them for experi­ments. There were 178,448 phrase tokens and 17,709 phrase types in the 1102 technical abstracts. We evaluated the accuracy of our lemmatization method (Section 5.2) and the effectiveness of our method in information retrieval (Section 5.3) expe­rimentally.</p><footnote label="1"> http://www.itpark.mn/ (October, 2007)</footnote></subsection><subsection number="5.2" title="Evaluating lemmatization"><p>Two Mongolian graduate students served as asses­sors. Neither of the assessors was an author of this paper. The assessors provided the correct answers for lemmatization. The assessors also tagged each word with its part of speech.</p><p>The two assessors performed the same task in­dependently. Differences can occur between two assessors on this task. We measured the agreement of the two assessors by the Kappa coefficient, which ranges from 0 to 1. The Kappa coefficients for performing lemmatization and tagging of parts of speech were 0.96 and 0.94, respectively, which represents almost perfect agreement (Landis and Koch, 1977). However, to enhance the objectivity of the evaluation, we used only the phrases for which the two assessors agreed with respect to the part of speech and lemmatization.</p><p>We were able to use the noun and verb dictiona­ries of Sanduijav et al. (2005). Therefore, we com­pared our lemmatization method with Sanduijav et al. (2005) and Khaltar et al. (2006) in terms of ac­curacy.</p><p>Accuracy is the ratio of the number of phrases correctly lemmatized by the method under evalua­tion to the total number of target phrases. Here, the target phrases are noun, verb, adjective, and num­eral phrases.</p><p>Table 1 shows the results of lemmatization. We targeted 15,478 phrase types in the technical ab­stracts. Our experiment is the largest evaluation for Mongolian lemmatization in the literature. In con­trast, Sanduijav et al. (2005) and Khaltar et al. (2006) used only 680 and 1167 phrase types, re­spectively, for evaluation purposes.</p><p>In Table 1, the accuracy of our method for nouns, which were targeted in all three methods, was higher than those of Sanduijav et al. (2005) and Khaltar et al. (2006). Because our method and that of Sanduijav et al. (2005) used the same verb dictionary, the accuracy for verbs is principally the same for both methods. The accuracy for verbs was low, because a number of verbs were not in­cluded in the verb dictionary and were mistakenly lemmatized as noun phrases. However, this prob­lem will be solved by enhancing the verb dictio­nary in the future. In total, the accuracy of our me­thod was higher than those of Sanduijav et al. (2005) and Khaltar et al. (2006).</p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">s</doubt><table caption="Figure 5: Fragment of noun suffix dictionary." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Case</p></td><td class="cell"><p>Suffix</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Genitive Accusative Dative Ablative</p></td><td class="cell"><p><b>h, </b><b>bl, b</b><b>i</b><b>H, HH, HHH</b></p><p><b>bir, HHr, r</b></p><p><b>a, </b><b>t</b></p><p><b>aac (uac), ooc (</b><b>hoc</b><b>), </b><b>33c</b><b>, eec</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Figure 6: Fragment of suffix segmentation rules for loanwords." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Suffix</p></td><td class="cell"><p>Segmentation rule for loanwords</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>HHH</b></p><p>Genitive</p></td><td class="cell"><p>If (* + <b>Jioraim)</b></p><p>Remove (<b>hhh</b>) , Add (<b>h</b>)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>HHr</b></p><p>Accusative</p></td><td class="cell"><p>If (* + <b>noraftr)</b></p><p>Remove <b>(HHr), </b>Add (<b>h</b>)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="6" global="6"/><p>We analyzed the errors caused by our method in Figure 7. In the column "Example", the left side and the right side of an arrow denote an error and the correct answer, respectively.</p><p>The error (a) occurred to nouns, adjectives, and numerals, in which the ending of a content word was mistakenly recognized as a suffix and was re­moved. The error (b) occurred because we did not consider irregular nouns. The error (c) occurred to loanword nouns because the loanword identifica­tion rule was not sufficient. The error (d) occurred because we relied on a verb dictionary. The error (e) occurred because a number of nouns were in­correctly lemmatized as verbs.</p><p>For the errors (a)-(c), we have not found solu­tions. The error (d) can be solved by enhancing the verb dictionary in the future. If we are able to use part of speech information, we can solve the error (e). There are a number of automatic methods for tagging parts of speech (Brill, 1997), which have promise for alleviating the error (e).</p><p><b>5.3    Evaluating the effectiveness of lemmatiza-tion in information retrieval</b></p><p>We evaluated the effectiveness of lemmatization methods in indexing for information retrieval. No test collection for Mongolian information retrieval is available to the public. We used the 1102 tech­nical abstracts to produce our test collection.</p><p>Figure 8 shows an example technical abstract, in which the title is "Advanced Albumin Fusion Technology" in English. Each technical abstract contains one or more keywords. In Figure 8, key­words, such as <b>"uycm</b><b>&gt;i </b><b>HHj</b><b>iflac </b>(blood serum)" and <b>"axac </b>(placenta)" are annotated.</p><p>We used two different types of queries for our evaluation. First, we used each keyword as a query, which we call "keyword query (KQ)". Second, we used each keyword list as a query, which we call "list query (LQ)". The average number for key­words in the keywords list was 6.1. For each query, we used as the relevant documents the abstracts that were annotated with the query keyword in the keywords field. Thus, we were able to avoid the cost of relevance judgments.</p><p>The target documents are the 1102 technical ab­stracts, from which we extracted content words in the title, abstract, and result fields as index terms. However, we did not use the keywords field for indexing purposes. We used Okapi BM25 (Robert­son et al., 1995) as the retrieval model.</p><p>We used the lemmatization methods in Table 2 to extract content words and compared the Mean Average Precision (MAP) of each method using KQ and LQ. MAP has commonly been used to evaluate the effectiveness of information retrieval. Because there were many queries for which the average precision was zero in all methods, we dis­carded those queries. There were 686 remaining KQs and 273 remaining LQs.</p><p>The average number of relevant documents for each query was 2.1. Although this number is small, the number of queries is large. Therefore, our eval­uation result can be stable, as in evaluations for question answering (Voorhees and Tice, 2000).</p><p>We can derive the following points from Table 2. First, to clarify the effectiveness of the lemmatiza-tion in information retrieval, we compare "no lemmatization" with the other methods. Any lem-matization method improved the MAP for both KQ and LQ. Thus, lemmatization was effective for information retrieval in Mongolian. Second, we compare the MAP of our method with those of Sanduijav et al. (2005) and Khaltar et al. (2006). Our method was more effective than the method of Sanduijav et al. (2005) for both KQ and LQ. How­ever, the difference between Khaltar et al. (2006) and our method was small for KQ and our method<page local="7" global="7"/></p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>#Phrase types</p></td><td class="cell"><p>Sanduijav et al. (2005)</p></td><td class="cell"><p>Khaltar</p><p>et al.</p><p>(2006)</p></td><td class="cell"><p>Our method</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Noun</p></td><td class="cell"><p>13,016</p></td><td class="cell"><p>57.6</p></td><td class="cell"><p>87.7</p></td><td class="cell"><p>92.5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Verb</p></td><td class="cell"><p>1,797</p></td><td class="cell"><p>24.5</p></td><td class="cell"><p>23.8</p></td><td class="cell"><p>24.5</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Adjective</p></td><td class="cell"><p>609</p></td><td class="cell"><p>82.6</p></td><td class="cell"><p>83.5</p></td><td class="cell"><p>83.9</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Numeral</p></td><td class="cell"><p>56</p></td><td class="cell"><p>41.1</p></td><td class="cell"><p>80.4</p></td><td class="cell"><p>81.2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Total</p></td><td class="cell"><p>15,478</p></td><td class="cell"><p>63.2</p></td><td class="cell"><p>72.3</p></td><td class="cell"><p>78.2</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Figure 7: Errors of our lemmatization method." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Reasons of errors</p></td><td class="cell"><p>#Errors</p></td><td class="cell"><p>Example</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(a) Word ending is the same as a suffix.</p></td><td class="cell"><p>274</p></td><td class="cell"><p><b>COpT — COp</b></p><p>sort</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(b) Noun plural tense is irregular.</p></td><td class="cell"><p>244</p></td><td class="cell"><p><b>aMbTaH — aMbT</b></p><p>animal</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(c) Noun loanword ends with two con­sonants.</p></td><td class="cell"><p>94</p></td><td class="cell"><p><b>gHHO3aep — gHHO3aeap</b></p><p>dinosaur</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(d) Verb does not exist in our verb dictionary.</p></td><td class="cell"><p>689</p></td><td class="cell"><p><b>KOgjiO — KOgjiOX</b></p><p>to code</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(e) Word corres­ponds to multiple part of speech.</p></td><td class="cell"><p>853</p></td><td class="cell"><p><b>OpOH — Op</b></p><p>country inter</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><p>Title: <b>AjibôyMHH yiiiiBspi</b><b>k</b><b>X g3Bii</b><b>i</b><b>HiiT3T TexHOioru</b></p><p>Author's name: <b>flopa</b>c <b>flaHgufi</b></p><p>Keywords: <b>i</b><b>r</b><b>ycHb</b><b>i </b><b>HHigsc, 3X3c </b><b>.</b><b>..</b></p><p>Abstract: <b>CyganraaHb</b><b>i </b><b>axnuH TaniaH </b><b>5, 10% </b><b>hhh...</b></p><p>Result: <b>AibSyMHHW yycMaii yiiirgBspi</b><b>k</b><b>X, coHroH </b><b>.</b><b>..</b></p><table caption="Table 3: t-test result of the differences between lemmatization methods."></table><figure caption="Figure 8: Example of technical abstract. Table 2: MAP of lemmatization methods."></figure><p>was less effective than Khaltar et al.(2006) for LQ. This is because although we enhanced the lemma-tization for verbs, adjectives, numerals, and loan­words, the effects were overshadowed by a large number of queries comprising conventional Mon­golian nouns. Finally, our method did not outper­form the method using the correct lemmatization.</p><p>We used the paired t-test for statistical testing, which investigates whether the difference in per­formance is meaningful or simply because of chance (Keen, 1992). Table 3 shows the results, in which "&lt;" and "&lt;&lt;" indicate that the difference of two results was significant at the 5% and 1% levels, respectively, and "—" indicates that the difference of two results was not significant.</p><p>Looking at Table 3, the differences between no lemmatization and any lemmatization method, such as Sanduijav et al. (2005), Khaltar et al. (2006), our method, and correct lemmatization, were statistically significant in MAP for KQ. However, because the MAP value of no lemmati-zation was improved for LQ, the differences be­tween no lemmatization and the lemmatization me­thods were less significant than those for KQ. The difference between Sanduijav et al. (2005) and our method was statistically significant in MAP for both KQ and LQ. However, the difference between Khaltar et al. (2006) and our method was not sig­nificant in MAP for both KQ and LQ. Although, the difference between our method and correct lemmatization was statistically significant in MAP for KQ, the difference was not significant in MAP for LQ.</p></subsection></section><section number="6" title="Conclusion"><p>In Modern Mongolian, a content word can poten­tially be inflected when concatenated with suffixes. Identifying the original forms of content words is crucial for natural language processing and infor­mation retrieval.</p><p>In this paper, we proposed a lemmatization me­thod for Modern Mongolian. We enhanced the lemmatization method proposed by Khaltar et al. (2006). We targeted nouns, verbs, adjectives, and numerals. We also improved the lemmatization for loanwords.</p><p>We evaluated our lemmatization method expe­rimentally. The accuracy of our method was higher than those of existing methods. We also applied our lemmatization method to information retrieval and improved the retrieval accuracy.</p><p>Future work includes using a part of speech tag­ger because the part of speech information is effec­tive for lemmatization.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Keyword query</p></td><td class="cell"><p>List query</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>No lemmatization vs. Correct lemmatization</p></td><td class="cell"><p>&lt;&lt;</p></td><td class="cell"><p>&lt;</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>No lemmatization vs. Sanduijav et al. (2005)</p></td><td class="cell"><p>&lt;&lt;</p></td><td class="cell"><p>—</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>No lemmatization vs. Khaltar et al. (2006)</p></td><td class="cell"><p>&lt;&lt;</p></td><td class="cell"><p>&lt;</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>No lemmatization vs. Our method</p></td><td class="cell"><p>&lt;&lt;</p></td><td class="cell"><p>&lt;</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Sanduijav et al. (2005) vs. Our method</p></td><td class="cell"><p>&lt;&lt;</p></td><td class="cell"><p>&lt;</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Khaltar et al. (2006) vs. Our method</p></td><td class="cell"><p>—</p></td><td class="cell"><p>—</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Our method vs. Correct lemmatization</p></td><td class="cell"><p>&lt;</p></td><td class="cell"><p>—</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p>Keyword query</p></td><td class="cell"><p>List query</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>No lemmatization</p></td><td class="cell"><p>0.2312</p></td><td class="cell"><p>0.2766</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Sanduijav et al. (2005)</p></td><td class="cell"><p>0.2882</p></td><td class="cell"><p>0.2834</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Khaltar et al. (2006)</p></td><td class="cell"><p>0.3134</p></td><td class="cell"><p>0.3127</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Our method</p></td><td class="cell"><p>0.3149</p></td><td class="cell"><p>0.3114</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Correct lemmatization</p></td><td class="cell"><p>0.3268</p></td><td class="cell"><p>0.3187</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><references><p>Vinsensius Berlian Vega S N and Stéphane Bressan. 2001. Indexing the Indonesian Web: Language iden­tification and miscellaneous issues. <i>Tenth Interna­tional World Wide Web Conference, Hong Kong.</i></p><p>Eric Brill. 1997. Natural Language Processing Using Very Large Corpora. Kluwer Academic Press.</p><p>Johan Carlberger, Hercules Dalianis, Martin Hassel, and Ola Knutsson. 2001. Improving Precision in Informa­tion Retrieval for Swedish using Stemming. <i>Proceed­ings of NODALIDA '01 - 13th Nordic Conference on Computational Linguistics.</i></p><page local="8" global="8"/><p>Terumasa Ehara, Suzushi Hayata, and Nobuyuki Kimu-ra. 2004. Mongolian morphological analysis using ChaSen. <i>Proceedings of the lOth Annual Meeting of the Association for Natural Language Processing, </i>pp. 709-712. (In Japanese).</p><p>Çuna F. Ekmekçioglu, Michael F. Lynch, and Peter Willett. 1996. Stemming and n-gram matching for term conflation in Turkish texts. <i>Information Re­search News, </i>Vol. 7, No. 1, pp. 2-6.</p><p>David A. Hull. 1996. Stemming algorithms - a case study for detailed evaluation. <i>Journal of the Ameri­can Society for Information Science and Technology, </i>Vol. 47, No. 1, pp. 70-84.</p><p>Lauri Karttunen and Kenneth R. Beesley. 2003. Finite State Morphology. <i>CSLI Publications. </i>Stanford.</p><p>Micheal E. Keen. 1992. Presenting results of experi­mental retrieval comparisons. <i>Information Processing and Management, </i>Vol. 28, No. 4, pp.</p><p>491-502.</p><p>Badam-Osor Khaltar, Atsushi Fujii, and Tetsuya Ishi-kawa. 2006. Extracting loanwords from Mongolian corpora and producing a Japanese-Mongolian bilin­gual dictionary. <i>Proceedings of the 2lst International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, </i>pp. 657-664.</p><p>Tuomo Korenius, Jorma Laurikkala, Kalervo Järvelin, and Martti Juhola. 2004. Stemming and Lemmatization in the Clustering of Finnish Text Documents. <i>Proceedings of the thirteenth Associa­tion for Computing Machinery international confe­rence on Information and knowledge management.</i></p><p>pp. 625-633.</p><p>Richard J. Landis and Gary G. Koch. 1977. The mea­surement of observer agreement for categorical data. <i>Biometrics, </i>Vol. 33, No. 1, pp. 159-174.</p><p>Leah S. Larkey, Lisa Ballesteros, and Margaret E. Con-nel. 2002. Improving Stemming for Arabic Informa­tion Retrieval: Light Stemming and Co-occurrence Analysis. <i>Proceedings of the 25th annual interna­tional ACM SIGIR conference on Research and de­velopment in information retrieval, </i>pp. 275-282.</p><p>Massimo Melucci and Nicola Orio. 2003. A Novel Me­thod for Stemmer Generation Based on Hidden Mar­kov Models. <i>Proceedings of the twelfth international conference on Information and knowledge manage­ment, </i>pp. 131-138.</p><p>Mirko Popovic and Peter Willett. 1992. The effective­ness of stemming for natural-language access to Slo­vene textual data. <i>Journal of the American Society for Information Science and Technology, </i>Vol. 43, No.</p><p>5, pp. 384-390.</p><p>Martin F. Porter. 1980. An algorithm for suffix strip­ping. <i>Program, </i>Vol. 14, No. 3, pp. 130-137.</p><p>Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. 1995. Okapi at TREC-3. <i>Proceedings of the Third Text REtrieval Conference, NIST Special Publication</i> 500-226. pp. 109-126.</p><p>Enkhbayar Sanduijav, Takehito Utsuro, and Satoshi Sato. 2005. Mongolian phrase generation and mor­phological analysis based on phonological and mor­phological constraints. <i>Journal of Natural Language Processing, </i>Vol. 12, No. 5, pp. 185-205. (In Japa­nese) .</p><p>Sock Y. Tai, Cheng O. Ong, and Noor A. Abdullah.</p><p>2000. On designing an automated Malaysian stem­mer for the Malay language. <i>Proceedings of the fifth international workshop on information retrieval with Asian languages, Hong Kong, </i>pp. 207-208.</p><p>Bayarmaa Ts. 2002. Mongolian grammar for grades I-IV. (In Mongolian).</p><p>Ellen M. Voorhees and Dawn M. Tice. 2000. Building a Question Answering Test Collection. <i>Proceedings of the 23rd Annual International ACM SIGIR Confe­rence on Research and Development in Information</i> <i>Retrieval, </i>pp. 200-207.</p><p>Jinxi Xu and Bruce W. Croft. 1998. Corpus-based stemming using co-occurrence of word variants.</p><p><i>ACM Transactions on Information Systems, </i>Vol. 16, No. 1, pp. 61-81.</p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">B</doubt></references></body></article>