A graphy is the Chinese minimal autonomous or­thographic unit and it approximatively matches the glyph concept in computer science. The fol­lowing glyphs are different Chinese graphies: I, fm, %, S5, S. A morph (noted | m |) is the small­est meaningful unit representable by a sequence of graphies. Morphs are atomic so that they cannot be representable by a smaller sequence of morphs. The following sequences of graphies are different the graphy S does not carry any meaning and is not a morph. A morpheme (noted ||M|) is a set of morphs sharing the same lexical content ignoring grammatical inflection or variants (Table 1). Chi­nese morphs cannot be inflected, unlike European languages, but some graphies have variants.

Morpheme Morph

r* |.. r* |..

Table 1: Morphemes and related morphs

A word-form (noted ( w )) is an autonomous and insepa able sequence of mo phs. Autonomy means that it can be enunciated individually and can take place in a syntactic pa adigm. Insepa-ability means that b eaking the sequence causes the loss of the elationship between elements. A lexeme (noted ( W ))) is a set of word-forms shar­ing the same lexical content ignoring inflection or variants (Table 2).

Word-form

Table 2: Lexemes and associated word-forms

A phrase (noted [ s ]) is a syntactic combina­tion of word-forms. The syntactic nature of the combination implies that the phrase components are relatively free. A locution (noted [ S ]) is a set of lexicalized phrases sharing the same lexical content ignoring inflection or variants (Table 3).

Locution Phrase

[ tfte] [(*)(&)][(*T)(te)] ...

The lexical units we look for are lexemes and lo­cutions. Finding lexical units means identifying words-forms and phrases having a lexical content. We use two criteria to define the lexical content: the compositionality criterion and the referential-ity criterion (Table 4). Units which fulfill at least one of these criteria are said to have a lexical con­tent.

The compositionality criterion (or lexicaliza-tion criterion) is relative to the relationship be­tween the sense of the unit and the sense of its components. The question is whether or not the sense of the unit can be deduced from the combi­nation of its components. The referentiality crite­rion is related to the relationship between the unit and the referent concept or object. The question is whether or not the referent has specific properties for the speakers. This criterion is strongly depen­dent on human judgment and the working domain.

Morph detection

Table 4: Referential and Compositional units

The Table 4 presents examples of four criterion combinations. Referentiality and compositional­ity criteria are always applied at the highest association level, thus [^Pè-^s] ]) is compositional, although {^ft) and {^^i) are not compositional. Word-forms are not necessarily compositional or referential, thus the unit IX-fc) does not refer to any concept and we can use the combination of its components to interpret it: (( -tft )) + || -fc ||. Ref­erentiality does not imply lexicalization, thus the compositional unit is referential because it refers to the German car brands or char­acteristics in the automobile context.

Segment detection Lexical unit candidate selection Max level reached ?

Candidates reorganization 4- [ Presentation & user interaction J-* -

Figure 1 : Method overview 3 Methodology

The method (Figure 1) follows the linguistic in­tuitions developed in the previous section. We identify morpho-syntactic units and select those that are likely to have a lexical content to obtain lexical unit candidates (LUCs). The word-forms and phrases are respectively generated by associ­ations of morphs or word-forms and association of word-forms or phrases. We consequently use an incremental process, which associates LUCs as they are selected. The incremental process is initiated by detecting every morph and spliting the corpus into segments. Then we enumerate all the morpho-syntactic unit couples and use lexical content criteria to select the couples to associate. This process is repeated until the maximum num­ber of associations is reached. At the end, the LUCs are reorganized and submitted to the user. The user's answers are used to filter the remaining LUCs.

As stated in Section 2.1, we consider that the morph is the minimal morpho-syntactic unit. Ev­ery glyph is considered as a morph unless it can be included in an ancient loanword morph

(( äft ))) or a foreign transcription morph

((;È*.#J)> {ê:ÏLK)). In an ambiguous case the longest possibility is accepted. Foreign transcrip­tions are phonetic interpretations of foreign words using the pronunciation of the Chinese graphies. The set of graphies used for transcription is well-

known and closed. We trained a CRF taggerusing simple features based on current, next and previous graphies to extract foreign transcriptions (the training corpus is described in Section 4.1). Ancient loanwords importation process is not pro­ductive anymore, thus they are detected using a loanword list.

The aim of the segment detection step is to split the corpus into segments (i.e. a succession of Chinese graphies). Chinese texts contain two kinds of delimiters which are not likely to be components of a lexical unit, delimiter-words and delimiter-expressions. Delimiter-words are enu­merable with a common word dictionary and include prepositions (*t, ft), adverbs (#., -k, pronouns fßJL), interrogative pronouns C$1 conjunctions (i^Ü-, 'fa,

discourse structure words (Sit, tonal particles (fî, p&) and tool-words (^). Delimiter-expressions include numerical expres­sions (^^^^, -2-^), temporal expressions (4^H&-i-, ^vAAl;^), circumstantial expres­sions (AL..fHê, ^...^) , which are easily de-scribable using shallow context-free grammars. Delimiters are removed from the corpus and used to delimit the segments. The inflexions (T, Ü, Ji"), which introduce inflectional variations, are also removed from the corpus but do not delimit the segments. The delimiters identification is con­trolled by rules. For instance tonal particles are removed only if they are the end of a segment, dis­course structure words are removed only if they are the beginning of a segment. Delimiters and inflexions are not removed if they are inside a se­quence of graphies which is present in a common-word dictionary.

In this step, lexical unit candidates (LUCs) are extracted by selecting morpho-syntactic unit cou­ples, which are likely to have a lexical content. The first assumption is that lexical units can al­ways be decomposed into binary trees. Only a small number of lexical units do not satisfy this

Sentence with delimiters noted {delimiter}:

^. #*i&{f }^®{^}#n##.

Obtained segments noted [segment]:

Figure 2: Segment detection example assumption (e.g. ^-JLl,), in such case it is possible to select a non-linguistically motivated way to decompose the unit into binary associa­tions. Thus, every couples of contiguous morpho-syntactic units are iteratively enumerated for each segment. The second assumption is that asso­ciation measures are good statistical evidence to detect lexical content. Thereby, the association strength of morpho-syntactic couples is used as a main criterion to identify relevant candidates.

Consider G the alphabet of all Chinese gra­phies, M = G+ the language describing the morpho-syntactic units, Sn a set of segments at step n, sn = m1,m2,...,mn the ith segment of Sn where Vm e sn | m e M and sn the set of all morpho-syntactic unit couples in Sn seg­ments. Given the morpho-syntactic unit couple mi; m;+i e sn (denoted as m^+i), the lexical con­tent criteria (LCC(m^+i)) matches if the follow­ing conditions are fulfilled:

1. Neither mi nor mi+i has not been associated at the current step n.

2. Nb(mi) = 1 or Nb(mi+i) = 1.

3. AS(mi,i+i) > T.

where Nb(x) is the number of occurrences of x, AS (x, y) returns the association score of the couple x, y computed with a given association mea­sure, and T is the association threshold relative to the association measure (cf. 4.1).

Let So the initial set of segments where Vs0 e So, s0 is a segment (cf. 3.2) such that Vm e s0, m is a morph (cf. 3.1). The LUC list is composed of morpho-syntactic couples produced by the asso­ciation operator © to compute Smax (algorithm 1) with max the maximum number of iteration.

with © the association operator whose result is a morpho-syntactic unit, Sn[mi © m2] the replace­ment of mi and m2 by the morpho-syntactic unit mi © m2 in the corresponding segment. See the Section 5 for more details about the maximum number of iteration setting.

Once LUCs are extracted, we map every LUC to the couple of morpho-syntactic units it is com­posed of. These units are called components. Some LUCs are generated from two different cou­ples at the candidate selection step. For instance, is discovered in two ways: 7k$f©?k3k or ^.^ik©;t\ We always choose the most fre­quent option. When the "LUC/couple" map is created, we sort the LUCs by their correspond­ing couple association scores. Finally, if a LUC is ranked in the list before its components we move the components to the position just before it in the list and use the same rule to recursively check the moved components. The candidates list is ex­pected to be ordered by likelihood deduced by an association measure and compositional order.

The lexicon extraction task aims at submitting a ranked list of candidates to the user in order to help him produce lexical resources. The user is expected to check the list in this order and the method uses the user answers to discard not yet verified candidates. To do so, the user is asked to answer the following questions for each LUC according to the definition given in the Section 2:

The morphs and the segment detection step use data from a reference corpus: The Lancaster Cor­pus of Mandarin Chinese (McEnery and Xiao, 2004). The corpus is composed of text samples choose in various domain and genre corpora, it contains two millions of glyphs and it is anno­tated according to the Beijing University anno­tation guideline. This corpus is mainly used to extract delimiter-words, to produce the gram­mar for delimiter-expressions and to extract a common-word dictionary. All foreign transcrip­tions are also annotated for the CRF tagger train­ing (cf. 3.1).

The lexical unit detection step is evaluated us­ing four well-known association measures: Point-wise Mutual Information (PMI), Poisson-Striling (PS) (Quasthoff and Wolff, 2002), Log-likelihood (LL), Pointwise Mutual Information Cube (PMI) (Daille, 1994). These measures are detailed in ta­ble 5. The significant association threshold is in­tuitively given by the statistical interpretation of Variables

x,y : words x : all words but x px : x probability fx : x frequency

Table 5: Association score calculation the formulas for MI and PS. Thus, these measures are used for LCC's selection criterion 2 and T is set to 0 (cf. 3.3). A threshold can not be deduced from PS and PMI, therefore they are only used for LCC's comparison criteria 3 & 4.

To prepare the evaluation we randomly selected twenty texts in an evaluation corpus and anno­tated lexical units according to the linguistic de­scription given in Section 2. For each sample text, we obtained a set of lexical unit trees (Ta­ble 3) corresponding to all the encountered lex­ical units. N-trees are used for units which can not be transformed into binary tree. Two evalua­tion sets are defined, the shallow set which con­tains the root nodes of the lexical unit trees and the deep set which contains the inner nodes of the lexical unit trees. Given the four examples of Figure 3, the shallow set contains [fiîPè-Ha]], [4 and (f 4Sfc) ; and the deep

(4$), (f 4Sfc) and (f 4S). Experiments with different parameters produce different candidate lists and an expert interven­tion is required to evaluate each candidate list. To avoid this problem, all the repeated sequences of non-inflectional graphies are generated from the annotated sample texts and intersected with the LUC list. The obtained list is a projection of the candidate list on the sample texts. This trick al­lows us to extract all LUCs appearing in the sam-

Figure 3: Lexical unit trees ple texts and evaluate them automatically. 5 Experiments

The experiments are conducted on insurance do­main corpus containing ten million graphies. This evaluation corpus is composed of news and ar­ticles collected automatically from Chinese in­surance companies websites. The text fields are extracted with an xhtml parser. Several text fields, such as menus or buttons, are repeated and duplicates are removed to avoid noise. The presented method, referred as ILex (Incremen­tal Lexicon Extractor), is applied using the pre­viously mentioned 4 measures (cf. 4.1). The evaluation is based on couple of measures, the first measure is dedicated to candidates selection (LCC 2.) and the second to candidates compar­ison (LCC 3. & 4.). The comparison measure is also used to sort the candidates (cf. 3.4). The max­imal number of iterations is set to 3 (for a maxi­mal depth of 4), which is the maximum number of associations required to compose the majority of lexical units in the reference corpus. The preci­sion and recall are computed on the deep set in or­der to consider all valid lexical units, the recall on the shallow set is given to see the results on wider lexical units (Table 6). The results show that PMI-LL couple performs better overall than the other measures. It can be noticed that the scores are rel­atively close (±1.8% for precision and ±7.0% for deep recall) meaning that the choice of the associ­ation measure has a low influence over the results. For the further experiments are conducted with PMI-LL, which achieves the best recall score.