<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="580"/><title>CLGVSM: Adapting Generalized Vector Space Model to Cross-lingual Document Clustering</title><pubinfo>Proceedings of the 5th International Joint Conference on Natural Language Processing,pages 580-588, Chiang Mai, Thailand, November 8 - 13, 2011. ©2011 AFNLP</pubinfo><author surname="Tang" givenname="Guoyu"><org  name="Institute for Infocomm Research" country="Singapore"/></author><author surname="Xia" givenname="Yunqing"><org  name="Tsinghua University" country="China" city="Beijing"/></author><author surname="Zhang" givenname="Min"><org  name="Tsinghua University" country="China" city="Beijing"/></author><author surname="Li" givenname="Haizhou"><org  name="Institute for Infocomm Research" country="Singapore"/></author><author surname="Zheng" givenname="Fang"><org  name="Institute for Infocomm Research" country="Singapore"/></author></firstpageheader><frontmatter><p><b>CLGVSM: Adapting Generalized Vector Space Model to Cross-lingual Document Clustering</b></p><p><b>12 1 2 2 1</b></p><p><b>Guoyu Tang ' , Yunqing Xia , Min Zhang , Haizhou Li ,Fang Zheng</b></p><p><footnote anchor="1"/>Dept. of Comp. Sci. &amp; Tech., Tsinghua University, Beijing 100084, China</p><p>sweetyuer@gmail.com,{yqxia,fzheng}@tsinghua.edu.cn</p><p><footnote anchor="2"/>Institute for Infocomm Research, A-STAR, Singapore</p><p>{mzhang, hli}@i2r.a-star.edu.sg</p></frontmatter><abstract>Cross-lingual document clustering (CLDC) is the task to automatically or­ganize a large collection of cross-lingual documents into groups considering con­tent or topic. Different from the tradition­al hard matching strategy, this paper ex­tends traditional generalized vector space model (GVSM) to handle cross-lingual cases, referred to as CLGVSM, by incor­porating cross-lingual word similarity measures. With this model, we further compare different word similarity measures in cross-lingual document clus­tering. To select cross-lingual features ef­fectively, we also propose a <i>soft-matching </i>based feature selection method in CLGVSM. Experimental results on benchmarking data set show that (1) the proposed CLGVSM is very effective for cross-document clustering, outperform­ing the two strong baselines vector space model (VSM) and latent semantic analy­sis (LSA) significantly; and (2) the new feature selection method can further im­prove CLGVSM. </abstract></header><body><section number="1" title="Introduction"><p>The globalization of business environment urges organizations to maintain documents in different language. Obviously, organizations and research communities nowadays encounter the challenge of cross-lingual document clustering (CLDC). Document clustering seeks to automatically or­ganize a large collection of documents into groups of similar documents. Various document clustering technologies have been proposed to deal with monolingual documents.</p><p>The classical solution to monolingual docu­ment clustering is vector space model (VSM), which explores bag of words (BOW) to construct feature space. Each document is converted to a VSM vector. Serious problem occurs when words are matched to the features using the hard matching strategy. For example, when the word <i>coast </i>is selected as a feature, word <i>seashore </i>will not contribute to hard matching unless it is also selected as a feature.</p><p>Different semantic document representation models have been proposed to address the short­coming of VSM. Some semantic document rep­resenting models such as Latent Semantic Analy­sis (LSA) (Landauer et al., 1998) and Latent Di-richlet Allocation (Blei et al., 2003) implicitly capture statistical semantics by mapping docu­ments to a lower dimension space. Other models such as Generalized Vector Space Model (GVSM) (Wang et al., 1985) extract statistical semantics in an explicit way by directly estimat­ing measures of correlations between words.</p><p>The above models are designed for monolin­gual document sets and cannot be applied to cross-lingual scenario unless a bridge is created to connect cross-lingual features. Carbonell et al. (1997) use semantic models in parallel corpus. As features are selected from a common paral­lel/comparable corpus, which is usually different from the test cross-lingual document, over-fitting problem inevitably happens.</p><p>Other researchers propose to translate features or documents with bilingual dictionary or ma­chine translation tools (Mathieu et al., 2004).However, ambiguity happens constantly and it is difficult to determine translation of a word. Meanwhile, if one translation of a word is selected as feature, the hard matching problem still occurs and becomes more serious.</p><p>In this paper, we extend the monolingual GVSM to handle cross-lingual cases, referred to as Cross-lingual GVSM (CLGVSM). Besides term correlation, we make use of word similarity in CLGVSM. For the cross-lingual we use statis­tical word similarity measure with the parallel corpus. We further improve cross-lingual word similarity by incorporating dictionary or transla­tion probability.<page local="2" global="581"/> Experimental results show that the best result is achieved when combining the Second Order Co-occurrence Pointwise Mutual Information (SOCPMI) measure on the test da­taset and translation probability in development dataset.</p><p>Selecting cross-lingual features is a key issue in cross-lingual document clustering. In this work, we propose a <i>soft-matching </i>based feature selection method in CLGVSM. In the new fea­ture selection method, most representative terms are selected in semantic space according to Soft Term Frequency and Soft Document Frequency. In this way, a non-feature word can improve weight of the semantically similar features and make contribution to document clustering. Ex­perimental results show that CLGVSM outper­forms both LSA and VSM significantly with the help of proper word similarity measure.</p><p>The rest of this paper is organized as follows. In section 2, related work is surveyed. In section 3, the CLGVSM model is discussed. Experi­mental results as well as discussion are presented in Section 4. We conclude this paper in Section 5.</p></section><section number="2" title="Related work"><subsection number="2.1" title="Monolingual Document Representation Models"><p>The most commonly used model for document representation is the vector space model (VSM). It is assumed in VSM that terms are independent of each other and thus any semantic relations between them are ignored. Proposed by Landau­er et al. (1998), LSA seeks to decompose the term-document matrix using singular value de­composition, in which each feature is a linear combination of all words.</p><p>Proposed by Wang et al. (1985) and further improved by Farahat and Kamel (2010), GVSM is proved an effective document representation model to address limitation of VSM. The model estimates similarity between documents based on how much their terms are related. Wang et al. (1985) pointed out orthonormal basis in VSM and proposed a new model to remove the as­sumption. Farahat and Kamel (2010) improved GVSM by developing better estimation of term correlation and applying dimension reduction techniques in a semantic space. Results show that the improved GVSM is advantageous over other representation models such as LSA.</p><p>Other document representation models are based on lexical ontologies such as WordNet, to represent documents in the concept space (Hotho et al., 2003). Similar representation models are seeks to exploit knowledge within an encyclope­dia. Explicit Semantic Analysis (Cimiano et al., 2009) is a famous model that represents words as vectors in a space of concepts represented by articles from Wikipedia.</p><p>Most of those semantic models are designed for monolingual document sets, and cannot be used in cross-lingual scenario directly.</p></subsection><subsection number="2.2" title="Cross-lingual Document Clustering"><p>The difficulty of CLDC is how to deal with cross-language issue. The straightforward solu­tion is document translation. In TDT3<footnote anchor="1"/>, four sys­tems attempted to used Machine Translation sys­tems (Leek et al., 1999). The results show that using a machine translation tool leads to around 50% performance loss, compared with monolin­gual topic tracking. This ascribed mainly to the poor accuracy of machine translation systems.</p><p>Dictionary and corpus are two popular ways to get cross-language information. Some researches (Evans and Klavans, 2003) use dictionary to translate documents. Others (Mathieu et al., 2004) use dictionary to translate features or keywords. But it is hard to select proper translation of am­biguous words. Mathieu et al. (2004) use bilin­gual dictionaries to translate named entities and keywords and modify the cosine similarity for­mula to calculate similarity between bilingual documents. Pouliquen et.al (2004) rely on a mul­tilingual thesaurus called Eurovoc to create cross-lingual article vectors.</p><p>Wei et al. (2008) use LSA to construct a mul­tilingual semantic space onto which words and document in either language can be mapped and dimensions are reduced again according to doc­uments to be cluster. Yogatama and Tanaka-Ishii (2009) use propagation algorithm to merge mul­tilingual spaces from comparable corpus and spectral method to cluster documents. Li et al. (2007) use Kernel Canonical Correlation Analy­sis, a method of finding the maximally correlated projections of documents in two languages for cross-language Japanese-English patent retrieval and document classification. Unlike document classification, document clustering lacks training data. So semantic space is constructed from the parallel/comparable corpus, and the dimensions are selected on the basis of their importance in parallel/comparable corpus, which is usually dif­ferent from the target multilingual documents.</p><footnote label="1">http://www.itl.nist.gov/iad/mig//tests/tdt/1999/index.html</footnote><page local="3" global="582"/><p>In this work, our proposed CLGVSM use se­mantic similarity to solve word matching prob­lem caused by different languages. Semantic space is constructed based on word similarity and in our feature selection method, features are select on the basis of their importance in docu­ments to be clustered.</p></subsection><subsection number="2.3" title="Cross-lingual Word Similarity"><p>In both monolingual GVSM (Wang et al, 1985) and improved GVSM (Farahat and Kamel, 2010), correlation, correlation between words is com­puted in documents to be clustered and correla­tion of the best performance in document cluster­ing is calculate as covariance of words with the assumption that words are random variables with Gaussian distributions. In this work we use word similarity which is calculated as cosine similarity of term vector covariance. This measure can be called as COV measure.</p><p>But this similarities method is estimated in test documents which lacks cross-lingual information.</p><p>Various measures for cross-lingual word se­mantic similarity have been proposed to explore statistical techniques and semantic network.</p><p>Research works propose to use <i>WordNet </i>by Resnik (1999) to measure similarity between English words. Liu and Li (2002) adopt HowNet calculate word similarity in machine translation. Xia et al. (2011) propose to explore cross-lingual word similarity by observing concept definition provided by HowNet.</p><p>Corpus-based measures for semantic similarity are found more interesting. The classical method is Pointwise Mutual Information (PMI) (Church and Hanks, 1990). Many researches are based on</p><p>PMI, such as PMI-IR (Turney, 2001) and Second Order Co-occurrence PMI (SOCPMI) (Islam and Inkpen, 2006). SOCPMI is proved better than PMI-IR and some other similarity measures (Is­lam and Inkpen, 2006).</p><p>In this work, we implement three representa­tive measures: HowNet-based measure (Xia et al., 2011), SOCPMI measure (Islam and Inkpen, 2006) and COV measure (Farahat and Kamel, 2010).</p></subsection></section><section number="3" title="Cross-Lingual Generalized VSM 3.1    Generalized VSM"><p>Let <i>D</i><i> </i><i>=</i><i> </i><i>{dj\j</i><i> </i><i>=</i><i> </i>1.... \ be a set of A?" docu­ments which contain M terms, A" be <i>aMxN</i><i> </i>ma­trix whose element <i>xtj</i><i> </i>represents the weight of term <i>ti </i>in document <i>dj.</i><i> </i>GVSM (Wang et al, 1985) estimates correlation between documents based on how their terms are related. The GVSM model presents document in a non-orthogonal space and similarity between two documents is calculated as follows.</p><doubt alpha="60.0" length="5" tooSmall="False" monospace="0.0">d\Gd2</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(1)</doubt><p><b>c</b>/f <i>Gdi yJd^Gd</i></p><p>where <i>G </i>is an <i>M </i>x <i>M </i>association matrix which represents correlations between terms and is usu­ally computed as inner-products of term vectors in some space. An example of 3 x 3 association matrix is given as follows.</p><p>where every row and column represents a term, respectively.</p><p>In tradition GVSM (Wang et al, 1985), terms are represented as vectors in the dual space of documents and the association measures between terms are calculated as the cosine of the angle between their vectors in the dual space. Accord-ingly,G can be calculated as:</p><doubt alpha="57.1" length="14" tooSmall="False" monospace="0.0">G=L~l/2XXTL-V2</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(2)</doubt><p>where <b><i>L </i></b>is a diagonal matrix whose elements are the lengths of term vectors in the dual space.</p><p>And in improved GVSM (Farahat and Kamel, 2010), the best 6'which is covariance matrix of terms is calculated as</p><doubt alpha="53.3" length="15" tooSmall="False" monospace="0.0">Gr-ov - —^—QHQT</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(4)</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(5)</doubt><doubt alpha="60.0" length="5" tooSmall="False" monospace="0.0">Tic—1</doubt><p><i>Gcov </i>maps uncorrected terms to near-orthogonal directions and negatively correlated terms to opposite directions in the semantic space, while traditional <i>G </i>maps both uncorrelat-ed terms and negatively correlated terms to near-orthogonal directions. Thus we use cosine simi­larities between term vectors in case of Gcov as one of our similarity measures. As the GVSM models proposed, <i>G </i>is estimated from documents to be clustered; they cannot acquire cross-lingual information and cannot deal with cross-lingual issues directly. So we extend GVSM to cross-lingual GVSM by using cross-lingual word simi­larity measures in section 3.2 where <b><i>Q </i></b>is random sample of A'and<page local="4" global="583"/></p><doubt alpha="40.0" length="10" tooSmall="False" monospace="0.0">H= 1 -—eeT</doubt><subsection number="3.2" title="Cross-Lingual GVSM"><p>Note that before (Farahat and Kamel, 2010); term correlation was used in GVSM to construct association matrixes <i>G </i>with the inner-product of term vectors in some semantic space. Length of term vectors quantifies how important of the term in the documents. Thus correlation of terms is not totally the same with similarity of terms. But it is difficult for term correlation to adapt into cross-lingual case. Term vectors generated from test data lack cross-lingual information. Carbonell et.al (1997), generate term vectors from development data. Noise occurs because term importance differs in two dataset.</p><p>For those reasons mentioned above, we choose word similarity instead of term correlation to construct word association matrix. And the other advantage of using word similarity is that we can ignore noisy similarity values which contribute little to document similarity calculation. In that case, the associate matrix becomes sparse and computational time can be saved. Therefore in this work, we explore the following several word similarity measures in constructing word associa­tion and setup a similarity threshold.</p><p><b>Knowledge-based Similarity Measures</b></p><p>We choose to use cross-lingual word similarity based on HowNet (Xia et al., 2011) which makes use of concept graph in HowNet. HowNet is concept based and the atom unit is sememe, so similarity between words is actually reflected by the sememes they carry. The key idea of cross-lingual word similarity calculation is to locate bilingually definitions for given words so that the language barrier is overcome. For details please refer to Xia et al. (2011).</p><p><b>Statistical Similarity Measures</b></p><p>Statistical similarity actually reflects conceptual relevance between words as it considers merely word co-occurrences within a corpus. We evalu­ate two statistical similarity measures: SOCPMI and COV in this work.</p><p>SOCPMI was proposed by Islam and Inkpen (2006), in which PMI is applied to rank the neighboring words with in a corpus. The measure is proved accurate because it calculates relevance between two words that do not co-occur fre­quently. Note that the original SOCPMI measure is designed to deal with monolingual word simi­larity. We extend this measure in this work to calculate similarity between cross-lingual words. The goal is achieved by counting neighboring words with the same language in the corpus and computing the cross-lingual PMI in a parallel corpus.</p><p>As we mentioned above, associate matrix <i>G </i>constructed by covariance of term vectors achieve the best performance in monolingual document clustering. In this paper we use the cosine similarity instead of inner-product in COV similarity measure.</p><p>The two above word similarity measures both need to be calculated in a cross-lingual paral­lel/comparable corpus.</p><p><b>Combining Similarity with Dictionary or Translation probability</b></p><p>The statistical word similarity measures are de­veloped with general development corpus. How­ever, we believe word co-occurrence in the test documents is also very useful. Thus we improve cross-lingual word similarity in the following way: statistical monolingual word similarity is computed from test documents first and a dic­tionary or translation probability as a bridge is used to get cross-lingual word similarity.</p><p>When using dictionary as bridge, assuming word <i>Wj</i><i> </i>which has a translation list <b><i>T?={tJk;</i></b> <b><i>k </i></b>= <b><i>1.</i></b><b><i>..NJ} </i></b>and word <b><i>w</i></b>/which has a translation <b><i>list </i></b><b><i>If</i></b><b><i> ={t]k;k = I...NJ}, </i></b>we choose the highest value between similarities of w.',and each words in TTand similarities of <b><i>Wj </i></b>and each words in <b><i>Tjas </i></b>similarity between <b><i>Wi</i></b><b><i> </i></b>and <b><i>Wj.</i></b><b><i></i></b></p><doubt alpha="100.0" length="2" tooSmall="True" monospace="0.0">JJ</doubt><p>When using translation probability as bridge, assuming word <b><i>Wj </i></b>which has a translation proba­bility list <b><i>P?={tjk. Pik;k = l...N}v} </i></b>and word <b><i>wj </i></b>which has a translation list <b><i>Pj </i></b><b><i>={fjk</i></b><b><i> - Pjk. ; k </i></b>= <b><i>1...NJ<footnote anchor="1"/></i></b>}, the similarity of i/'^and <b><i>Wj </i></b>can be calculated as follows:</p><doubt alpha="43.8" length="16" tooSmall="False" monospace="0.0">S/'mPR(»!J—&gt;wj)=</doubt><doubt alpha="0.0" length="25" tooSmall="False" monospace="0.0">\;;■        "       ■ (6)</doubt><doubt alpha="53.8" length="13" tooSmall="False" monospace="0.0">Sim™(wj,Wj) =</doubt><p>where £tmMono returns monolingual word simi­larity.</p></subsection><subsection number="3.3" title="Feature selection for GVSM"><p><b>Term Importance based on GVSM</b></p><p>In the VSM model, importance of a term is pro­portional to Term Frequency <i>(TF) </i>in a single document, and inversed proportional to Docu­ment Frequency <i>(DF) </i>in a document set. We ar­gue that the theory can be improved.</p><page local="5" global="584"/><p>Consider a document that contains term <i>crimi­nal </i>for 3 times and term <i>imprisonment </i>for 10 time. We find that term <i>criminal </i>is still very im­portant though its <i>TF </i>is low. This is because term <i>imprisonment </i>is semantically similar to <i>criminal </i>and it appears many times. If the classical term matching method is named <i>hard matching, </i>we can call our method <i>soft matching. </i>We incorpo­rate the <i>soft-matching </i>idea to the GVSM model.</p><p>In the GVSM model, importance of a term can be reflected by the following statistics.</p><p>(1) Soft Term Frequency</p><p>The soft term frequency <b><i>(TF*) </i></b>considers the term and the semantically similar terms. Given term/ and document */, we first retrieve the se-mantically similar terms T={t»}i-i j^Tfrom document*/. We define the soft term frequency (/ /'") as follows.</p><doubt alpha="66.7" length="24" tooSmall="False" monospace="0.0">TFs(t,d)=EiTFlxSim(t, U)</doubt><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">(9)</doubt><p>where <b><i>TF; </i></b>denotes term frequency of term / within document <b><i>d.</i></b></p><p>Note that the soft term frequency is calculated within a single document.</p><p>(2) Soft Document Frequency</p><p>The soft document frequency <b><i>(DFa) </i></b>considers not only number of documents that contain the term, but also the number of documents that con­tains the semantically similar terms. Given term* and document set <b><i>D = </i></b>{dj}j=i..jv . Let <b><i>dj = {ti,j}i=i</i></b><i>...MD </i>denote terms within document*/,. We define the soft document frequency <b><i>(DFS) </i></b>as follows.</p><doubt alpha="58.8" length="34" tooSmall="False" monospace="0.0">DFs{t) =Zdj€DUiaMSirn(t,tLj))}(10)</doubt><p>Note that the soft document frequency is cal­culated within a document set. We use maximum instead of summation in order to reduce the ef­fects of word pairs with low similarities.</p><p><b>Feature Selection in GVSM</b></p><p>With GVSM-based term importance, features can be selected appropriately. Following the idea of TF-IDF, we refine inverse document frequen­cy as follows.</p><doubt alpha="50.0" length="20" tooSmall="False" monospace="0.0">IDFs(t) = log(—ff--)</doubt><p><b><i>DFs(t).</i></b></p><p>where iVdenotes number of documents.</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">(11)</doubt><p>The soft weighting equation of term <b><i>t </i></b>in doc­uments/is as follows.</p><doubt alpha="3.4" length="29" tooSmall="False" monospace="0.0">„■■■,/.,/.7T-./,/■//&gt;/"/.(12)</doubt><p>If we select features for a document based solely on term weight, some semantically similar terms might be selected because they hold close weights. This results in fewer representatives feature set. Before feature selection we update term <b><i>TF*as</i></b><b><i> </i></b>follows :</p><p>1) Setup an initial term list.</p><p>2) Sort terms in the list according to their <b><i>TFa'm</i></b><b><i> </i></b>descending order.</p><p>3) Move the first term/ointo the tentative fea­ture list.</p><p>4) For each remaining term in the list, update its <b><i>TF*</i></b><b><i> </i></b>using the following equation.</p><doubt alpha="45.0" length="20" tooSmall="False" monospace="0.0">TF*+l(t,d)=TF*(t.d)-</doubt><doubt alpha="3.3" length="30" tooSmall="False" monospace="0.0">■],,s,,.,./„.;.//■./.■.,/.(13)</doubt><p>whereia- denotes a term in document <b><i>d, </i></b>and <b><i>g </i></b>the iteration round number.</p><p>5) Delete terms a weight less than 0in the list.</p><p>6) Repeat step 2) ~ 5) until the term list be­comes empty.</p><p>Once the tentative feature list is obtained, we then calculate weight for each candidate features using Eq (12). The features of each document are then ranked according to the weight and joined together to represent document set.</p><p><b>Document representation in GVSM</b></p><p>With the feature set available, we describe how a document is represented with the features. Leti^ <b><i>= ifi}i=i</i></b><i>...MF</i>denote the feature set, and T = {*j}j=i...MD denote terms within document*/. We now try to map <b><i>d </i></b>to the feature space.</p><p>For each feature, it should be mapped no mat­ter whether it appears in the document set. But in order to avoid redundant information, we map only one term with each feature.</p><p>So for feature list <b><i>F </i></b>sort by <b><i>TF<footnote anchor="3"/> </i></b>in document (/, the actually weight of feature <b><i>fj </i></b>in document <b><i>d </i></b>is as follows:</p><p>First, retrieve the most similar term <i>t* </i>which is not included in TM, which stores terms that have been matched.</p><doubt alpha="50.0" length="32" tooSmall="False" monospace="0.0">t*=argmaxt.€dt.£TMSim(t,/,) (14)</doubt><p>Then we re-calculate weight io°of term /^as follows.</p><doubt alpha="58.3" length="24" tooSmall="False" monospace="0.0">wa(fud) = TF(t*)IDFs(fi)</doubt><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">(15)</doubt><p>Finally, put l*into TM and repeat until all fea­tures are matched once.</p><page local="6" global="585"/></subsection><subsection number="3.4" title="Document Clustering based on GVSM"><p>With document similarity, we employ certain clustering algorithm to manage the cross-lingual documents with a few clusters. As clustering al­gorithm is not core of this work, we simply choose the classic document clustering algorithm, i.e., HAC (Hierarchical Agglomerative Cluster­ing) algorithm (Voorhees, 1986). To measure cluster-cluster similarity, we adopt the group-average link algorithm (Voorhees, 1986). The merging procedure repeats until a desired num­ber of clusters are obtained.</p></subsection></section><section number="4" title="Evaluation 4.1 Setup"><p><b>Development dataset: </b>We randomly extract 1M parallel sentence pairs from LDC corpora (i.e.,</p><doubt alpha="35.2" length="71" tooSmall="False" monospace="0.0">LDC2004E12, LDC2004T08, LDC2005T10, LDC2003E14, LDC2002E18m LDC2005T06,</doubt><p>LDC2003E07 and LDC2004T07) as our devel­opment data to train the bilingual corpus-based term similarity and get translation probability. <b>Dictionary: </b>Translation pairs are extracted from HowNet.</p><p><b>Translation Probability: </b>We compute transla­tion probability by Giza++ (Och and Ney, 2000) in development data.</p><p><b>Test dataset: </b>Four datasets are tested in this pa­per.</p><p><b>Table2. </b>Statistics on the two CLTC datasets.</p><p><b>TDT4 datasets</b></p><p>We first extract two datasets from the TDT4 evaluation dataset (see statistics in Table 1).</p><p><b>CLTC datasets</b></p><p>The second dataset is extracted from our own cross-lingual topic corpus (CLTC). The news articles are retrieved from Gigaword (English and Chinese), and the topics are labeled by hu­man (see statistics in Table 2).</p><p><b>Evaluation criteria</b></p><p>We adopt the evaluation criteria proposed by Steinbach et al. (2000). The calculation starts from maximum F-measure of each cluster. Let <i>A; </i>represent the set of articles managed in a sys­tem-generated cluster <b><i>Cj, </i></b><i>Aj</i><i> </i>the set of articles managed in a human-generated cluster &lt;■-,. F-measure of the system-generated cluster <i>a </i>is cal­culated as follows.</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">(16)</doubt><p>where <b><i>pij, nj </i></b>and /^represent precision, recall and measure of cluster when compared with clustery , respectively.</p><p>We also use relative F-measure to compare systems over all dataset which is used by Farahat and Kamel (2010).In this approach, the F meas­ure for a particular data set are normalized rela­tive to the best value obtained using different representation models when applying the same clustering algorithm to the same data set:</p><doubt alpha="45.5" length="11" tooSmall="False" monospace="0.0">Fr=F.pV(17)</doubt><p>where <i>Fi</i><i> </i>denotes F-measure values obtained us­ing different representation models.</p><p>The relative F measures are then averaged for different data sets.</p><subsection number="4.2" title="Evaluation"><p><b>Experiment 1: Different word similarity cal­culation measures</b></p><p>This experiment seeks to compare different cross-lingual word similarity (CLWS) measures. Seven CLWS measures are implemented: <b>HN: </b>HowNet-based cross-lingual word similari­ty measure.</p><p><b>SOCPMIADEV: </b>SOCPMI similarity measure learned from development data. <b>SOCPMI&amp;DIC: </b>SOCPMI similarity measure calculated in test documents and dictionary as cross-lingual bridge.</p><p><b>SOCPMI&amp;TranPro: </b>SOCPMI similarity meas­ure directly computed in test documents and translation probability on development set as cross-lingual bridge.</p><p><b>COVADEV: </b>COV similarity measure learned from development data.</p><p><b>COV&amp;DIC: </b>COV similarity measure computed in test documents and dictionary as cross-lingual bridge.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Corpus</p></td><td class="cell"><p>TDT41 (2002) (Topic#/Story#)</p></td><td class="cell"><p>TDT42 (2003) (Topic#/Story#)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>English</p></td><td class="cell"><p>38/1270</p></td><td class="cell"><p>33/617</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Chinese</p></td><td class="cell"><p>37/657</p></td><td class="cell"><p>32/560</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Common</p></td><td class="cell"><p>40/1927</p></td><td class="cell"><p>37/1177</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Table1. </b>Statistics on the twoTDT4 datasets.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Corpus</p></td><td class="cell"><p>CLTC1 (Topic#/Story#)</p></td><td class="cell"><p>CLTC2 (Topic#/Story#)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>English</p></td><td class="cell"><p>20/200</p></td><td class="cell"><p>20/600</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Chinese</p></td><td class="cell"><p>20/200</p></td><td class="cell"><p>20/600</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Common</p></td><td class="cell"><p>20/400</p></td><td class="cell"><p>20/1200</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="7" global="586"/><p><b>Table 4. </b>Average of relative F-measure (ARF) of CLDC systems with different CLWS measures.</p><p><b>COV&amp;TranPro: </b>COV similarity measure in test documents and translation probability as cross-lingual bridge.</p><p>All the CLDC systems use HAC algorithm to do clustering documents. The thresholds of simi­larity measures in this paper is all set 0.4 based on our empirical study. Experiment results on four datasets are as Table 3. Table 4 computed from Table 3 shows the average of relative F-measure (ARF) over all data sets for different CLWS measures.</p><p>We can observe from Table 3 and Table 4 that the performance of HowNet is much worse than other systems in all dataset. We look into the intermediate results to check the reasons. We find semantic similarities between words com­puted based on HowNet are too high. For exam­ple, word similarity between <i>Federal Reserve </i>and <i>bank </i>is assigned 1 by HowNet. Error analy­sis shows that HowNet-based CLWS measure puts much emphasis upon the semantic property of given word rather than semantic itself. So it tends to assign bigger CLWS values to semanti-cally similar word pairs, no matter how semanti-cally relevant they are. This would obviously jeopardize document clustering. With such an observation, we conclude that HowNet-based CLWS measure is not suitable for document clustering.</p><p>We can also observe that systems with transla­tion probability outperform those with dictionary when the same monolingual word similarity measures are used. For instance, SOCPMI&amp;TranPro outperforms SOCPMI&amp;DIC by 4.5% on average on relative F-measure and COV&amp;TranPro outperforms COV&amp;DIC by 1.9%. Two reasons are worth noting. First, dictionary extracted from HowNet have more OOV than translation probability computed from develop­ment corpus. Translation probability is more dis­criminative than dictionary when word is ambig­uous. It tries to get word similarity from the most frequency translation.</p><p>Seen from Table 4 that SOCPMI&amp;TranPro outperforms SOCPMIADEV by 1.5% on average relative F-measure and COV&amp;TranPro outper­forms COVADEV by 5.5% on average relative F-measure. As both systems use the same devel­opment data to get cross-lingual information and the different is that systems computed word simi­larity in test dataset take use of word occurrence information in test dataset so we can conclude that with combining word similarity in test da­taset and translation probability can is useful in cross-lingual document clustering.</p><p>And over all seven systems, SOCPMI&amp;TRAN achieves the best result on average, so we select</p><p>SOCPMI&amp;TRAN as our word similarity measures and next experiments both use this word similarity measure.</p><p><b>Experiment 2: Different feature selection vs. dimension reduction methods</b></p><p>This experiment aims to compare the proposed feature selection method and the existing ones. Three CLDC systems are implemented. SFS:    feature selection we proposed by <b><i>TF*-</i></b>! ! ' / and soft matching is used in GVSM. <b>HFS: </b>feature selection by TF-IDF and hard matching is used in GVSM. <b>NFS: </b>feature selection is not used, which equals to system SOCPMI&amp;TranPro in Experiment 1.</p><p><b>Table5. </b>Highest F-measure of CLDC systems with/without feature selection.</p><doubt alpha="50.9" length="53" tooSmall="False" monospace="0.0">SystemlSFSIHFSINFS~F-measureI0.998|   0.980   | 0.976</doubt><p><b>Table6. </b>Average of relative F-measure of CLGVSM systems with/without feature selection.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Dataset"-\</p></td><td class="cell"><p>HN</p></td><td class="cell"><p>SOCPMI ADEV</p></td><td class="cell"><p>SOCPMI &amp;DIC</p></td><td class="cell"><p>SOCPMI &amp;TranPro</p></td><td class="cell"><p>COV ADEV</p></td><td class="cell"><p>COV &amp;DIC</p></td><td class="cell"><p>COV &amp;TranPro</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>TDT41</p></td><td class="cell"><p>0.783</p></td><td class="cell"><p>0.880</p></td><td class="cell"><p>0.854</p></td><td class="cell"><p>0.892</p></td><td class="cell"><p>0.824</p></td><td class="cell"><p>0.868</p></td><td class="cell"><p>0.907</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>TDT42</p></td><td class="cell"><p>0.797</p></td><td class="cell"><p>0.880</p></td><td class="cell"><p>0.835</p></td><td class="cell"><p>0.880</p></td><td class="cell"><p>0.860</p></td><td class="cell"><p>0.840</p></td><td class="cell"><p>0.851</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CLTC1</p></td><td class="cell"><p>0.764</p></td><td class="cell"><p>0.818</p></td><td class="cell"><p>0.834</p></td><td class="cell"><p>0.877</p></td><td class="cell"><p>0.782</p></td><td class="cell"><p>0.854</p></td><td class="cell"><p>0.874</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CLTC2</p></td><td class="cell"><p>0.667</p></td><td class="cell"><p>0.856</p></td><td class="cell"><p>0.804</p></td><td class="cell"><p>0.839</p></td><td class="cell"><p>0.805</p></td><td class="cell"><p>0.833</p></td><td class="cell"><p>0.840</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Table 3. </b>Highest F-measure of CLDC systems with different CLWS measures.</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>System</p></td><td class="cell"><p>HN</p></td><td class="cell"><p>SOCPMI ADEV</p></td><td class="cell"><p>SOCPMI &amp;DIC</p></td><td class="cell"><p>SOCPMI &amp;TranPro</p></td><td class="cell"><p>COV ADEV</p></td><td class="cell"><p>COV</p><p>&amp;DIC</p></td><td class="cell"><p>COV &amp;TranPro</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>ARF</p></td><td class="cell"><p>0.855</p></td><td class="cell"><p>0.976</p></td><td class="cell"><p>0.945</p></td><td class="cell"><p><b>0.991</b></p></td><td class="cell"><p>0.929</p></td><td class="cell"><p>0.965</p></td><td class="cell"><p>0.986</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>""""---System Dataset""---....,</p></td><td class="cell"><p>SFS</p></td><td class="cell"><p>HFS</p></td><td class="cell"><p>NFS</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>TDT41</p></td><td class="cell"><p>0.900</p></td><td class="cell"><p>0.903</p></td><td class="cell"><p>0.892</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>TDT42</p></td><td class="cell"><p>0.899</p></td><td class="cell"><p>0.881</p></td><td class="cell"><p>0.880</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CLTC1</p></td><td class="cell"><p>0.876</p></td><td class="cell"><p>0.869</p></td><td class="cell"><p>0.877</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>CLTC2</p></td><td class="cell"><p>0.891</p></td><td class="cell"><p>0.847</p></td><td class="cell"><p>0.839</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="8" global="587"/><p>Experiment results on four test datasets are given in Table 5. Table 6 computed from Table 5 shows the ARF over all data sets for different feature selection methods.</p><p>Seen from Table 6, system with feature selec­tion we proposed using soft matching outper­forms system with feature selection using hard matching by 1.8% on average relative F-measure It also outperforms system without feature selec­tion by 2.2% on average relative F-measure.</p><p>This reveals that feature selection does im­prove GVSM. The reason why it works is that with <b><i>TF<footnote anchor="9"/>mdDFa,</i></b><b><i> </i></b>it can select the most repre­sentative terms as feature set and with proper document representation method, documents are properly matched into feature space.</p><p><b>Experiment 3: Different document represen­tation models</b></p><p>This experiment aims to compare CLGVSM with VSM and LSA. Three CLDC systems are im­plemented:</p><p><b>CLGVSM: </b>Our system, which equals FS system in Experiment 2.</p><p><b>VSM: </b>A baseline system that uses VSM to rep­resent documents and cosine similarity to com­pute document systems. HowNet dictionary is used to match terms in different languages. <b>LSA: </b>LSA uses dictionary to match terms in dif­ferent languages and make use of LSA in test dataset. The number of LSA dimensions is set to</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">200,</doubt><p>Experiment results on two data sets are given in Table 7. Table 8 computed from Table 7 shows the ARF over all data sets for different document representation models.</p><doubt alpha="47.1" length="51" tooSmall="False" monospace="0.0">SystemICLGVSMIVSMILSATDT41        0.900 0.877 0.885</doubt><doubt alpha="10.0" length="30" tooSmall="False" monospace="0.0">TDT42        0.899 0.835 0.881</doubt><doubt alpha="13.3" length="30" tooSmall="False" monospace="0.0">CLTC1        0.876 0.792 0.867</doubt><p>CLTC2      |q.891       | 0.776      | 0.841 <b>Table7. </b>Highest F-measure of CLDCsystems with different document representation models.</p><doubt alpha="53.7" length="54" tooSmall="False" monospace="0.0">SystemICLGVSMIVSMILSA F-measure |1| 0.920      | 0.974</doubt><p><b>Table8. </b>Average of relative F-measure of CLDCsys-tems with different document representation models.</p><p>We can observe from Table 8 that CLGVSM outperforms VSM by 8.0% on average relative F-measure. It means CLGVSM improve cross-lingual document clustering by using SOCPMIATRAN similarity measure. Observa­tion shows that the word similarity measure makes significant contribution to document clus­tering. Using second order co-occurrence infor­mation of words, SOCPMI assigns word pair with higher PMI a higher similarity. This coin­cides perfectly with the real demand in word similarity measuring. For example, word similar­ity between ^EflftfF<footnote anchor="1"/> <i>(criminal) </i>and <i>imprison­ment </i>is assigned 0.49 by SOCPMI. When <i>$&amp;f$ ft-f </i>is chosen as a feature, document containing <i>imprisonment </i>holds a reasonable similarity with document containing ^EflftfF<footnote anchor="1"/> even though they do not contain common word.</p><p>Results also show that CLGVSM outperforms LSA by 2.6% on average relative F-measure. It means CLGVSM is better than LSA in cross-lingual    document    clustering    by using</p><p>SOCPMIATRAN similarity measure. The follow reason is worth noting. When dictionary is used to match words in LSA, the semantic relation between different translations of one term in one document is added and this brings much noise. While in CLGVSM, cross-lingual terms are soft matched by SOCPMIATRAN term similarity. <b>5 Conclusion</b></p><p>In this paper, we extend monolingual generalized VSM (GVSM) to handle cross-lingual cases, re­ferred to as CLGVSM, by incorporating cross-lingual word similarity measures.Under GVSM, we compare different word similarity measures in cross-lingual document clustering. We pro­pose new feature selection method for CLGVSM and experiments show it improves document clustering. We also compare CLGVSM and other well-known document representation models such as VSM and LSA and experiments show it outperform both VSM and LSA significantly.</p><p>Three conclusions can be drawn in this paper. Firstly, HowNet-based word similarity method is less suitable for document clustering. Secondly, translation probability computed from a devel­opment dataset as a cross-lingual bridge per­forms better than HowNet dictionary. At last, combining word similarity in test dataset and translation probability in development dataset can help cross-lingual document clustering.</p><p>In the future, we will apply CLGVSM in more languages pairs and extend it in more than two languages. As GVSM represents document with semantic space, we can utilize GVSM to handle sparse data problem in short text clustering.</p><page local="9" global="588"/><p><b>Acknowledgment</b></p><p>This work is partially supported by NSFC (60703051) and MOST (2009DFA12970). We thank the reviewers for the valuable comments.</p></subsection></section><references><p>D. M. Blei, A. Y. Ng, and M.I. Jordan. 2003. Latent dirichlet allocation. J. Machine Learning Research (3):993-1022.</p><p>K. W. Church and P. Hanks. 1990. Word association norms, mutual information, and lexicography. Computational Linguistics, 16(1):22-29.</p><p>P. Cimiano, A. Schultz, S. Sizov, P. Sorg, and S. Staab. Explicit vs. latent concept models for cross-language information retrieval. Proc. of IJCAI'09, 2009.</p><p>C. Corley and R. Mihalcea. 2005. Measuring the se­mantic similarity of texts, Proc. of the ACL Work­shop on Empirical Modeling of Semantic Equiva­lence and Entailment, p.13-18, June 30-30, 2005, Ann Arbor, Michigan</p><p>Z. Dong and Q. Dong. 2006. HowNet and the Compu­tation of Meaning. World Scientific Publishing Co. Inc., River Edge, NJ, USA.</p><p>D. K. Evans and J. L. Klavans. 2003. A Platform for Multilingual News Summarization, Technical Re­port. Department of Computer Science, Columbia University.</p><p>A. K. Farahat, M. S.Kamel.2010. Statistical semantic for enhancing document clustering. Knowledge and Information Systems.</p><p>A. Hotho, S. Staab,G. Stumme. 2003. WordNet im­proves text document clustering. Proc. of SIGIR2003 semantic web workshop.ACM, New York, pp. 541-544.</p><doubt alpha="66.7" length="138" tooSmall="False" monospace="0.0">A. Islam and D. Inkpen. 2006. Second order co­occurrence PMI for determining the semantic simi­larity of words. Proc. LREC'2006: 1033-1038</doubt><p>A. Islam and D. Inkpen. 2008 Semantic text similarity using corpus-based word similarity and string simi­larity, ACM Transactions on Knowledge Discov­ery from Data (TKDD) v.2 (2), pp.1-25, July</p><doubt alpha="0.0" length="4" tooSmall="False" monospace="0.0">2008</doubt><p>T. K. Landauer and S. T. Domais. 1997. A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction and Representa­tion   of  Knowledge.   Psychological Review.</p><p>104(2):211-240.</p><p>T. Landauer, P. W. Foltz and D. Laham. 1998. Intro­duction to Latent Semantic Analysis. Discourse Processes 25: 259-284.</p><p>T. Leek, H. Jin, S. Sista, and R. Schwartz. 1999. The BBN cross-lingual topic detection and tracking system. Proc. of TDT' 1999.</p><p>Y. Li, J. Shawe-Tylor, 2007, Advanced learning algo­rithms for cross-language patent retrieval and clas­sification. Information Processing and Manage­ment v.43(5), pp 1183-1199, September, 2007.</p><p>Q Liu, S Li. 2002.Word similarity computing base don Hownet. Computational Linguistics and Chi­nese Language Processing. (in Chinese)</p><p>B. Mathieu, R. Besancon and C. Fluhr. 2004. Multi­lingual Document Clusters Discovery. Proc. of</p><p>RIAO'2004: 1-10.</p><p>F. J. Och, H. Ney. 2000. Improved Statistical Align­ment Models. Proc. of the 38th Annual Meeting of the Association for Computational Linguistics:</p><p>440-447.</p><p>B. Pouliquen, R. Steinberger, C. Ignat, E. Käsper, I. Temnikova. 2004. Multilingual and cross-lingual news topic tracking. Proc. of COLING'2004:959-965.</p><p>P. Resnik. 1999. Semantic similarity in a taxonomy: An information based measure and its application to problems of ambiguity in natural language. Journal of Artificial Intelligence Research, V.11:95-130.</p><p>R. Rosenfeld. 1996. A maximum entropy approach to adaptive statistical language modeling. Computer speech and language. 10:187-228.</p><doubt alpha="64.6" length="48" tooSmall="False" monospace="0.0">P. D. Turney. 2001. Mining the Web for Synonyms:</doubt><p>PMI-IR versus LSA on TOEFL. Proc. of ECML'2001: 491-502.</p><p>E. M. Voorhees. 1986. Implementing Agglomerative Hierarchic Clustering Algorithms for Use in Doc­ument Retrieval. Information Processing and Man­agement, 22(6): 465-76.</p><doubt alpha="53.1" length="49" tooSmall="False" monospace="0.0">C-P. Wei, C. C. Yang and C-M. Lin. 2008. A Latent</doubt><p>Semantic Indexing Based Approach to Multilin­gual Document Clustering. Decision Support Sys­tem. 45(3):606-620.</p><doubt alpha="46.7" length="45" tooSmall="False" monospace="0.0">S. K. M. Wong, W. Ziarko, P. C. N. Wong. 1985</doubt><p>Generalized vector model in information retrieval. Proc. of the 8th ACM SIGIR:18-25</p><p>Y. Xia, T. Zhao, and P. Jin. 2011. Measuring Chi­nese-English Cross-lingual Word Similarity with HowNet and Parallel  Corpus.  Proc.  of CI-</p><p>Cling'2011(II):221-233.</p><p>D. Yogatamaan, K.Tanaka. 2009. Multilingual Spec­tral Clustering Using Document Similarity Propa­gation. Proc. of EMNLP'2009: 871-879.</p></references></body></article>