<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1" global="434"/><title>A Semantic Feature for Relation Recognition Using a Web-based Corpus</title><author surname="Hung" givenname="Chen-Ming"><org  name="Academia Sinica" country="Taiwan"/></author></firstpageheader><frontmatter><p><b>A Semantic Feature for Relation Recognition Using A Web-based Corpus</b></p><p><b>Chen-Ming Hung</b></p><p>Institute of Information Science Academia Sinica, Taipei, Taiwan rglly@iis.sinica.edu.tw</p></frontmatter><abstract>Selecting appropriate features to represent an entity pair plays a key role in the task of relation recognition. However, existing syntactic features or lexical features cannot capture the interaction between two enti­ties because of the dearth of annotated rela­tional corpus specialized for relation recog­nition. In this paper, we propose a seman­tic feature, called the <i>latent topic feature, </i>which is topic-based and represents an en­tity pair at the semantic level instead of the word level. Moreover, to address the prob­lem of insufficiently annotated corpora, we propose an algorithm for compiling a train­ing corpus from the Web. Experiment results demonstrate that <i>latent topic features </i>are as effective as syntactic or lexical features. Moreover, the Web-based corpus can resolve the problems caused by insufficiently anno­tated relational corpora. </abstract></header><body><section number="1" title="INTRODUCTION"><p>Relation recognition is a challenging task because finding appropriate features to represent the rela­tionship between two entities is difficult and limited by the scarcity of annotated corpora. Prior works on relation recognition have focused on syntactic features, e.g., parsing trees (Culotta and Sorensen, 2004; Zelenko et al., 2003), and on lexical fea­tures, e.g., Part-Of-Speech (POS) features. These approaches show that syntactic features and lexical features outperform bag-of-words (BOW) on exist­ing annotated corpora such as the RDC corpus of the</p><p>ACE project. The superior performance achieved by syntactic and lexical features is due to their ability to capture the grammatical relations between two entities and the characteristics of the entities. For example, (Culotta and Sorensen, 2004) add hyper-nyms of entities to features derived from WordNet. However, neither syntactic nor lexical features can capture the interaction between two entities at the semantic level.</p><p>Another issue in the task of relation recognition is insufficiently annotated corpora. For example, given a pair <i>{the U.N. body, Kosovo}, </i>we can only find three sentences containing both entities in the RDC corpus, which is commonly used corpus in the relation recognition task. The problem of an in­sufficiently annotated corpus biases feature vectors and distorts the prediction of entity pairs. How­ever, (Huang et al., 2004; Hung and Chien, 2007) have shown that the Web can be used as an alterna­tive source of documents related to a given query. That is possibly because of the increasing size of the Web and the efficiency in commercial search en­gines, e.g., Google and Yahoo!.</p><p>To resolve the above problems, we propose a se­mantic feature called the <i>latent topic feature, </i>which is extracted by exploiting the <i>Latent Dirichlet Al­location </i>(LDA) algorithm. Unlike syntactic fea­tures or lexical features, <i>latent topic features </i>repre­sent entity pairs as random mixtures of latent topics, where each topic is characterized by a distribution of words. We prove experimentally that <i>latent topic features </i>are as effective as syntactic features or lexi­cal features in capturing the interaction between two entities. The experiment results are predictable. In the above <i>{ the U.</i><page local="2" global="435"/><i>N. body, Kosovo} </i>example, it may be difficult to determine the relationship between <i>U.N. body </i>and <i>Kosovo </i>straightforwardly. However, making the right guess about the relationship is eas­ier if <i>the U.N. body </i>is grouped with <i>army </i>and <i>gov­ernment. </i>Therefore, the <i>right guess </i>in this example is <i>management.</i></p><p>To overcome the problems caused by an insuffi­ciently annotated corpus, we exploit the Web as a source of training data for the relation recognition task. Given an entity pair, documents describing the entity pair are extracted from the Web via commer­cial search engines using both entities as the query. In other words, snippets returned from the Web are treated as documents related to the query. Our as­sumption, which has been proved in previously pub­lished works, is that returned snippets can capture the interaction between two entities. After the <i>latent topic features </i>extracted from returned snippets using the Web as the corpus, an SVM classifier is trained as the relation recognition classifier for use in the later experiments.</p><p>The remainder of the paper is organized as fol­lows. in Section 2, we discuss works related to fea­ture selection in the relation recognition task as well as using the Web as a corpus. The concept of <i>la­tent topic features </i>is presented in Section 3. We also explain how we represent a document in the vector space of a <i>latent topic feature. </i>Section 4contains an evaluation of the <i>latent topic feature. </i>We then present our conclusions in Section 5.</p></section><section number="2" title="RELATED WORK"><p>In the field of information extraction (IE), the goal of relation recognition is to find the relationship between two entities. Without considering en­tity detection, relation recognition depends heav­ily on the representation of entity pairs. (Zelenko et al., 2003) showed how to extract relations by computing the kernel functions between the ker­nels of shallow parse trees. The kernels are de­fined over a shallow parse representation of the text and used in conjunction with a Support Vector Ma­chine (SVM) learning algorithm to extract person­affiliation and organization-location relations. (Cu­lotta and Sorensen, 2004) extended this work to es­timate kernel functions between augmented dependency trees, while (Kambhatla, 2004) combined lex­ical features, syntactic features, and semantic fea­tures in a maximum entropy model. However, the semantic features discussed in (Kambhatla, 2004) still focus on the word level instead of the concep­tual level.</p><p>LDA is an aspect model that represents docu­ments as a set of <i>topics </i>instead of a bag-of-words. <i>Latent semantic indexing (LSI) </i>(Deerwester et al., 1990) and <i>probabilistic latent semantic indexing (PLSI) </i>(Hofmann, 1999) are also aspect models and have been widely used in the field of information retrieval. LSI simply assumes that each document is generated from single latent topic, while PLSI attempts to relax the assumption by using a mix­ture of latent topics for each document. However, PLSI is highly dependent on training documents; in other words, it cannot handle the probability of la­tent topics in a previously unseen document. In ad­dition, the number of parameters that must be esti­mated in PLSI grows linearly with the number of training documents. (Blei et al., 2003; Blei and Jordan, 2003) proposed LDA to resolve the above-mentioned limitations. It can easily generate an un­seen document under controllable parameters.</p><p>A number of works, e.g., (Huang et al., 2004), have investigated using the Web to acquire a train­ing corpus or acquire additional information not pro­vided by existing annotated corpora. (Huang et al., 2004) exploited the Web as a training corpus to train a classifier with user-defined categories. However, it is widely recognized that when using documents on the Web users must spend a great deal of time filtering out unrelated contents. (Hung and Chien, 2007) designed a bootstrapping method that adapts an existing corpus with an automatic verification al­gorithm in order to control the quality of returned snippets in each iteration. (Matsuo et al., 2006) used the Web to construct a social network system, called <i>POLYPHONET, </i>which visualizes the relationship between two personal names.</p></section><section number="3" title="LATENT TOPIC FEATURE"><p>In this section, we introduce the concept of using the Web to augment an insufficiently annotated corpus for relation recognition. Then we apply the LDA algorithm to the corpus to extract the <i>latent topic</i><page local="3" global="436"/></p><p><i>features </i>to represent entity pairs in the corpus for relation recognition.</p><figure caption="Figure 1: The framework of the proposed approach."></figure><subsection number="3.1" title="Compiling a Web-based Relational Corpus"><p>For an entity pair, <i>E = {ei,e2}, </i>where e<b>i </b>and <i>e2</i>are named entities, it is difficult to find sufficient sentences to describe their relationship from exist­ing annotated corpora. In other words, given an en­tity pair without a relation label, users cannot rec­ognize the pair. Even with a widely used thesaurus, like WordNet, we can only obtain hypernyms or syn­onyms of given entities. It is not possible to obtain knowledge about the interaction between two enti­ties.</p><p>To capture the interaction between two entities, we send both entities, <i>ei AND e2, </i>to commer­cial search engines and collect returned snippets as training documents for an entity pair, <i>E. </i>Snippets of returned search results are defined as the sur­rounding contexts of queries highlighted by com­mercial search engines. In other words, the full texts of search results are not considered in the collected corpus when filtering noisy information in full documents. Let <i>R </i>be the relation label of entity pairs <i>{Ei}..., EM}; </i>then, the training cor­pus for <i>R </i>is the collection of all returned snippets for <i>{Ei}..., EM}. </i>Through effective commercial search engines such as Google and Yahoo!, sen­tences describing the interaction between two enti­ties can easily be retrieved. Returning to the ex­ample <i>{the U.N. body, Kosovo}, </i>almost two mil­lion sentences with co-occurrences of the two en­tities are retrieved by Google. Another advantage of using the Web to retrieve relevant documents is the <i>auto — correction </i>ability of commercial search en­gines. The feature can correct a misspelled query or replace an uncommon word with a synonym or a common word which is correct, so that more related information about entity pairs can be retrieved from the Web as returned snippets. For example, Google can automatically link <i>the U.N. body </i>to <i>United Nations, </i>which is used more frequently in search­ing. Clearly, the number of returned snippets must be considered. Actually, based on experiment re­sults in Section 4, we set the number as five, which achieves the best performance.</p></subsection><subsection number="3.2" title="Modified LDA for the Relational Corpus"><p>LDA is an aspect model with three levels, namely, the corpus level, the document level, and the word level. Given a document, variables of the corpus it belongs to are sampled first, after which the variable of the document is sampled once. Finally, variables for words in the document are sampled.</p><p>For a document <i>d </i>in a corpus <i>D, </i>the modeling process is as follows:</p><p>1. Sample <i>6 ~ Dir(6\a).</i></p><doubt alpha="52.8" length="36" tooSmall="False" monospace="0.0">2. For each wordwnind,n&lt;E{1,..., N}:</doubt><doubt alpha="59.1" length="22" tooSmall="False" monospace="0.0">(a) samplezn~ Mult(6),</doubt><p>(b) sample a word <i>wn ~ p(wn\zn, </i><i>[3)</i><i> </i>from a multivariate Gaussian distribution condi­tioned on the topic <i>z</i><i>n.</i></p><p>Note that <i>o </i>is a vector of corpus-level variables whose dimensionality is equal to the number of la­tent topics; <i>6 </i>is a variable of the document and is as­sumed to follow Dirichlet distribution for the given corpus; and is a word-level variable. In addition, <i>Z = {zi,z2,... ,zN</i><i>} </i>are latent factors that gener­ate the document, and <i>z</i><i>n </i>is the latent topic that <i>w</i><i>n </i>is generated from. Finally, <i>N </i>is the length of the document <i>d.</i></p><p>An entity pair <i>E </i>in the relational corpus is similar to a document <i>d </i>in the text corpus. In other words, the corpus <i>DR </i>is comprised of returned snippets for all entity pairs <i>E</i><i>R </i>with the same relation label R. Therefore, given the parameters <i>o </i>and , we obtain the distribution of entity pair <i>E </i>as follows:</p><doubt alpha="45.0" length="20" tooSmall="False" monospace="0.0">JJ2p(6,zn,SE\a,[)d6,</doubt><doubt alpha="100.0" length="5" tooSmall="False" monospace="0.0">where</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">Ne</doubt><doubt alpha="39.5" length="43" tooSmall="False" monospace="0.0">p(6,zn,SE\a,[) = p(6\a)Y[p(zn\6)p(Wn\zn,3),</doubt><doubt alpha="66.7" length="3" tooSmall="False" monospace="0.0">n=i</doubt><page local="4" global="437"/><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">Ne</doubt><p><i>Ne </i>is the number of words in the returned snippets for E; and <i>wn </i>is the <i>nth </i>word in <i>Se </i>, the returned snippets of <i>E</i>. Table 1 summarizes notations used in the paper.</p><p>_<u>Table 1: Notations used in this paper.</u></p><p><b>SYMBOL DESCRIPTION</b> <i>ER</i><i>Se </i><i>\ER\ </i><i>Ne</i> relation label corpus for <i>R</i> <i>jth </i>entity pair in the relation label <i>R </i>returned snippets for an entity pair <i>E </i>number of entity pairs in the relation <i>R </i>number of words in <i>Se</i><i>nth </i>word in <i>Se</i> latent topic that <i>wn </i>is generated from</p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">R</doubt><doubt alpha="100.0" length="2" tooSmall="False" monospace="0.0">Wn</doubt><p>In Section 3.1, we discussed the advantages of us­ing the Web as a corpus to model entity pairs. In the modeling process, we estimate the probability of <i>wn </i>conditioned on <i>zn, p(wn\zn, [), </i>to maximize the probability of the entire corpus of R. The probabil­ity that we try to maximize is</p><doubt alpha="60.0" length="10" tooSmall="False" monospace="0.0">p(DR\a,ft)</doubt><p>H <i>p(ER\a,P).</i></p></subsection><subsection number="3.3" title="Latent Topic Feature"><p>In different corpora, <i>z </i>obtains a different distribu­tion to maximize the likelihood of the given corpus. In this section, we describe how to exploit <i>z </i>as fea­tures to represent a snippet for an entity pair <i>E, </i>i.e., <i>Se</i>. In Section 3.2, we noted that the parameters to be estimated in the aspect model are all probabilities of words in each latent topic <i>z. </i>Thus, we let the ex­pected number of words generated from latent topics be features of each entity pair. In other words, an en­tity pair <i>E </i>is represented as a feature vector whose length is equal to the number of latent topics and whose <i>ith </i>attribute is equal to where <i>c</i><i>ti </i>is the <i>ith </i>prior Dirichlet parameter.</p><doubt alpha="57.1" length="21" tooSmall="False" monospace="0.0">ai+\Wn\X p(Wn\Zn,ft),</doubt><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">n=1</doubt><p>In addition, because there is no solution good enough to determine the dimensionality of the fea­ture vector or the number of latent topics, we set the number of topics at thirty because it probably min­imize the computation cost without significantly af­fecting the performance.</p></subsection></section><section number="4" title="EXPERIMENTS"><p>In this section, we evaluate the performance of the <i>latent topic feature </i>in representing entity pairs extracted from the Relation Detection and Charac­terization (RDC) corpus of the Automatic Content Extraction 2003 model (ACE 2003)<footnote anchor="1"/>.</p><subsection number="4.1" title="The RDC Corpus"><p>In the RDC corpus, five relation types, <i>AT</i>, <i>NEAR, PART</i>, <i>ROLE</i>, and <i>SOC</i>, are defined; each rela­tion type has extended sub-relations. Table 2 sum­marizes the relations in the RDC corpus for ACE 2003. Based on Table 2, we find that the distribution of the number relations is very unbalanced, ranging from 2 to 773. In the following experiments, we only consider the <i>Role </i>relation because it has the largest numbers of sub-relations and it is easier to verify the recognition results manually. Note that a rela­tion is dropped if it has less than ten sub-relations in order to avoid the bias of learned classifiers. There­fore, the sub-relation <i>founder </i>in <i>Role </i>is dropped in the following experiments because it occurs less than ten times. <i>Other </i>is also dropped because its definition is unclear.</p><table caption="Table 2: Distribution over relation types in the RDC"></table><p>corpus (ACE 2003).</p><footnote label="1">http://projects.ldc.upenn.edu/ace/</footnote><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Relations</b></p></td><td class="cell"><p><b>Sub-Relations(Size)</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>AT</b></p></td><td class="cell"><p>Based-in(78) Residence(186)</p></td><td class="cell"><p>Located(773)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>NEAR</b></p></td><td class="cell"><p>Relative-location(73)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>PART</b></p></td><td class="cell"><p>Part-of(242) Other(2)</p></td><td class="cell"><p>Subsidiary(172)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ROLE</b></p></td><td class="cell"><p>Affiliate-partner(34)</p><p>Citizen-of(93)</p><p>Founder(6)</p><p>Management(294)</p><p>Owner(41)</p></td><td class="cell"><p>Client(33) General-Staff(460) Member(398) Other(98)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>SOC</b></p></td><td class="cell"><p>Associate(25)</p><p>Parent(23)</p><p>Spouse(22)</p><p>Other-Personal(10)</p><p>Other-Professional(88)</p></td><td class="cell"><p>Grandparent(3)</p><p>Sibling(5) Other-relative(24)</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="5" global="438"/></subsection><subsection number="4.2" title="Setting and Measurement"><p>the performance of <i>latent topic features.</i></p><p>We used the package of (Chang and Lin, 2001) to design the following experiments. In addition, v-SVM with a radial kernel function was used to learn the relation classifier. To determine the parameters in <i>v</i>-SVM, i.e., <i>7 </i>and <i>v</i>, we observed the perfor­mance of the <i>v</i>-SVM classifier by randomly select­ing 80% of the sentences in the RDC corpus as train­ing data and the remaining 20% as test data. In other words, we applied five-fold cross validation to build a temporary model for parameter estima­tion. Furthermore, it is well known that parame­ters in the SVM model must be optimized manually; therefore, we estimate <i>v </i>first and then estimate 7. <i>7 </i>is fixed while <i>v </i>is being estimated and vice versa. After estimation, the best result is achieved at the point that <i>7 </i>is equal to 2.5 <i>x </i>10-4 and <i>v </i>is equal to 0.05. We summarize the results in Figure 2. The top graph in Figure 2 is the accuracy curve, where fixed <i>7 </i>= 2.5 <i>x </i>10-4 and flexible <i>v</i>; the bottom graph is the accuracy curve with fixed <i>v </i>= 0.05 and flexible</p><p><i>7.</i></p><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">F</doubt><p>Figure 2: Accuracy of five-fold cross validation us­ing bigram features. Top: <i>v </i>with <i>7 </i>= 2.5 <i>x </i>10-4. Bottom: <i>7 </i>with <i>v </i>= 0.05.</p><p>For each sub-relation in <i>Role, </i>binary classifica­tion is used in the experiments and the F-measure of each sub-relation is used as the metric for assessing</p><doubt alpha="100.0" length="5" tooSmall="False" monospace="0.0">value</doubt><doubt alpha="100.0" length="6" tooSmall="False" monospace="0.0">Recall</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">x</doubt><doubt alpha="33.3" length="3" tooSmall="False" monospace="0.0">T),</doubt><p><i>Precision</i></p><p><i>Precision + Recall \ </i>of correct positive predictions <i>\ </i>of positive examples <i>\ </i>of correct positive predictions <i>\ </i>of positive predictions</p></subsection><subsection number="4.3" title="Web-based Corpus vs. Annotated Corpus"><p>We now evaluate the performance of relational clas­sifier on a Web-based corpus and on an annotated corpus. To assess the performance of on the anno­tated corpus, sentences in the RDC corpus contain­ing a co-occurrence of both given entities were ex­tracted as training data to learn a benchmark relation classifier. On the other hand, the Web-based corpus is compiled from snippets retrieved by using both entities as a query. The <i>latent topic feature </i>is ap­plied on both the Web-based corpus and the RDC corpus using the procedure described in Section 4.2. In addition, to analyze the effect of the number of returned snippets, we increased the number of snip­pets from 3 to 45 in increments of three and then summarized the relationship between the number of returned snippets and the achieved accuracy curve shown in Figure 3. In the figure, the training data is comprised of snippets of information returned by querying 80% of the entity pairs selected at random in the RDC corpus. The test data comprises snippets returned by querying the remaining 20% of entity pairs in the corpus.</p><p>From Figure 3, we observe that using five re­turned snippets for each entity pair achieves the best accuracy (0.85), which is substantially higher than the accuracy achieved by using annotated corpus (0.69). Note that using more returned snippets does not guarantee higher accuracy. For example, when 39 returned snippets are used for each entity pair, the accuracy (0.56) is almost the same as that (0.55) achieved by using only 3 returned snippets. More­over, it is significantly less than the accuracy (0.69) achieved by using the RDC corpus. This is rea­sonable because the greater the number of returned snippets, the larger the amount of noisy information introduced to the classifier, which degrades its per­formance.</p><doubt alpha="100.0" length="5" tooSmall="False" monospace="0.0">ecall</doubt><page local="6" global="439"/><doubt alpha="0.0" length="8" tooSmall="False" monospace="0.0">0.45 0.4</doubt><doubt alpha="89.3" length="28" tooSmall="True" monospace="0.0">Nuimbei of returned snippets</doubt><p>Figure 3: Accuracy of five-fold cross validation us­ing the Web-based corpus and the annotated corpus.</p></subsection><subsection number="4.4" title="Latent Topic Feature vs. Other Features"><p>In this section, we compare the performance of <i>latent topic features </i>with that of syntactic fea­tures and lexical features, i.e., <i>bag</i><i> </i><i>—</i><i> </i><i>of</i><i> — words </i>or <i>parts — </i><i>of</i><i> — speech. </i>Because of the superior performance achieved by using the Web-based cor­pus described in Section 4.3, we extracted features from the training corpus compiled from that corpus rather than the annotated corpus.</p><p>Based on the results reported in Section 4.3, five snippets were returned by the Web-based corpus for each entity pair. For each sub-relation, a one-class SVM was trained to perform binary classification.</p><p>Each sub-relation of <i>Role </i>in Table 3 is applied with binary classification using a one-class SVM. Table 5 summarizes the results of a comparison be­tween the <i>latent topic feature </i>and the features used by (Culotta and Sorensen, 2004). The lat­ter depends on dependency tree kernels, which rep­resent the grammatical dependencies in a sentence and are considered as syntactical features. In Table 5, <i>BOW </i>denotes bag-of-word, <i>sparse </i>represents a sparse kernel, and <i>contiguous </i>represents a contigu­ous kernel.</p><p>Surprisingly, for every sub sub-relation in Table 3, the <i>latent topic feature </i>consistently achieves a significantly higher average recall rate, but a lower average precision rate. This may be due to the <i>latent topic feature's </i>ability to capture informa­tion at the semantic level precisely, but it cannot distinguish the information at the word level eas­ily. In other words, the <i>latent topic feature </i>can capture the common semantic information, probably the <i>Role, </i>of all sub-relations, but it cannot tell the difference between <i>citizen — </i><i>of</i><i> </i>and <i>founder.</i><i> </i>Table 4 shows the results of applying binary classifi­cation to five relations in the RDC corpus. Although the precision rate for each relation is still low, the recall rate has been increased significantly. This de-mostrates the ability of the <i>latent topic feature </i>to capture semantic information.</p><table caption="Table 3: Binary classification results for each sub­relation ofRole._"></table><table caption="Table 4: Binary classification results for each rela­tion in the RDC corpus."></table><p>In Table 5, although the recall rate using the <i>latent topic feature </i>is much higher than that achieved by the other features, unfortunately, the <i>F — score </i>of the <i>latent topic feature </i>cannot be redeemed because of the much lower precision rate. Moreover, the <i>latent topic feature </i>is comparable to the sparse kernel method in a different way be­cause it has a low precision rate but a high recall rate. Finally, the <i>latent topic feature </i>achieves a higher average F-score than the bag-of-words fea­ture, which proves the assumption that the <i>latent topic feature </i>can better capture the interaction be­tween two entities than features at the word level.</p></subsection></section><section number="5" title="CONCLUSION"><p>We have proposed a concept called the <i>latent topic feature </i>for the task of relation recognition and evaluated it on the RDC of the ACE project. The feature captures the interaction between two entities at the semantic level rather than at the word level.<page local="7" global="440"/> Therefore, combining the <i>latent topic feature </i>with syntactic features and lexical features should achieve a better performance than using the features separately. In our future work, we will devise an ap­propriate way of combining <i>latent topic features </i>with syntactical and lexical features.</p><table caption="Table 4: Binary classification results for each relation in the RDC corpus." class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><i>Latent Topic Feature</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>F</b></p></td><td class="cell"><p>1 <b>Prec.</b></p></td><td class="cell"><p>1 <b>Rec.</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Aff.-Part.</b></p></td><td class="cell"><p>0.30</p></td><td class="cell"><p>0.18</p></td><td class="cell"><p>1.00</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Client</b></p></td><td class="cell"><p>0.40</p></td><td class="cell"><p>0.28</p></td><td class="cell"><p>0.71</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Citizen-Of</b></p></td><td class="cell"><p>0.62</p></td><td class="cell"><p>0.47</p></td><td class="cell"><p>0.91</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Gen.-Staff</b></p></td><td class="cell"><p>0.78</p></td><td class="cell"><p>0.64</p></td><td class="cell"><p>0.99</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Manage.</b></p></td><td class="cell"><p>0.56</p></td><td class="cell"><p>0.39</p></td><td class="cell"><p>1.00</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Member</b></p></td><td class="cell"><p>0.62</p></td><td class="cell"><p>0.46</p></td><td class="cell"><p>0.93</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Owner</b></p></td><td class="cell"><p>0.45</p></td><td class="cell"><p>0.29</p></td><td class="cell"><p>0.98</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><i>Latent Topic Feature</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>F</b></p></td><td class="cell"><p>1 <b>Prec.</b></p></td><td class="cell"><p>1 <b>Rec.</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>At</b></p></td><td class="cell"><p>0.61</p></td><td class="cell"><p>0.48</p></td><td class="cell"><p>0.84</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>NEAR</b></p></td><td class="cell"><p>0.36</p></td><td class="cell"><p>0.23</p></td><td class="cell"><p>0.88</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>PART</b></p></td><td class="cell"><p>0.58</p></td><td class="cell"><p>0.46</p></td><td class="cell"><p>0.80</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>ROLE</b></p></td><td class="cell"><p>0.71</p></td><td class="cell"><p>0.64</p></td><td class="cell"><p>0.79</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>SOC</b></p></td><td class="cell"><p>0.59</p></td><td class="cell"><p>0.45</p></td><td class="cell"><p>0.87</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><table caption="Table 5: Comparison betweenLatent topic featureand other features._"></table><p>Because of the lack of a sufficiently annotated corpus for relation corpus for relation recognition, we have also proposed using a Web-based corpus to train classifiers for the purpose. Our experiment results demonstrates that Web documents can accu­rately capture information about the interaction be­tween two named entities in the absence of an an­notated corpus. By using a Web-based corpus, the time cost to manually annotating a corpus for rela­tion recognition is expected to be significantly re­duced if the quality of returned snippetscan be con­trolled.</p><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p></p></td><td class="cell"><p><b>Average</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>F</b></p></td><td class="cell"><p><b>| Prec.</b></p></td><td class="cell"><p><b>Rec.</b></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>Latent Topic</i></p></td><td class="cell"><p>0.58</p></td><td class="cell"><p>0.45</p></td><td class="cell"><p>0.84</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Sparse</b></p></td><td class="cell"><p>0.59</p></td><td class="cell"><p>0.83</p></td><td class="cell"><p>0.46</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Contiguous</b></p></td><td class="cell"><p>0.62</p></td><td class="cell"><p>0.85</p></td><td class="cell"><p>0.49</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>BOW</b></p></td><td class="cell"><p>0.52</p></td><td class="cell"><p>0.73</p></td><td class="cell"><p>0.40</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Sparse+BOW</b></p></td><td class="cell"><p>0.62</p></td><td class="cell"><p>0.80</p></td><td class="cell"><p>0.50</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><b>Cont.+BOW</b></p></td><td class="cell"><p>0.63</p></td><td class="cell"><p>0.81</p></td><td class="cell"><p>0.52</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table></section><references><p>David M. Blei and Michael I. Jordan. 2003. Model­ing annotated data. In <i>Proceedings of the 26th SIGIR, </i>pages 127-134. ACM Press.</p><p>David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. <i>J. Mach. Learn. Res., </i>3:993-1022.</p><p>Chih-Chung Chang and Chih-Jen Lin, 2001. <i>LIBSVM: a library for support vector machines.</i></p><p>A. Culotta and J. Sorensen. 2004. Dependency tree ker­nels for relation extraction. <i>Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics.</i></p><p>Scott C. Deerwester, Susan T. Dumais, Thomas K. Lan­dauer, George W. Furnas, and Richard A. Harshman. 1990. Indexing by latent semantic analysis. <i>Jour­nal of the American Society of Information Science, </i>41(6):391-107.</p><p>Thomas Hofmann. 1999. Probabilistic latent semantic analysis. In <i>Proc. of Uncertainty in Artificial Intelli­gence, UAI'99, </i>Stockholm.</p><p>Chien-Chung Huang, Shui-Lung Chuang, and Lee-Feng Chien. 2004. Liveclassifier: creating hierarchical text classifiers through web corpora. In <i>WWW '04: Pro­ceedings of the 13th international conference on World Wide Web, </i>pages 184-192, New York, NY, USA.</p><p>Chen-Ming Hung and Leeq-Feng Chien. 2007. Web-based text classification in the absence of manually la­beled training documents. <i>J. Am. Soc. Inf. Sci. Tech-nol., </i>58(1):88-96.</p><p>Nanda Kambhatla. 2004. Combining lexical, syntactic, and semantic features with maximum entropy models for information extraction. In <i>The Companion Volume to the Proceedings of42st Annual Meeting ofthe Asso-ciationfor Computational Linguistics, </i>pages 178-181, Barcelona, Spain, July. Association for Computational Linguistics.</p><p>Yutaka Matsuo, Junichiro Mori, Masahiro Hamasaki, Keisuke Ishida, Takuichi Nishimura, Hideaki Takeda, Koiti Hasida, and Mitsuru Ishizuka. 2006. Poly­phonet: an advanced social network extraction system from the web. In <i>WWW '06: Proceedings of the 15th international conference on World Wide Web, </i>pages 397-406, New York, NY, USA.</p><p>Dmitry Zelenko, Chinatsu Aone, and Anthony Richardella. 2003. Kernel methods for relation extraction. <i>J. Mach. Learn. Res., </i>3:1083-1106.</p></references></body></article>