<?xml version="1.0"?><!DOCTYPE article SYSTEM "/project/take/software/searchbench_offline_processing/paperxml_generator/aclextractor/src/python/../resource/dtd/paperxml.dtd"><article><header><firstpageheader><page local="1"/><title>Automatic Discovery of Attribute Words from Web Documents</title><pubinfo>C A R. Dale et al. (Eds.): IJCNLP 2005, LNAI 3651, pp. 106-118,2005. © Springer-Verlag Berlin Heidelberg 2005</pubinfo><author surname="Tokunaga" givenname="Kosuke"><org  name="Japan Advanced Institute of Science and Technology" country="Japan" city="Nomi"/></author><author surname="Kazama" givenname="Jun'ichi"><org  name="Korea Institute of Science and Technology" country="Korea" city="Seoul"/></author><author surname="Torisawa" givenname="Kentaro"><org  name="Nara Institute of Science and Technology" country="Japan" city="Nara"/></author></firstpageheader><frontmatter><p><b>Automatic Discovery of Attribute Words from Web Documents</b></p><p>Kosuke Tokunaga, Jun'ichi Kazama, and Kentaro Torisawa</p><p>Japan Advanced Institute of Science and Technology (JAIST), Asahidai 1-1, Nomi, Ishikawa, 923-1292 Japan <b>{kosuke-t, kazama, torisawa}@jaist.ac.jp</b></p></frontmatter><abstract><b>Abstract. </b>We propose a method of acquiring attribute words for a wide range of objects from Japanese Web documents. The method is a simple unsupervised method that utilizes the statistics of words, lexico-syntactic patterns, and HTML tags. To evaluate the attribute words, we also es­tablish criteria and a procedure based on question-answerability about the candidate word. </abstract></header><body><section number="1" title="Introduction"><p>Knowledge about how we recognize objects is of great practical importance for many NLP tasks. Knowledge about <i>attributes, </i>which tells us from what view­points objects are usually understood or described, is one of such type of knowl­edge. For example, the attributes of <i>car </i>objects will be <i>weight, engine, steering wheel, driving </i>feel,and <i>manufacturer. </i>In other words, attributes are items whose values we want to know when we want to know about the object. More analyti­cally, we tend to regard <i>A </i>as an attribute for objects of class <i>C </i>when <i>A </i>works as if function <i>v </i>= A(o),o <i>G C </i>where <i>v </i>is necessary to us to identify <i>o </i>(especially to distinguish <i>o</i><i> </i>from <i>o'</i>(= o) <i>G</i><i> </i>C). Therefore, obvious applications of attributes are ones such as summarization [1,2] and question-answering [3]. Moreover, they can be useful as features in word clustering [4] or machine learning. Although the knowledge base for attributes can be prepared manually (e.g., WordNet [5]), problems are cost and coverage. To overcome these, we propose a method that automatically acquires attribute knowledge from the Web.</p><p>To acquire the attributes for a given class, <i>C </i>(e.g., <i>car), </i>the proposed method first downloads documents that contain class label <i>C </i>(e.g., "car") from the Web.1We extract the candidates of attribute words from these documents and score them according to the statistics of words, lexico-syntactic patterns, and HTML tags. Highly scored words are output as attributes for the class. Lexico-syntactic patterns and other statistics have been used in other lexical knowledge acquisi­tion systems [3,4,6,7,8]. We specifically used lexico-syntactic patterns involving the Japanese postposition "no" as used in [8] such as "C no A"where <i>A </i>is an attribute word, which is almost equivalent to pattern <i>"A </i>of <i>C</i>"usedin[7]to<page local="2"/></p><footnote label="1">We use to denote both the class and its class label (the word representing the class). We also use to denote both the attribute and the word representing it.</footnote><p>Automatic Discovery of Attribute Words from Web Documents 107</p><p>find part-whole relations. Novel features of our method are its use of Web search engines to focus on documents highly relevant to the class and its use of statistics concerning attribute words and surrounding HTML tags.</p><p>One of the difficulties in studying attribute knowledge is that there are no standard definitions of attributes, or criteria for evaluating obtained attributes. In this paper, we propose a simple but effective definition of attributes that matches our motivation and applications, i.e., whether we can ask a question about the attribute and whether there is an answer to that question <i>(question answerability</i>). For example, one can ask as "Who is the manufacturer of this car?", and someone might answer "Honda", because we want to know the <i>manu­facturer </i>when we concerned about cars. We designed a procedure for evaluating attributes based on this idea. As the literature points out [9,10], attributes can include many types of relations such as property (e.g., <i>weight), </i>part-of (e.g., <i>engine), </i>telic (e.g., <i>driving feel), </i>and agentive (e.g., <i>manufacturer). </i>However, we ignored type distinctions in this study. First, because attributes are useful even if the type is not known, and second, because defining attributes as one of these types and evaluating them only complicates the evaluation process, making the results unstable. The use of linguistic tests to define attributes is not that new. Woods [11] devised a test on whether we can say "The <i>A </i>of <i>o </i>is v." Although we followed this procedure, we focused more on attributes that are important for our understanding of an object by using <i>question-answerability </i>as our criterion.</p></section><section number="2" title="Acquisition Method"><subsection number="2.1" title="Basic Observations on Attributes"><p>Our method is based on the following three observations.</p><p>1. Attributes tend to occur in documents that contain the class label and not in other documents.</p><p>2. Attributes tend to be emphasized by the use of certain HTML tags or occur as items in HTML itemizations or tables in Web documents.</p><p>3. Attributes tend to co-occur with the class label in specific lexico-syntactic patterns involving the postposition "no."</p></subsection><subsection number="2.2" title="Extraction of Candidate Words"><p>To acquire the attributes of class <i>C</i>, we first download documents that contain class label <i>C </i>using a Web search engine, according to the first observation. We refer to this set of documents as a <i>local document set (LD(C</i>)). All the nouns appearing in the local document set are regarded as candidates of attribute words. Here, the nouns are words tagged as "proper nouns", "sahen nouns" (nouns that can become averb withthe suffix "suru"), "location", or "unknown" (e.g., words written in katakana) by a Japanese morphological analyzer, JUMAN [12]. Note that we restricted ourselves to single word attributes in this study. The obtained candidate words are scored in the next step.</p><page local="3"/><p>T<b>able 1. </b>Lexico-syntactic patterns for attribute acquisition. (We added possible En­glish translations for the patterns in parenthesis).</p></subsection><subsection number="2.3" title="Ranking of Candidate Words"><p>We rank the candidate words according to a score that reflects the observations described in Sect. 2.1. The overall score takes the following form.</p><doubt alpha="35.8" length="53" tooSmall="False" monospace="0.0">V(C,A)=n(C, A)• f (C, A) • t(C, A) • dftdf (C, A),(1)</doubt><p>where <i>A </i>is the candidate word to be scored and <i>C </i>is the class. n(C, A)and <i>f (C, </i>A) are scores concerning lexico-syntactic patterns. t(C, A) is a score con­cerning the statistics of HTML tags to reflect the second observation. Finally, <i>dfidf (C, </i>A) is the score related to word statistics. This reflects the first obser­vation. By multiplying these sub-scores, we expect that they will complement each other. We will explain the details on these sub-scores in the following.</p><p>As previously mentioned, we use lexico-syntactic patterns including the Japa­nese postposition "no" as clues. The patterns take the form "C no <i>APOST" </i>where <i>POST </i>is a Japanese postposition or a punctuation mark.2 The actual patterns used are listed in Table 1. Score n(C, A)is the number of times <i>C </i>and <i>A </i>co-occur in these patterns in the local document set <i>LD(C).</i></p><p>Score <i>f (C, A) </i>requires more explanation. Roughly, <i>f (C, </i>A)is the number of times <i>C </i>and <i>A </i>co-occur in the patterns without the last postposition (i.e., pat­tern <i>"C </i>no <i>A") </i>collected from 33 years of parsed newspaper articles.3 Note that pattern matching was done against the parsed dependency structures.4 The rea­son this score was used in addition to n(C, A) was to obtain more reliable scores by increasing the number of documents to be matched. This may sound contra­dictory to the fact that the Web is the largest corpus in the world. However, we found that we could not obtain all the documents that contained the class label because existing commercial Web search engines return URLs for a very small fraction of matched documents (usually up to about 1,000 documents). Although we could use hit counts for the patterns, we did not do this to avoid overloading the search engine (each class has about 20,000 candidate words).</p><p>Score t(C, A)is the number of times <i>A </i>appears in <i>LD(C</i>) surrounded by HTML tags. More precisely, we count the number of times <i>A </i>appears in the form: <i>"&lt;tag1 &gt;A&lt;tag2</i>&gt;" where the number of characters between HTML tags<page local="4" global="109"/></p><footnote label="2">Note that there are actually no spaces between words in Japanese. The spaces are for easier understanding.</footnote><footnote label="3">Yomiuri newspaper 1987-2001, Mainichi newspaper 1991-1999, and Nikkei newspa­per 1983-1990; 3.01 GB in total. We used a Japanese dependency parser [13] .</footnote><footnote label="4">The differences from n(C, A) were introduced to reuse the existing parsed corpus.</footnote><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">108</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>c</i></p></td><td class="cell"><p>no</p></td><td class="cell"><p><i>A</i><i> </i>ha (A of <i>C [verb])</i></p></td><td class="cell"><p><i>C</i></p></td><td class="cell"><p>no</p></td><td class="cell"><p><i>A</i><i> </i>de (by <i>A</i><i> </i>of <i>C)</i></p></td><td class="cell"><p><i>C</i><i> </i>no <i>A</i><i> </i>e (to A of <i>C)</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>c</i></p></td><td class="cell"><p>no</p></td><td class="cell"><p><i>A </i>ga <i>(A </i>of <i>C [verb])</i></p></td><td class="cell"><p><i>C</i></p></td><td class="cell"><p>no</p></td><td class="cell"><p><i>A </i>made (even/until <i>A </i>of <i>C)</i></p></td><td class="cell"><p><i>C </i>no AA(A of <i>C,)</i></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>c</i></p></td><td class="cell"><p>no</p></td><td class="cell"><p><i>A </i>wo <i>([verb] </i><i>A </i>of <i>C)</i></p></td><td class="cell"><p><i>C</i></p></td><td class="cell"><p>no</p></td><td class="cell"><p><i>A </i>kara (from <i>A </i>of <i>C)</i></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p><i>c</i></p></td><td class="cell"><p>no</p></td><td class="cell"><p><i>A </i>ni (at/in <i>A </i>of <i>C)</i></p></td><td class="cell"><p><i>C</i></p></td><td class="cell"><p>no</p></td><td class="cell"><p><i>A </i>yori (from/than <i>A </i>of <i>C)</i></p></td><td class="cell"><p></p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><doubt alpha="40.0" length="55" tooSmall="False" monospace="0.0">&lt;B&gt;^a-*V—&lt;/B&gt;&lt;BR&gt;MS&lt;BR&gt;S§ß400g,&amp;t~2iS,^4v-y&gt;?jV—2 $C,t&gt;</doubt><p><b>Fig. 1. </b>Example HTML document</p><p>(i.e., the length of A) is 20 at maximum. The tags <i>(&lt;tag1 &gt; </i>and <i>&lt;tag2</i>&gt;)can be either a start tag (e.g., &lt;A&gt;)or anend tag (e.g., &lt;/A&gt;). This score is intended to give high values for words that are emphasized or occur in itemizations or tables.Forexample,in the HTML document in Fig.1, the words <i>V</i><i> </i><i>—</i><i> </i>(Thai-curry)", "MU (ingredient)", (spice)", <i>"^^TVV—,?</i></p><p>(coriander, cumin)", and "ffDytf (recipe)" are counted.</p><p>Finally, <i>dfidf (C, </i>A), which reflects the first observation, is calculated as:</p><p><i>dftdf{C,A) = df{A,LD{C))-idf{A),   </i>id/(A) = log^göj.</p><p>d<i>f</i><i> (A, X </i>) denotes the number of documents where <i>A </i>appears in documents <i>X</i>. <i>G </i>is a large set of randomly collected Web documents, which we call the <i>global document set. </i>We derived this score from a similar score, which was used in [14]to measure the association between a hypernym and hyponyms.</p></subsection></section><section number="3" title="Evaluation Criteria"><p>This section presents the evaluation criteria based on <i>question-answerability </i>(QA tests). Based on the criteria, we designed an evaluation procedure where the evaluators were asked to answer either by yes or no to four tests at maximum, i.e., a hyponymy test (Sect. 3.4), a QA test (Sect. 3.1) and a suffix augmented QA test (Sect. 3.2) followed by a generality test (Sect. 3.3).</p><subsection number="3.1" title="Question-Answerability Test"><p>By definitions we used, attributes are what we want to know about the object. Therefore, if <i>A </i>is an attribute of objects of class <i>C, </i>we can arrange questions (consisting of <i>A </i>and <i>C) </i>that require the values for <i>A </i>as the answer. Then someone should be able to answer the questions. For example, we can ask "Who is the director of this movie?" because <i>director </i>is an attribute of <i>movie</i>.Theanswer might be someone such as "Stanley Kubrick." We designed the QA test shown in Fig. 2 to assess the correctness of attribute <i>A </i>for class <i>C </i>based on this criterion. Several points should be noted. First, since the value for the attribute is actually defined for the object instance (i.e., <i>v </i>= A(o),o <i>G </i>C), we should qualify class label <i>C </i>using "kono (this)" to refer to an object instance of class C.</p><p>Second, since we cannot know what question is possible for <i>A </i>beforehand, we generate all the question types listed in Fig. 2 and ask whether any of them are acceptable.</p><page local="5" global="110"/><p>Are any of the following questions grammatically correct, natural, and answerable?</p><p>1. r© <i>C © A </i>«M?(kono <i>C </i><i>no </i><i>A </i>ha nani?/What is the <i>A </i>of this <i>C</i>?) 2. r© <i>C © A </i>fàft?(kono <i>C </i>no <i>A </i><i>ha </i>dare?/Who is the <i>A </i>of this C?) 3. r© <i>C © A </i>Ö^O?(kono <i>C </i>no <i>A </i>ha itu?/When is the <i>A </i>of this C?) 4. r© <i>C © A </i>Ö^r ?(kono <i>C </i>no <i>A </i>ha doko?/Where is the <i>A </i>of this C?) 5. r© <i>C © A </i>Ö^n?(kono <i>C </i>no <i>A </i>ha dore?/Which is the <i>A </i>of this C?) 6. r© <i>C © A </i>Ö^&lt;0?(kono <i>C </i>no <i>A </i>ha ikutu?/Howmanyisthe <i>A </i>of this C?) 7. r© <i>C</i><i> </i><i>©</i><i> </i><i>A</i><i> </i><i>Ö'cj</i><i> </i>?(kono <i>C</i><i> </i>no <i>A</i><i> </i>ha dou?/How much is the <i>A</i><i> </i>of this <i>C</i><i> </i>?) <b>Fig.</b><b> 2. </b>Question-answerability Test</p><p>Third, the question should be <i>natural </i>as well as grammatically correct. Nat­uralness was explained to the evaluators as positively determining whether the question can be their first choice in usual conversations. In our point of view, at­tributes should be important items for people in describing objects. We assumed that attributes that conformed to the naturalness criterion would be such impor­tant attributes. For example, <i>stapler </i>is not an attribute of <i>company </i>in our sense, although almost all companies own <i>stapler</i>s. Our naturalness criterion can re­flect this observation since the question "What is the stapler of this company?" is unnatural as a first question when talking about a company, and therefore we can successfully conclude that <i>stapler </i>is not an attribute. Note that Woods' linguistic test [11] (i.e., whether "the attribute of an object is a value" can be stated or not) cannot reject <i>stapler </i>since it does not have the naturalness re­quirement (e.g., we can say "the stapler of [used by] SONY is Stapler-X").5 In addition, note that such importances can be assessed more easily in the QA test, since questioners basically ask what they think is important at least at the time of utterance. However, we cannot expect such an implication even though the declarative sentence is acceptable.</p><p>Finally, the answer to the question does not necessarily need to be written in language. For example, values for attributes such as <i>map, picture</i>,and<i>blueprint </i>cannot be written as language expressions but can be represented by other media. Such attributes are not rare since we obtain attributes from the Web.</p></subsection><subsection number="3.2" title="Suffix Augmented QA Test"><p>Some attributes that are obtained can fail the QA test even if they are correct, especially when the surface form is different from the one they actually mean. This often occurs since Japanese is very elliptic and our method is restricted to single word attributes. For example, the word seito (students) can be used to represent the attribute seito suu (number of students) as in the sentence below.</p><doubt alpha="44.7" length="47" tooSmall="False" monospace="0.0">kono     gakko    no      seito        ha500nin</doubt><doubt alpha="56.3" length="103" tooSmall="False" monospace="0.0">this      school   of       students   is       500 NUM (The number of students of this school is 500.)</doubt><footnote label="5">Stapler might be an important attribute of companies for stationery sellers. However, we focus on attributes that are important for most people in most situations.</footnote><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">110</doubt><page local="6" global="111"/><p>Automatic Discovery of Attribute Words from Web Documents 111</p><doubt alpha="66.7" length="48" tooSmall="False" monospace="0.0">—WC(number of)TjSc(method for)%j(name of)^3(-er)</doubt><p><b>— </b>H#f!5 ([amount of] time of) H#g!l (time of)    Bf83 (period of) <i>MPft</i><i> </i>(location of) <b>— </b>iîïîSf (amount of money for) SBl (degree of)    Hl=t (state of) <b>— </b>©~c&gt; (nominalized adjectives e.g., "height of" "prettiness of")</p><p><b>Fig. 3. </b>Allowed augmentation</p><p>These attributes whose parts are elided (e.g., seito representing seito suu)are also useful since they are actually used in sentences as in the above example. Therefore, they should be assessed as correct attributes in some way. Although the most appropriate question for seito representing seito suu is (6) in Fig. 2, it is unfortunately ungrammatical since ikutu cannot be used for the number of persons. Therefore, seito representing seito suu will fail the QA test.6</p><p>In Japanese, most of the elided parts can be restored by adding appropriate suffixes (as "suu" (number of) in the previous example) or by adding "no" + nominalized adjectives. Thus, when the attribute word failed the first QA test, we asked the evaluators to re-do the QA test by choosing an appropriate suffix or a nominalized adjective from the list of allowed augmentations and adding it to the end of the evaluated word. Figure 3 lists the allowed augmentations.7'8</p></subsection><subsection number="3.3" title="Generality Test"><p>Although our primal aim was to acquire the attributes for a given class, i.e., , to find attributes that are common to all the instances of the class, we found, in preliminary experiments, that some uncommon (but interesting) attributes were assessed as correct according to the QA test depending on the evaluator. An example is <i>subtitle </i>for the class <i>movie. </i>Strictly speaking, <i>subtitle </i>is not an attribute of all movies, since all movies do not necessarily have subtitles. For example, only foreign films have subtitles in Japan. However, we think this attribute is also useful in practice for people who have a keen interest in foreign films. Thus, the evaluators were asked whether the attribute was common for most instances of the class when the attribute was judged to be correct in the QA test. We call attributes that passed this generality test <i>general attributes, </i>and those that failed but passed the QA test <i>relaxed attributes </i>(note that general attributes is a subset of relaxed attributes). We compare the accuracies for the relaxed and general attributes in the experiments.</p><footnote label="6">Seito (representing students ) might pass the QA test with question type (2) in Fig. 2. However, this is not always the case since some evaluators will judge the question to be unnatural.</footnote><footnote label="7"> Postposition "no (of)" before the suffix is also allowed to be added if it makes the question more natural.</footnote><footnote label="8"> The problem here might not occur if we used many more question types in the first QA test. However, we did not do this to keep the first QA test simple. With the same motivation, we kept the list of allowed suffixes short (only general and important suffixes). The uncovered cases were treated by adding nominalized adjectives.</footnote><page local="7" global="112"/><p>Finally, we should note that we designed the evaluation procedure so that the evaluators could be asked whether candidate <i>A </i>is a hyponym of <i>C </i>before the QA tests. If <i>A </i>is a hyponym of <i>C, </i>we can skip all subsequent tests since <i>A </i>cannot be an attribute of <i>C. </i>We added this test because the output of the system often contains hyponyms and these tend to cause confusion in the QA tests since expression "C no A" is natural even when <i>A </i>is a hyponym of <i>C </i>(e.g., "anime no Dragon Ball (Dragon Ball [of/the] anime)").</p></subsection></section><section number="4" title="Experiments"><subsection number="4.1" title="Experimental Setting"><p>We first selected 32 word classes from 1,589 classes acquired from the Web with an automatic hypernym-hyponym acquisition method [14]. Here, we regarded the hypernym as the class label. Since our purpose was just to evaluate our method for classes from the Web, we selected classes that were obtained successfully. We randomly chose the 22 classes listed in Table 2 for human evaluation from these 32 classes.9 The hyponyms were used to help the evaluators to disambiguate the meaning of class labels (if ambiguity existed).</p><p>To collect <i>LD(C), </i>we used the Web search engine goo (http://www.goo.ne.jp). The size of <i>LD(C) </i>was 857 documents (URLs) on class average. There were about 20, 000 candidate words on class average. As global document set <i>G </i>re­quired for the calculation of <i>dfidf (C, </i>A), we used 1.0 <i>x </i>10<footnote anchor="6"/> randomly downloaded Web documents.</p><p><b>Table 2. </b>Classes used in evaluation</p><doubt alpha="65.6" length="288" tooSmall="False" monospace="0.0">ffirfl (city), Ifttlf (museum),MB(national holiday), WS (police),MWt(facility),±¥(university),ftn(newspaper), (garbage), tttt (shrine), Ä (bird), ÉiK; (hospital), fit® (plant), 111 (river), h ^|5£ (elementary school), Ö (music tune), EHHf (library), ^c/dij (branch office), (web site), UU</doubt><doubt alpha="57.7" length="52" tooSmall="False" monospace="0.0">(town), -fey-)" — (sensor), WO (training),SttW (car)</doubt><p>We output the top 50 attributes for each class ranked with our proposed method and with alternative methods that were used for comparison. We gath­ered outputs for all the methods, removing duplication (i.e., taking the set union) to achieve efficient evaluation, and re-sorted them randomly to ensure that the assessment was unbiased. Four human evaluators assessed these gathered at­tributes class-by-class in four days using a GUI tool implementing the evaluation procedure described in Sect. 3. There were a total of 3, 678 evaluated attributes. Using the evaluation results, we re-constructed the evaluations for the top 50 for each method. The kappa value [15], which indicates inter-evaluator agreement, was 0. 533 for the general attribute case and 0.593 for the relaxed attribute case. According to [15], these kappa values indicate "moderate" agreement.</p><footnote label="9">This selection was due to time/cost limitations.</footnote><doubt alpha="51.9" length="77" tooSmall="False" monospace="0.0">112       3.4   Hyponymy Test12       K. Tokunaga, J. Kazama, and K. Torisawa</doubt><page local="8" global="113"/><p>Automatic Discovery of Attribute Words from Web Documents 113 <b>4.2   Accuracy of Proposed Method</b></p><p>Figure 4 has accuracy graphs for the proposed method for relaxed attributes. The graph on the left shows per-evaluator precision when the top (repre­sented by <i>x </i>axis ) attributes were output. The precision is the average over all classes. Although we cannot calculate the actual recall, the <i>x </i>axis corresponds to approximate recall. We can see that ranking with the proposed method has a positive correlation with human evaluation, although the assessments varied greatly depending on the evaluator. The graph on the right shows curves for average (with standard deviation), 3-consensus, and 4-consensus precision. 3-consensus (4-consensus) is precision where the attribute is considered correct by at least three (four) evaluators. Figure 5 has graphs for the general attribute case the same as for the relaxed case. Although there is a positive correlation between ranking with the proposed method and human evaluators, the precision was, not surprisingly, lower than that for the relaxed case. In addition, the lower kappa value (0.533 compared to 0.593 for the relaxed case) indicated that the generality test was harder than the QA tests.</p><p>The accuracy of the proposed method was encouraging. Although we cannot easily determine which indicator is appropriate, if we use the majority rule (3-</p><doubt alpha="0.0" length="4" tooSmall="True" monospace="0.0">0.95</doubt><doubt alpha="0.0" length="8" tooSmall="True" monospace="0.0">0.9 0.85</doubt><doubt alpha="0.0" length="8" tooSmall="True" monospace="0.0">0.8 0.75</doubt><doubt alpha="0.0" length="8" tooSmall="True" monospace="0.0">0.7 0.65</doubt><doubt alpha="0.0" length="3" tooSmall="True" monospace="0.0">0.6</doubt><doubt alpha="0.0" length="4" tooSmall="True" monospace="0.0">0.55</doubt><doubt alpha="70.4" length="27" tooSmall="True" monospace="0.0">Evaluatorl — -*— Evaluator2</doubt><doubt alpha="66.7" length="15" tooSmall="True" monospace="0.0">Evaluator3■—-o—</doubt><doubt alpha="75.0" length="12" tooSmall="True" monospace="0.0">Evaluator4 —</doubt><doubt alpha="25.0" length="40" tooSmall="True" monospace="0.0">15     20     25     30 35 Rank (recall)</doubt><doubt alpha="0.0" length="10" tooSmall="True" monospace="0.0">40   45 50</doubt><doubt alpha="0.0" length="1" tooSmall="True" monospace="0.0">1</doubt><doubt alpha="0.0" length="10" tooSmall="True" monospace="0.0">20   25 30</doubt><doubt alpha="76.9" length="13" tooSmall="True" monospace="0.0">Rank (recall)</doubt><doubt alpha="0.0" length="15" tooSmall="True" monospace="0.0">35   40   45 50</doubt><p><b>Fig. 4. </b>Accuracy of relaxed attributes <b>Fig.</b><b> 5. </b>Accuracy of general attributes<page local="9" global="114"/></p><doubt alpha="83.7" length="43" tooSmall="True" monospace="0.0">Evaluator1 Evaluator2 Evaluator3 Evaluator4</doubt><doubt alpha="18.2" length="55" tooSmall="True" monospace="0.0">10    15   20   25   30   35   40   45 50 Rank (recall)</doubt><doubt alpha="0.0" length="1" tooSmall="True" monospace="0.0">5</doubt><doubt alpha="0.0" length="2" tooSmall="True" monospace="0.0">10</doubt><doubt alpha="0.0" length="5" tooSmall="True" monospace="0.0">10 15</doubt><doubt alpha="0.0" length="3" tooSmall="True" monospace="0.0">0.9</doubt><doubt alpha="0.0" length="3" tooSmall="True" monospace="0.0">0.8</doubt><doubt alpha="50.0" length="2" tooSmall="False" monospace="0.0">□I</doubt><doubt alpha="0.0" length="3" tooSmall="True" monospace="0.0">0.7</doubt><doubt alpha="0.0" length="3" tooSmall="True" monospace="0.0">0.5</doubt><doubt alpha="0.0" length="3" tooSmall="True" monospace="0.0">0.4</doubt><p>T<b>able 3. </b>Top 20 attributes of several classes obtained by proposed method</p><p>consensus in our case) employed in [7], the proposed method obtained relaxed attributes with 0.852 precision and general attributes with 0.727 precision for the top 20 outputs. Table 3 lists the top 20 attributes obtained with the proposed method for several classes. The numeral before (after) "/" is the number of evaluators who judged the attribute as correct as a relaxed (general) attribute. We can see that many interesting attributes were obtained.</p></subsection><subsection number="4.3" title="Effect of Scores"><p>In this analysis, we assessed the effect that sub-scores in Eq. (1) had on the acquisition accuracy by observing the decrease in precision when we removed each score from Eq. (1). First, we could observe a positive effect for most scores in terms of the precision averaged over evaluators. Moreover, interestingly, the tendency of the effect was very similar for all evaluators, even though the as­sessments varied greatly depending on the evaluator as the previous experiment showed. Due to space limitations, we will only present the latter analysis here.</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">114</doubt><table class="main" frame="box" rules="all" border="1" regular="False"><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Classes</p></td><td class="cell"><p>Attributes</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>Ä</p><p>(bird)</p></td><td class="cell"><p>(picture)[4/4] £W (name)[4/2] 8* (sort)[4/4] -f9X)- (illustration)[3/3] WIS (characteristics)[4/4] S S (disease)[4/2] ÉÉS (life)[4/4] RB (topic)[3/2] SMS (relation)[0/0] (image)[4/4] S (nest)[4/4] » Sf (song)[4/4] * (shape)[4/4] «S (info.)[4/4] ft# (world)[0/0] f (song)[4/4] »fl (animal)[0/0] ^—i (page)[3/2] ÉÉS (ecology)[4/4] ffl (wing)[4/4]</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(hospital)</p></td><td class="cell"><p>-t. — A^—i (home page)[4/1] ÄS (facility)[3/3] «S (info.)[4/4] MO (intro.)[4/4] (info. desk)[4/4] Sg (authorization)[3/3] £ft (name)[4/2] EW (doctor)[4/4] S#S (psychiatry)[4/2] fffl (reputation)[4/4] fcfjS (handling)[4/4] BR (phone)[2/2] KS (medical care)[4/4] ?êS (treatment)[4/4] ES (medical service)[3/3] B fä (function)[3/3] KÄ (director)[4/4] Ifii (valuation)[4/4] KS (medical examination)[4/4] ^—i (page)[2/2] WS (admin.)[4/3] —gP (part)[1/1]</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>««</p><p>(plant)</p></td><td class="cell"><p>45W (name)[4/2] 8* (species)[4/4] (picture)[4/4] 8? (seed)[4/4] JÄÖ (cultivation)[4/3] SS (observa-tion)[4/3] #ft (characteristics)[4/4] KM (explanation)[4/4] H* (image)[4/4] Mï (surveillance)[4/3] T — 9 (data)[4/4] »ft (evolution)[3/3] (description)[4/4] (list)[2/2] ^ (leaf)[4/3] S# (preserva-tion)[2/2] ftVy (design)[1/1] 4W (growth)[4/4]</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(river)</p></td><td class="cell"><p>7*S (water level)[4/4] ilS (upstream)[4/4] £W (name)[4/2] SS (environment)[4/4] 7*Ä (water qual-ity)[4/4] gS6 (history)[4/4] SS (head stream)[4/4] SÄ (picture)[4/4] * (water)[4/4] *ffi (surface)[4/4] W BÏ (location)[4/4] Sft (current)[4/4] *H (waterside)[4/4] *S (river head)[4/4] (four seasons)[3/3] # ft (characteristics)[4/4] ^ (inside)[1/1] Bï!) (streamside)[4/4]         (nature)[4/4] (babbling)[4/4]</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(elementary school)</p></td><td class="cell"><p>S» (activity)[4/4] BjtOiB* (efforts)[4/3] îlltlè (athletic meeting)[4/4] (child)[4/4] -t. — A^—i (home page)[4/0] éci (head teacher)[4/4] (classroom)[4/4] (school song)[4/4] JMÜ (student)[4/4] U# (school building)[4/4] O (event)[4/4] ¥S (learning)[3/3] fêft (feeding service)[4/3] ^—i (page)[2/2] # W8g (gym)[4/4]          (class)[3/3]             (mail)[0/0]          (grade)[1/1] »SÄ (opening ceremony)[4/4]</p><p>(music)[2/2]</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(music tune)</p></td><td class="cell"><p>EH (lyrics)[4/1] (title)[4/2] S# (performance)[4/4] 5X1 (list)[0/0] ^^—i (image)[4/4] ffH (lyrics writing)[4/1] (musical score)[4/4] £W (name)[4/2] BS (content)[3/3] -i&gt;y;v (genre)[4/4] «S (info.)[4/4] &lt;^yf (point)[4/4] ft# (world)[1/1] ^Of&lt;— (melody)[4/4] Sft (end)[3/2] B45 (title)[4/2] tfl (inside)[0/0] ff8 (composition)[4/4] r — ? (theme)[4/4] T — 9 (data)[4/2]</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(library)</p></td><td class="cell"><p>(source material)[4/4] -t. — A^—i (home page)[4/2] ^—i (page)[3/1] gS6 (history)[4/4] SB (establish-ment)[4/4] -&gt;XrA (system)[4/4] Ä» (book stock)[4/4] 3f— (copy)[2/2] * (book)[4/4] WW (location)[4/4] fOT (use)[4/4] f-—t?X (service)[4/4] r'—9^ — X (database)[4/3] H» (book)[4/4] (newspaper)[4/4] ff SS (close)[4/4] BS (catalog)[3/3] M?K (display)[4/2] ÄS (facility)[2/2] «S (info.)[4/4]</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(town)</p></td><td class="cell"><p>AP (population)[4/4] gS6 (history)[4/4] -t. — A^—i (home page)[4/0] S* (sightseeing)[4/4] «S (info.)[3/3] BR (finance)[4/4] ÄS (facility)[4/4] STftB (heritage)[4/2] SS (environment)[4/4] H* (hot spring)[3/1] RB (topic)[3/2] (four seasons)[3/3] ^&lt;yf (event)[4/3] H»8g (library)[4/3] Xft (culture)[4/4] JÏW (landscape)[4/4]                (symbol)[4/3] S* (industry)[4/3] B* (agriculture)[4/2]         (town council)[3/3]</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>(sensor)</p></td><td class="cell"><p>«S (info.)[4/4] SK (sensitivity)[4/3] 8* (sort)[4/3] SB (position)[4/4] BjtOfttt (install)[4/4] Sft (devel-opment)[4/4] 8K (accuracy)[4/4] »MX (size)[4/4] tfctt (specification)[4/4] HK (temperature)[2/1] T—9 (data)[4/4] tejf (set)[4/4] SB (install)[4/4] BIB (function)[4/4] Kff (technology)[4/4] #Ä (feature)[4/4] ^ — i (page)[3/3] iftS (height)[3/2]         (adoption)[3/3] JSÂ (application)[4/4]</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"><p>WO</p><p>(training)</p></td><td class="cell"><p>BS (content)[4/4] B89 (purpose)[4/4]         (practice)[4/4] r — ? (theme)[4/3] /D^7A (program)[4/4] B W (lecturer)[4/4]         (plan)[4/4] 45ft (name)[4/2] ^ia— (menu)[4/4] SS (report)[4/4] Sft (target)[4/4] (outcome)[4/4] 3ÏÂ (satisfaction)[2/2] W (place/atmosphere)[3/3]            (state of existence)[2/2] Bffl (detail)[4/4] Bè (opportunity)[1/1] SM (capacity)[4/4] SB (participation)[4/4] (other)[0/0]</p></td><td class="cell"></td></tr><tr class="row"><td class="cell"></td><td class="cell"></td><td class="cell"></td><td class="cell"></td></tr></table><page local="10" global="115"/><doubt alpha="76.9" length="13" tooSmall="True" monospace="0.0">Rank (recall)</doubt><doubt alpha="64.2" length="95" tooSmall="True" monospace="0.0">Proposed - pattern (web) ■■­Proposed - pattern (news) &gt;■ Proposed - tag !■■ Proposed - dfidf &lt;■</doubt><doubt alpha="13.0" length="77" tooSmall="True" monospace="0.0">5      10      15     20     25      30     35     40     45 50 Rank (recall)</doubt><doubt alpha="0.0" length="3" tooSmall="True" monospace="0.0">0.1</doubt><doubt alpha="0.0" length="6" tooSmall="True" monospace="0.0">0.05 0</doubt><doubt alpha="0.0" length="5" tooSmall="True" monospace="0.0">-0.05</doubt><doubt alpha="0.0" length="4" tooSmall="True" monospace="0.0">-0.1</doubt><doubt alpha="0.0" length="10" tooSmall="True" monospace="0.0">-0.15 -0.2</doubt><doubt alpha="0.0" length="10" tooSmall="True" monospace="0.0">-0.25 -0.3</doubt><doubt alpha="71.8" length="85" tooSmall="True" monospace="0.0">Proposed - pattern (web) Proposed - pattern (news) Proposed - tag *■ Proposed - dfidf</doubt><doubt alpha="0.0" length="45" tooSmall="True" monospace="0.0">5    10   15   20   25   30   35   40   45 50</doubt><doubt alpha="0.0" length="4" tooSmall="True" monospace="0.0">0.05</doubt><doubt alpha="100.0" length="1" tooSmall="False" monospace="0.0">b</doubt><doubt alpha="0.0" length="5" tooSmall="True" monospace="0.0">-0.15</doubt><doubt alpha="0.0" length="4" tooSmall="True" monospace="0.0">-0.2</doubt><p><b>Fig. 6. </b>Effect of scores. Left: relaxed attribute. Right: general attribute.</p><p>We calculated the change in precision "per evaluator", and then calculated the averaged change, i.e., the change averaged over evaluators. Figure 6 plots the averaged change and standard deviations. The effect of n(C, A) is represented by "Proposed - pattern (web)", that of <i>f (C, A) </i>by "Proposed - pattern (news)", that of t(C, A) by "Proposed - tag", and that of <i>dfidf (C, A) </i>by "Proposed -dfidf". In the relaxed attribute case, we can see that most of the scores were ef­fective at almost all ranks regardless of the evaluator (negative difference means positive effect). The effect of <i>f (C, </i>A)and t(C, A) was especially remarkable. Al­though n(C, A) has a similar curve to <i>f (C, </i>A), the effect is weaker. This may be caused by the difference in the number of documents available (As we previ­ously described, we currently cannot obtain a large number of documents from the Web). The effect <i>dfidf (C, </i>A) had was two-fold. This contributed positively at lower ranks but it contributed negatively at higher ranks (around the top 1-5). In the general attribute case, the positive effect became harder to observe although the tendency was similar to the relaxed case. However, we can see that <i>f (C, </i>A) still contributed greatly even in this case. The effect of t(C, A), on the other hand, seems to have weakened greatly.</p></subsection></section><section number="4" title=".4   Effect of Hypernym"><p>If we have a hypernym-hyponym knowledge base, we can also collect the local document set by using the hyponyms in the class as the keywords for the search engine instead of using the class label (hypernym). In this experiment, we com­pared the proposed method with this alternative. We collected about the same number of documents for the alternative method as for the proposed method to focus on the quality of collected documents. We used hyponyms with the alter­native method instead of class label <i>C </i>in patterns for n(C, A) (thus <i>n(Hs, </i>A)to be precise). <i>f(C, A) </i>was unchanged. Figure 7 plots the results in the same way as for the previous analysis (i.e., difference from the proposed method). We can see that the class label is better than hyponyms for collecting local documents at least in the current setting.</p><page local="11" global="116"/><doubt alpha="0.0" length="1" tooSmall="True" monospace="0.0">5</doubt><doubt alpha="0.0" length="4" tooSmall="True" monospace="0.0">-0.1</doubt><doubt alpha="0.0" length="11" tooSmall="True" monospace="0.0">0.04 0.02 0</doubt><doubt alpha="0.0" length="28" tooSmall="True" monospace="0.0">-0.02 -0.04 -0.06 -0.08 -0.1</doubt><doubt alpha="14.7" length="68" tooSmall="True" monospace="0.0">10     15     20     25     30     35     40     45 50 Rank (recall)</doubt><doubt alpha="0.0" length="23" tooSmall="True" monospace="0.0">-0.02 -0.04 -0.06 -0.08</doubt><p><b>Fig. 7. </b>Effect of hypernyms. Left: relaxed case. Right: general case.</p></section><section number="5" title="Discussion 5.1   Related Work"><p>Several studies have attempted to acquire attributes or attribute-value pairs [1,3,7,8,16]. Yoshida [1] proposed a method of integrating tables on the Web. Although his method consequently acquired attributes, he did not evaluate the accuracy of attributes. Yoshida et al. [16] proposed a method of identifying attribute-value pairs in Web documents. However, since this method only iden­tified the attributes obtained with the method in [1], the coverage might be bounded by the coverage of tables for attributes. Moreover, these methods did not utilize the statistics for words or lexico-syntactic patterns as ours did. Taka-hashi et al. [8] extracted triples <i>(object, attribute, value) </i>from newspaper articles using lexico-syntactic patterns and statistical scores. However, they focused only on proper nouns and selected the attribute candidates manually. Freishmann et al. [3] extracted attribute-value pairs with a high degree of precision by filtering the candidates extracted with lexico-syntactic patterns by using a model learned with supervised learning. Although this approach is promising, their method was limited to person names and we must prepare training data to apply the method to other types of objects.</p><subsection number="5.2" title="Future Directions"><p><b>Clues based on QA tests. </b>The current ranking, Eq. (1), does not exploit the observation behind the criteria in Sect. 3. Only the lexico-syntactic patterns "C no A" slightly reflect the criteria. Higher accuracy might be achieved by using patterns that directly reflect the QA tests, e.g., statistics from FAQ lists. The hyponym tests in Sect. 3.4 can also be reflected if we use a hyponymy database. In addition, it is not surprising that the proposed method was not efficient at acquiring general attributes since the score was not meant for that (although the use of class labels might be a contributing factor, ambiguous class labels<page local="12" global="117"/></p><p>Automatic Discovery of Attribute Words from Web Documents 117</p><p>cause problems at the same time). The hyponym database might be exploited to measure the generality of attributes.</p><p><b>Full use of the Web. </b>The current method cannot use all Web documents due to limitations with search engines. The more Web documents we have, the more useful the score <i>n( C, A). </i>We are currently planning to prepare our own non-restricted Web repository. Using this, we would also like to elaborate on the comparison described in Sect. 4.4 between the use of hypernyms (class labels) and hyponyms (instance words) in collecting the local document set.</p><p><b>Assessment of Coverage. </b>Currently, the actual recall with the proposed method is unknown. It will be important to estimate how many attributes are needed for practical applications, e.g., by manually analyzing the use of pattern " <i>C</i><i> </i>no <i>A"</i><i> </i>exhaustively for a certain class, <i>C. </i>In addition, since we selected classes that were successfully obtained with a hyponymy acquisition method, we cannot deny the possibility that the proposed method has been evaluated for the classes for which reliable statistics can easily be obtained. Thus, the evaluation of more difficult (e.g., more infrequent) classes will be an important future work.</p><p><b>Type Acquisition. </b>What types of questions and what types of suffix augmen­tations are possible for a given attribute (i.e., the type of attribute value) might also be useful, e.g., in value extraction and in determining type of the attribute (in the sense of "property or part-of"). This was left for the evaluators to chose arbitrarily in this study. We would like to extract such knowledge from the Web using similar techniques such as word statistics and lexico-syntactic patterns.</p><p><b>6Conclusion</b></p><p>We presented a method of acquiring attributes that utilizes statistics on words, lexico-syntactic patterns, and HTML tags. We also proposed criteria and an evaluation procedure based on question-answerability. Using the procedure, we conducted experiments with four human evaluators. The results revealed that our method could obtain attributes with a high degree of precision.</p></subsection></section><references><p>1. Yoshida, M.: Extracting attributes and their values from web pages. In: Proc. of the ACL 2002 Student Research Workshop. (2002) 72-77</p><p>2. Yoshida, M., Torisawa, K., Tsujii, J.: Integrating tables on the world wide web. Transactions of the Japanese Society for Artificial Intelligence <b>19 </b>(2004) 548-560</p><p>3. Fleischman, M., Hovy, E., Echihabi, A.:   Offline strategies for online question answering: Answering questions before they are asked.  In: Proc. of ACL 2003.</p><doubt alpha="0.0" length="10" tooSmall="False" monospace="0.0">(2003) 1-7</doubt><doubt alpha="61.2" length="129" tooSmall="False" monospace="0.0">4. Almuhareb, A., Poesio, M.: Attribute-based and value-based clustering: An eval­uation. In: Proc. of EMNLP 2004. (2004) 158-165</doubt><p>5. Fellbaum, C., ed.: WordNet: An electronic lexical database. The MIT Press (1998) 7.<page local="13" global="118"/> Berland, M., Charniak, E.: Finding parts in very large corpora. In: Proc. of ACL</p><doubt alpha="62.6" length="115" tooSmall="False" monospace="0.0">6.Hearst, M.A.:  Automatic acquisition of hyponyms from large text corpora. In: Proc. of COLING '92. (1992) 539-545</doubt><doubt alpha="0.0" length="11" tooSmall="False" monospace="0.0">'99. (1999)</doubt><doubt alpha="64.4" length="146" tooSmall="False" monospace="0.0">8. Takahashi, T., Inui, K., Matsumoto, Y.: Automatic extraction of attribute relations from text (in Japanese). IPSJ, SIG-NLP. NL-164 (2004) 19-24</doubt><p>9. Guarino, N.: Concepts, attributes and arbitrary relations: some linguistic and on-tological criteria for structuring knowledge base. Data and Knowledge Engineering</p><doubt alpha="0.0" length="14" tooSmall="False" monospace="0.0">(1992) 249-261</doubt><doubt alpha="66.2" length="65" tooSmall="False" monospace="0.0">10. Pustejovsky, J.: The Generative Lexicon. The MIT Press (1995)</doubt><p>11. Woods, W.A.: What's in a Link: Foundations for Semantic Networks. In: Repre­sentation and Understanding: Studies in Cognitive Science. Academic Press (1975) 12. Kurohashi, S., Nagao, M.: Japanese morphological analysis system JUMAN version 14. Shinzato, K., Torisawa, K.: Acquiring hyponymy relations from web documents.</p><doubt alpha="33.3" length="18" tooSmall="False" monospace="0.0">3.61 manual (1999)</doubt><doubt alpha="65.6" length="160" tooSmall="False" monospace="0.0">13. Kanayama, H., Torisawa, K., Mitsuishi, Y., Tsujii, J.: A hybrid Japanese parser with hand-crafted grammar and statistics.   In: Proc. of COLING 2000. (2000)</doubt><doubt alpha="0.0" length="7" tooSmall="False" monospace="0.0">411-417</doubt><doubt alpha="42.1" length="38" tooSmall="False" monospace="0.0">In: Proc. of HLT-NAACL04. (2004) 73-80</doubt><p>15. Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorial</p><doubt alpha="43.8" length="32" tooSmall="False" monospace="0.0">data. Biometrics33(1977) 159-174</doubt><p>16. Yoshida, M., Torisawa, K., Tsujii, J.: Chapter 10 (Extracting Attributes and Their Values from Web Pages). In: Web Document Analysis. World Scientific (2003)</p><doubt alpha="0.0" length="3" tooSmall="False" monospace="0.0">118</doubt></references></body></article>