Marti Hearst (1992) was the first to use a pat­tern-based approach to extract hyponym relations from a raw corpus. She used an iterative process to semi-automatically learn patterns. However, a corpus of 20MB words yielded only 400 examples. Our pattern-based algorithm is very similar to the one used by Hearst. She uses seed examples to manually discover her patterns whearas we use a minimal edit distance algorithm to automatically discover the patterns.

The second class of algorithms uses co­occurrence statistics (Hindle 1990, Lin 1998). These systems mostly employ clustering algo­rithms to group words according to their meanings in text. Assuming the distributional hypothesis (Harris 1985), words that occur in similar gram­matical contexts are similar in meaning. Curran and Moens (2002) experimented with corpus size and complexity of proximity features in building automatic thesauri. CBC (Clustering by Commit­tee) proposed by Pantel and Lin (2002) achieves high recall and precision in generating similarity lists of words discriminated by their meaning and senses. However, such clustering algorithms fail to name their classes.

Caraballo (1999) was the first to use clustering for labeling is-a relations using conjunction and apposition features to build noun clusters. Re­cently, Pantel and Ravichandran (2004) extended this approach by making use of all syntactic de­pendency features for each noun.

In the first phase of the algorithm, feature vec­tors are extracted for each word that occurs in a semantic class. Each feature corresponds to a grammatical context in which the word occurs. For example, "catch_" is a verb-object context. If the word wave occurred in this context, then the con­text is a feature of wave.

We then construct a mutual information vector MI(e) = (mie1, mie2, miem) for each word e, where mief is the pointwise mutual information between word e and context f, which is defined as:

Following (Pantel and Lin 2002), a committee for each semantic class is constructed. A commit­tee is a set of representative elements that unambi­guously describe the members of a possible class. For example, in one of our experiments, the com­mittees for semantic classes (A) and (B) from Sec­tion 3 were:

A) peach, pear, pineapple, apricot, mango, raspberry, lemon, blueberry

B) Phil Donahue, Pat Sajak, Arsenio Hall, Geraldo Rivera, Don Imus, Larry King, David Letterman

By averaging the feature vectors of the commit­tee members of a particular semantic class, we obtain a grammatical template, or signature, for that class. For example, Figure 1 shows an excerpt of the grammatical signature for semantic class (B). The vector is obtained by averaging the fea­ture vectors of the words in the committee of this class. The "V:subj:N:joke" feature indicates a sub­ject-verb relationship between the class and the verb joke while "N:appo:N:host" indicates an ap­position relationship between the class and the noun host. The two columns of numbers indicate the frequency and mutual information scores.

To name a class, we search its signature for cer­tain relationships known to identify class labels. These relationships, automatically learned in (Pantel and Ravichandran 2004), include apposi­tions, nominal subjects, such as relationships, and like relationships. We sum up the mutual informa­tion scores for each term that occurs in these rela­tionships with a committee of a class. The highest scoring term is the name of the class.

The syntactical co-occurrence approach has worst-case time complexity O(nk), where n is the number of words in the corpus and k is the feature-space (Pantel and Ravichandran 2004). Just to parse a 1 TB corpus, this approach requires ap­proximately 10.2 years (see Table 2).

We present an algorithm for learning patterns at multiple levels. Multilevel representation is de­fined as the different levels of a sentence such as the lexical level and POS level. Consider two strings a(1, n) and b(1, m) of lengths n and m re­spectively. Let a1(1, n) and a2(1, n) be the level 1 (lexical level) and level 2 (POS level) representa­tions for the string a(1, n). Similarly, let b1(1, m) and b2(1, m) be the level 1 and level 2 representa­tions for the string b(1, m). The algorithm consists of two parts: calculation of the minimal edit dis­tance and retrieval of an optimal pattern. The minimal edit distance algorithm calculates the number of edit operations (insertions, deletions and replacements) required to change one string to another string. The optimal pattern is retrieved by

Table 3. Top 15 lexico-syntactic patterns discovered by our system.

Y such as X

Y, including X, Y, such as X Y, especially X

keeping track of the edit operations (which is the second part of the algorithm).

Algorithm for calculating the minimal edit distance between two strings Algorithm for optimal pattern retrieval

We experimentally set (by trial and error):

cost (substitution) = 0 if axi=bij

The above algorithm takes O(y) time for every pair of strings of length at most y. Hence, if there are x strings in the collection, each string having at most length y, the algorithm has time complexity O(xy) to extract all the patterns in the collection.

Applying the above algorithm on a corpus of 3GB with 50 is-a relationship seeds, we obtain a set of 600 lexico-POS. Following are two of them:

e.g. ...caldera or lava lake...

e.g. ... leukemia, the cancer of ...

Note that we store different POS variations of the anchors X and Y. As shown in example 1, the POS variations of the anchor X are (JJ NN, JJ NN NN, NN). The variations for anchor Y are (JJ JJ NN, JJ, etc.). The reason is quite straightforward:

we need to determine the boundary of the anchors X and Y and a reasonable way to delimit them would be to use POS information. All the patterns produced by the multi-level pattern learning algo­rithm were generated from positive examples. From amongst these patterns, we need to find the most important ones. This is a critical step because frequently occurring patterns have low precision whereas rarely occurring patterns have high preci­sion. From the Information Extraction point of view neither of these patterns is very useful. We need to find patterns with relatively high occur­rence and high precision. We apply the log likeli­hood principle (Dunning 1993) to compute this score. The top 15 patterns according to this metric are listed in Table 3 (we omit the POS variations for visibility). Some of these patterns are similar to the ones discovered by Hearst (1992) while other patterns are similar to the ones used by Fleischman et al. (2003).

To extract hyponym relations, we use a fixed number of patterns across a corpus. Since we treat each sentences independently from others, the algorithm runs in linear time O(n) over the corpus size, where n is number of sentences in the corpus.

We use a 15GB newspaper corpus consisting of TREC9, TREC 2002, Yahoo! News ~0.5GB, AP newswire ~2GB, New York Times ~2GB, Reuters ~0.8GB, Wall Street Journal ~1.2GB, and various online news website ~1.5GB. For our experiments, we extract from this corpus six data sets of differ­ent sizes: 1.5MB, 15 MB, 150 MB, 1.5GB, 6GB and 15GB.

For the co-occurrence model, we used Minipar (Lin 1994), a broad coverage parser, to parse each data set. We collected the frequency counts of the grammatical relationships (contexts) output by Minipar and used them to compute the pointwise mutual information vectors described in Section 3.1. For the pattern-based approach, we use Brill's POS tagger (1995) to tag each data set.

We performed a manual evaluation to estimate the precision of both systems on each dataset. For each dataset, both systems extracted a set of is-a

Table 4. Is-a relationships assigned to three randomly selected words (using pattern-based system on 15GB dataset).

Random Word Human WordNet Pattern-based System (Ranked)

Sanwa Bank bank none subsidiary / lender / bank

MCI Worldcom Inc. telecommunications company none phone company / competitor / company cappuccino beverage none item / food / beverage Table 5. Average precision, top-3 precision, and MRR Table 6. Lenient average precision, top-3 precision, for both systems on each dataset. and MRR for both systems on each dataset.

relationships. Six sets were extracted for the pat­tern-based approach and five sets for the co­occurrence approach (the 15GB corpus was too large to process using the co-occurrence model -see dependency parsing time estimates in Table 2).

From each resulting set, we then randomly se­lected 50 words along with their top 3 highest ranking is-a relationships. For example, Table 4 shows three randomly selected names for the pat­tern-based system on the 15GB dataset. For each word, we added to the list of hypernyms a human generated hypernym (obtained from an annotator looking at the word without any system or Word­Net hyponym). We also appended the WordNet hypernyms for each word (only for the top 3 senses). Each of the 11 random samples contained a maximum of 350 is-a relationships to manually evaluate (50 random words with top 3 system, top 3 WordNet, and human generated relationship).

We presented each of the 11 random samples to two human judges. The 50 randomly selected words, together with the system, human, and WordNet generated is-a relationships, were ran­domly ordered. That way, there was no way for a judge to know the source of a relationship nor each system's ranking of the relationships. For each relationship, we asked the judges to assign a score of correct, partially correct, or incorrect. We then computed the average precision of the system, human, and WordNet on each dataset. We also computed the percentage of times a correct rela­tionship was found in the top 3 is-a relationships of a word and the mean reciprocal rank (MRR). For each word, a system receives an MRR score of 1 / M, where M is the rank of the first name judged correct. Table 5 shows the results comparing the two automatic systems. Table 6 shows similar results for a more lenient evaluation where both correct and partially correct are judged correct.

For small datasets (below 150MB), the pattern-based method achieves higher precision since the co-occurrence method requires a certain critical mass of statistics before it can extract useful class signatures (see Section 3). On the other hand, the pattern-based approach has relatively constant precision since most of the is-a relationships se­lected by it are fired by a single pattern. Once the co-occurrence system reaches its critical mass (at 150MB), it generates much more precise hypo-nyms. The Kappa statistics for our experiments were all in the range 0.78 - 0.85.

Table 7 and Table 8 compare the precision of the pattern-based and co-occurrence-based methods with the human and WordNet hyponyms. The variation between the human and WordNet scores across both systems is mostly due to the relative cleanliness of the tokens in the co-occurrence-based system (due to the parser used in the ap­proach). WordNet consistently generated higher precision relationships although both algorithms approach WordNet quality on 6GB (the pattern-based algorithm even surpasses WordNet precision on 15GB). Furthermore, WordNet only generated a hyponym 40% of the time. This is mostly due to the lack of proper noun coverage in WordNet.

On the 6 GB corpus, the co-occurrence approach took approximately 47 single Pentium-4 2.5 GHz processor days to complete, whereas it took the pattern-based approach only four days to complete on 6 GB and 10 days on 15 GB.

The co-occurrence model has higher precision than the pattern-based algorithm on most datasets.