The Text REtrieval Conference Series (TREC) has greatly encouraged Question Answering (QA) re­search in the recent years. The QA main task in the recent TREC-12 involved retrieving short concise answers to factoid and list questions, and answer nuggets for definition questions (Voorhees, 2003). The list task in TREC-12 required systems to as­semble a set of distinct and complete exact answers as responses to questions like "What are the brand names of Belgian chocolates? ". Unlike the questions in previous TREC conferences, TREC-12 list ques­tions did not specify a target number of instances to return but expected all answers contained in the cor­pus. Current QA systems (Harabagiu et al., 2003; Katz et al., 2003) usually extract a ranked list of fac­toid answers from the top returned documents by retrieval engines. This is actually the traditional way to find factoid answers. The only difference between answering list questions and factoid questions here is that list QA systems allow for multiple answers, whose scores are above a cut-off threshold.

An analysis of the results of TREC-12 list QA systems (Voorhees, 2003) reveals that many of them severely suffer from two general problems: low re­call and non-distinctive answers. The median aver­age F1 performance of list runs was only 21.3% while the best performer could only achieve 39.6% (Table 1). This unsatisfactory performance exposes the limitation of using only traditional Information Retrieval and Natural Language Processing tech­niques to find an exhaustive set of factoid answers as compared to only one.

TREC-12 Run Tag Avg Fi

In contrast to the traditional techniques, the Web is used extensively in systems to rally round factoid questions. QA researchers have explored a variety of uses of the Web, ranging from surface pattern min­ing (Ravichandran et al., 2002), query formulation (Yang et al., 2003), answer validation (Magnini et al., 2002), to directly finding answers on the Web by data redundancy analysis (Brill et al., 2001). These systems demonstrated that with the help of the Web they could generally boost baseline performance by 25%-30% (Lin 2002).

The well-known redundancy-based approach iden­tifies the factoid answer as an N-gram appearing most frequently on the Web (Brill et al. 2001). This idea works well on factoid questions because factoid questions require only one instance and web docu­ments contains a large number of repeated informa­tion about possible answers. However, when dealing with list questions, we need to find all distinct in­stances and hence we cannot ignore the less frequent answer candidates. The redundancy-based approach fails to spot novel or unexpectedly valuable informa­tion in lower ranked web pages with few occur­rences.

In this paper, we propose a novel framework to employ the Web to support list question answering. Based on the observations that multiple answer in­stances often appear in the list or table of a single web page while multiple web pages may also con­tain information about the same instance, we differ­entiate these two types of web pages. For the first category, which we call Collection Page (CP), we need to extract table/list content from the web page. For the second category, which we call Topic Page (TP), we need to find distinct web pages relating to different answer instances. We will demonstrate that the resulting system, FADA (Find All Distinct An­swers), could achieve effective list question answer­ing in the TREC corpus.

Our goal is to find as many distinct exact answers on the Web as possible. This requires us to:

• perform effective and exhaustive search; and

• extract distinct answers.

In order to perform effective search, we employ question transformation to get effectual web queries. However, this is not a trivial task. If the query is too general, too many documents may be retrieved and the system would not have sufficient resources to scan through all of them. If the query is too specific, no pages may be retrieved.

Given millions of web pages returned by search engines, our strategy is to divide-and-conquer by first identify Collection Pages (CP) that contain a list of answer instances. For example, for the question "What breeds of dog have won the "Best in Show" award at the Westminster Dog Show?", we can find a Collection Page as shown in Figure 1 (top). Such a web page is a very good resource of answers. In general, we observe that there is a large number of named entities of the type desired appearing in a Collection Page, typically in a list or table. Our in­tuition is that if we can find a Collection Page that contains almost all the answers, then the rest of the work is simply to extract answers from it or related web pages by wrapper rule induction.

Another kind of "good" web page is a Topic Page, that contains just one answer instance (Figure 1, bot­tom). It typically contains many named entities, which correspond to our original query terms and some other named entities of the answer target type. Given the huge amount of web data, there will be many Topic Pages that refer to the same answer in­stance. There is hence a need to group the pages and to identify a pertinent and distinctive page in order to represent a distinct answer.

Web page class Description

Collection Page Containing a list of answers Topic Page The best page to represent an answer instance

Relevant Page Relevant to an answer instance by providing either support or objection to the Topic Page

Irrelevant Page Not related to any answer instance

Figure 1 : Examples of Collection Page (top) and Topic Page (bottom)

The rest of the top returned web pages could be either relevant or irrelevant to the question. In sum­mary, we need to classify web pages into four classes: Collection Page, Topic Page, Relevant Page, and Irrelevant Page (Table 2), based on their func­tionality and contribution in finding list answers.

Based on the above considerations, we propose a general framework to find list answers on the Web using the following steps:

a) Retrieve a good set of web documents.

b) Identify Collection Pages and distinct Topic Pages as main resources of answers.

c) Perform clustering on other web pages based on their similarities to distinct Topic Pages to form clusters that correspond to distinct answer in­stances.

d) Extract answers from Collection Pages and Topic Page clusters.

Agichtein et al. (2001) presented a technique on learning search engine specific query transforma­tions for question answering. A set of transformation rules are learned from a training corpus and applied to the questions at the search time. Related work could also be found in Kwok et al. (2001) where the user's question is processed by a parser to learn its syntactic structure and various query modulation techniques are applied to the initial questions to get high quality results for later answer extraction.

FADA performs question parsing to identify key question words and the expected answer type. It ex­tracts several sets of words from the original ques­tion and identifies the detailed question classes. It then formulates a number of queries by combining the known facets together with heuristic patterns for list questions.

We perform both shallow and full parsing on a question followed by Named Entity Recognition (NER) to get the known query facets and their types. The shallow parser we used is the free online mem­ory-based chunker and the full parser is MINIPAR. Both parsers are very efficient and usually parse 300 words within a second. The procedure of query pars­ing is as follows:

a) Remove head words. The head words in a ques­tion could be wh-question words and leading verbs. The list of head words includes "who, what, when, where, which, how, how much, how many, list, name, give, provide, tell", etc. Removing them enables us to get the correct subject/object relation and verb in the question. For example, for question "What breeds of dog have won the 'Best in Show' award at the Westminster Dog Show?", after removing the head word, the question be­comes "breeds of dog have won the 'Best in Show' award at the Westminster Dog Show".

b) Detect subject and object for the remaining ques­tion segments by shallow parsing. For example, after parsing the above question, we get:

Westminster/NNP Dog//NNP Show/NNP NP]

From the parsed sentence, we want to get the logi­cal subject as the sentence subject or its immedi­ate modifiers. Here we have the logical subject-"breeds of dog", verb-"won", and logical object-"the best in show award". If the resulting logical subject/object is the term "that" as in the follow­ing parsed query for "U.S. entertainers that later became politicians":

[NP U.S./NNP entertainers//NNS NP]

[NP1Subject that/WDT NP1Subject] [ADVP later/RB ADVP] [VP1 became/VBD VP1] [NP1Object politicians//NNS NP1Object]

we get the noun or noun phrase before the clause as the logical subject/object. Hence, we have the logical subject-"entertainers" , action-"became" , and logical object-"politician".

c) Extract all the noun phrases as potential descrip­tions from the remaining question segments, which are usually prepositional phrases or clauses. For the "dog breeds" example, we get the descrip-tions-"Westminster Dog Show" .

d) Apply named entity recognition to the resulting description phrases by using NEParser, a fine­grained named entity recognizer used in our TREC-12 system (Yang et al., 2003). It assigns tags like "person", "location", "time", "date", "number". For the "dog breed" example, "West­minster" gets the location tag.

After the above analysis, we obtain all the known facets provided in the original question. We then make use of this knowledge to form web queries to get the right set of pages. This is a crucial task in dealing with the Web. One of the query transforma­tion rules is given as follows:

The rule starts the query optionally with leading words (list, directory, category), optionally followed by a colon or "of", followed by subject phrase (<subj>), optionally followed by action (<action>), optionally followed by object (<object>) and de­scription phrases (<description1>...<descriptionN>). In the above pattern, "?" denotes optional, omit, and "|" alternative. For example, for the "dog breed" question, we form queries "breed of dog won best in show Westminster Dog Show", "directory breed of dog best in show Westminster Dog Show", and "list breed of dog won best in show" etc.

Transforming the initial natural language ques­tions into a good query can dramatically improve the chances of finding good answers. FADA submits these queries to well-known search engines (Google, AltaVista, Yahoo) to get the top 1,000 Web pages per search engine per query. Here we attempt to re­trieve a large number of web pages to serve our goal - find All Distinct answers. Usually, there are a large number of web pages which are redundant as they come from the same URL addresses. We remove the redundant web pages using the URL addresses as the guide. We also filter out files whose formats are nei­ther HTML nor plain text and those whose lengths are too short or too long. Hence the size of the re­sulting document set for each question varies from a few thousands to ten of thousands.

In order to group the web pages returned by the search engines into the four categories discussed earlier, it is crucial to find a good set of features to represent the web pages. Many techniques such as td.idf (Salton and Buckley, 1988) and a stop word list have been proposed to extract lexical features to help document clustering. However, they do not work well for question answering. As pointed out by Ye et al. (2003) in their discussion on the per­son/organization finding task, given two resume pages about different persons, it is highly possible that they are grouped into one cluster because they share many similar words and phrases. On the other hand, it is difficult to group together a news page and a resume page about the same target entity, due to the diversity in subject matter, word choice, liter­ary styles and document format. To overcome this problem, they used mostly named entity and link information as the basis for clustering. Compared to their task, our task of finding good web documents containing answers is much more complex. The fea­tures are more heterogeneous, and it is more difficult to choose those that reflect the essential characteris­tics of list answers.

In our approach, we obtain the query words through subject/object detection and named entity recognition. We found that there are a large number of named entities of the same type appearing in a Collection Page, typically within a list or table. And in a Topic Page, there is also typically a group of named entities, which could correspond to our origi­nal query terms or answer target type. Therefore, named entities play important roles in semantic ex­pression and should be used to reflect the content of the pages.

The Web track in past TREC conferences shows that URL, HTML structure, anchor text, hyperlinks, and document length tend to contain important heu­ristic clues for web clustering and information re­trieval (Craswell and Hawking, 2002). We have found that a Topic Page is highly likely to repeat the subject in its URL, title, or at the beginning of its page. In general, if the subject appears in important locations, such as in HTML tags <title>, <H1> and <H2>, or appears frequently, then the corresponding pages should be Topic Pages and their topic is about the answer target.

Followed the above discussion, we design a set of 29 features based on Known Named Entity Type, An­swer Named Entity Type, ordinary Named Entities, list, table, URL, HTML structure, Anchor, Hyper­links, and document length to rep resent the web pages. Table 3 lists the features used in our system. In the table and subsequent sections, NE refers to Named Entity.

We trained two classifiers: the Collection Page classifier and the Topic Page classifier. The former classifies web pages into Collection Pages and non-collection pages while the later further classifies the non-collection pages into Topic Pages and Others. Both Classifiers are implemented using Decision Tree C4.5 (Quinlan 1993). We used 50 list questions from TREC-10 and TREC-11 for training and TREC-12 list questions for testing. We parse the questions, formulate web queries and collect web pages by using the algorithm described in Section 2.

Each sample is represented using the features listed in Table 3. Some of the decision rules are as follows:

Class TP

Rule a) implies that good Collection Pages should have many outlinks, NEs and especially answer NEs. Rule b) implies that good Topic Pages should have many NEs but relatively few links and answer NEs. Rule c) show that Others have deeper URL depth; while Rule d) shows that they have fewer NEs.

Based on Web page classification, we form the ini­tial sets of Collection Pages CPSet, Topic Pages TPSet and OtherSet. In order to boost the recall, we first use the outgoing links of Collection Pages to find more Topic Pages. These outgoing pages are potential Topic Pages but not necessarily appearing among the top returned web documents. Our subse­quent tests reveal that the new Topic Pages intro­duced by links from Collection Pages greatly increase the overall answer recall by 23%. The new Topic Page set becomes:

TPSet' = TPSet + {outgoing pages of CPs}

Second, we select distinct Topic Pages. We com­pare the page similarity between each pair of Topic Pages using the algorithm below.

for each pair {tpit tpj} in TPSet' if (sim(tpi,tpj)> 9)

if &ANE_in_tpi > KANE_in_tpjmove tpj into OtherSet;

Here the page similarity function sim() is a linear combination of overlaps between Known NE, An-swer_NE, URL similarity and link similarity. e is preset at 0.75 and may be overridden by the user. ttANE_in_tpi is the number of named entities of answer type in Topic Page tpi. For those pairs with high similarity, we keep the page that contains more named entities of answer type in TPSet' and move the other into OtherSet. The resulting Topic Pages in TPSet' are distinct and will be used as cluster seeds for the next step.

Third, we identify and dispatch Relevant Pages from OtherSet into appropriate clusters based on their similarities with the cluster seeds.

for each rpi in OtherSet {

insert rpi into clusterk; else insert rpi into IrrelevantSet; }

Here T is preset at 0.55, and sim() is defined as above. Each cluster corresponds to a distinct answer instance. The Topic Page provides the main facts about that answer instance while Relevant Pages provide supporting materials for the unique answer instance. The average ratio of correct clustering is 54.1% in our experiments.

Through web page clustering, we avoid early an­swer redundancy, and have a higher chance to find­ing distinct answers on the noisy Web.

We used the 37 TREC-12 list questions to test the overall performance of our system and compare the answers we found in the TREC AQUAINT corpus (after answer projection (Brill et al. 2001)) with the answers provided by NIST.

We have presented the techniques used in FADA, a system which aims to find complete and distinct answers on the Web using question parsing, web page classification/clustering and content extraction. By using the novel approach, we can achieve a recall of 0.422 and F1 of 0.464, which is significantly bet­ter than the top performing systems in the TREC-12 List QA task. The method has been found to be ef­fective. Our future work includes discovering an­swers on non-text web information, such as images. Much text information is stored as images on the web, and hence, cannot be accessed by our approach, and some do contain valuable information.