Recent work in automated question answering has suggested that extracting precise short answers is more difficult than extracting full sentence or pas­sage length ones. Also, retrieving answers from smaller document spaces may be more difficult than retrieving answers from larger ones, if the smaller spaces have less redundant coverage of potential an­swers. This paper examines the problem of extract­ing short answers from a single sentence, where there is virtually no redundancy. The focus on word-level analysis of a single sentence may uncover aspects of semantics which have heretofore been masked in sentence granularity answers.

To approach this sentence comprehension task, we introduce a novel robust statistical model. One of the barriers to building statistical models for ques­tion answering has been the lack of suitable training data, and the cost associated with annotating data when available (Ittycheriah et al., 2001). Here we ex­plore the use of raw, unannotated "found" data from the Web and present the first system that exploits whole sentence explanations in training a short an­swer extraction model. An example of these data is shown in Figure 1.

The model we propose integrates information from two different sources: semantic class preferences (re­strictions on the type of answer expected, e.g. a person or a plant) and answer context (words likely to appear near the answer). Answer context is ex­ploited using a new local alignment method. After training, the complete model achieves 70% match accuracy on one dataset, and 59% on another.

Class Preference Answer Context

Question : Who 'invented 'eyeglasses?

Explanation : Marco Polo reported seeing many pairs of eyeglasses worn by the Chinese as early as 1275, 500 years before lens grinding became an art in the West.

Answer : The Chinese (not Marco Polo or the West)

Figure 1: TS Trivia Database Entry #42764. Both class preference and context must be used to extract the correct answer.

A practical problem in developing a statistical model for question answering is the lack of training data. We explore the use of unmodified, unannotated trivia games as data, as suggested in (Mann, 2001).

With the increasing mass appeal of the Web, trivia games are becoming big business. The general pub­lic submits questions, and trivia game companies award prizes to those who correctly answer the most questions, and in the process profit from advertising. Some trivia databases are quite large, containing up to two hundred thousand entries (Ford, 1999).

In this paper we use two trivia databases as main resources: "Phishy" - PH (MacDonald, 1998) and "TriviaSpot" - TS (Trivia Machine Inc., 1999). PH has approximately 5,000 questions, each with the correct answer. TS is larger (more than 50,000 ques­tions), but only a small part (11,000 questions) was made available for this research. Each TS database entry, along with the correct answer, includes three wrong answers and in many cases an explanation. The explanations in TS vary in content. Some are justifications for the answer as in Figure 1. Others provide additional information (e.g. "Leonardo Da Vinci outlined the idea of contact lenses in 1508") or refute wrong answers (e.g. "Franklin wore glasses, but didn't invent them"). Both of these databases contain questions and answers written by many dif­ferent people, and there is no guarantee that the answers to the questions are correct.

Trivia games like these are an appealing source of data for those interested in question answering. First, they cover a wide domain of knowledge and are grammatically complex. Second, the questions typically ask for simple facts whose answer is often only a word or two, which is similar to other question answering tasks. Finally, trivia databases encode a tremendous amount of semi-structured information. In this paper we extract two types of knowledge from this heterogeneous database: class preference for se­mantic tags, and mixture parameters for our final model. These databases doubtlessly contain addi­tional information.

While the TS database contains explanations, the PH database does not. To generate explanations for the PH questions, we searched the Web using the Google search engine with both the question words and the short answer as the query terms. We re­trieved the top 100 ranked pages for each query and segmented the sentences automatically. From those sentences, we selected as explanations the sentences which had both the highest word overlap with the question and contained a word in the short answer.

As a result of this search procedure, we collected nearly 22,000 explanations for approximately 3,000 questions. Of course, the explanations were quite noisy. Some, though they included the short answer terms, did not have enough information to conclude that the short answer in fact answered the question. Many explanations were ungrammatical or odd mix­tures of sentence fragments and spurious punctua­tion. Since the explanations were collected automat­ically, in some cases we found multiple explanations for the same question. For evaluation purposes, a sample of 1,000 sentences were randomly selected from each corpus.

Many questions give a clear class preference on the types of answers they expect. For example, the question "The Star of Africa is what type of gem?' strongly prefers the answer to be a gem. Intuitively, class preferences can be described as words in the question which are somewhat unlikely to appear in the answer sentence, and instead serve to restrict the types of entities suitable as answers. Sometimes, these preferences can be found in the WH phrase it­self (e.g. "Who", "Where", "What type of fruit"). In other questions, more useful preferences can be found in different locations. For example in "What is the profession of James Herriot", profession is the class preference.

Class preferences fall into two broad categories: preferences for proper nouns and for common nouns. Proper nouns exhibit great plasticity in use, some­times even ranging over lexical elements which are primarily function words (e.g. the rock band "The The"). In contrast, common noun classes, while open to novel constructions, are much more static. One example of a common noun class preference is for color names, which are dominated by a static fixed set, with rare additions in popular usage (e.g. "teal").

Since these two classes of preferences exhibit such different characteristics, different types of processes may be used to deal with both. In order to detect proper nouns, a dynamic algorithm which tags to­kens according to a set of contextual and intra-word features (Brecket al., 1999; Cucerzan and Yarowsky, 1999) must be used. For common nouns, a large, static hand-crafted knowledge base is more appro­priate. The next two sections describe methods to handle both types of categories.

The methods described above have only used class-based approaches for finding potential answers in sentences. These methods ignore potential infor­mation from shared structure which might augment the performance. Prior methods for using structural information have relied on picking out information from parses (Hovy et al., 2001) or complex inference chains (Harabagiu et al., 2000) in order to extract the correct words from the sentence.

In this paper, we examine an 'informal semantics' approach, which is at the other end of the spectrum. Without building an intermediary semantic repre­sentation, we use matching structure directly to in­dicate which words are best matches. The model we developed first finds anchors, words that appear in both the question and explanation. Once anchors are found, for any given tuple (anchor a, word w, and WH head c), three features are use to determine the quality of the local alignment:

• de(w,a), the distance in the explanation from w to a: Larger distance between the answer and

What were George Washington's false teeth made of?

What did the Taft-Hartley Act of 1947 attempt to regulate?

In what field of study would you find flying buttresses?

What was Gary Numan's first album on the Beggars Banquet label called?

How is Frederick Austerlitz better known?

Once a question has been detected as likely to have involved movement, the WH phrase is removed and placed at the end of the sentence. The end is not always the trace position, but in our test sets it was for an overwhelming number of questions. With our retrained WH segmenter and question parser (Sec­tion 3.3), this method for unfolding WH-movement improves overall system performance.

In this paper, we presented a novel robust statisti­cal model for a sentence comprehension task. The local alignment method presents a new robust way to match answer contexts from the question and the explanation. Combining the local alignment with a statistical model of class/semantic tag match model and an ontology match module, we were able to reach high levels of performance on this task (70% on the Phishy data and 59% on the TriviaSpot data).

Through the training of the mixture model for combining local alignment features and the class/semantic tag match, we demonstrated the value of explanations for statistical estimation of intra-sentential features. The use of explanations for training this model suggests that (question, ex­planation, answer) tuples may be useful in training future question answering systems.

In investigating this sentence comprehension problem, we took an informal semantics approach. This operational semantics may allow the system to see aspects of meaning which are lost in truth-functional semantics. In general, this sentence com­prehension task presents an opportunity for empir­ically testing semantic representations, without the overhead needed to complete the full information re­trieval task required in question answering (i.e. no document or sentence retrieval needed). Deficien­cies in the sentence comprehension module might be masked by a very effective document retrieval sys­tem, but exposed under this more constrained test.