A variety of data resources have been exploited for paraphrase extraction. For example, some re­searchers extract paraphrases from multiple trans­lations of the same foreign novel (Barzilay and McKeown, 2001; Ibrahim et al., 2003), while some others make use of comparable news arti­cles that report on the same event within a small time interval (Shinyama et al., 2002; Barzilay and Lee, 2003; Dolan et al., 2004). Besides the mono­lingual corpora, bilingual parallel corpora have also been used for extracting paraphrases (Ban­nard and Callison-Burch, 2005; Callison-Burch, 2008; Zhao et al., 2008). Their basic assumption is that phrases that align with the same foreign phrase may have the same meaning.

The above methods have achieved promising results. However, their performances are usually constrained due to the scale and domain limita­tion. As an alternative, researchers have tried to acquire paraphrases from large-scale web cor­pora (Lin and Pantel, 2001; Pa§ca and Dienes, 2005; Bhagat and Ravichandran, 2008) or directly based on web mining (Ravichandran and Hovy, 2002). These methods are guided by an extended version of distributional hypothesis, namely, if two phrases often occur in similar contexts, their meanings tend to be similar. The disadvantage of these methods is that the underlying assump­tion does not always hold. Phrases with opposite meanings can also occur in similar contexts, such as "X solves Y" and "X worsens Y" (Lin and Pan-tel, 2001). In addition, the extracted paraphrases are generally short fragments with two slots (vari­ables) at both ends.

Query logs are widely used in the IR commu­nity, especially for mining similar queries. For ex­ample, Wen et al. (2002) clustered queries based on user click information. Their basic idea is that if some queries result in similar user clicks, the meanings of these queries should be similar. Such methods have also been investigated in (Gao et al., 2007) for cross-lingual query suggestion and (Zhao et al., 2007) for synonymous questions identification. This paper is partly inspired by their studies. However, we do not simply use click information as clues for mining similar queries. Instead, we mine paraphrases across queries and clicked document titles.

In addition, query logs can be used for query expansion. For instance, Cui et al. (2002) extract probabilistic correlations between query terms and document terms by analyzing query logs, which are then used to select high-quality

H1: If a query q hits a title t, then q and t are likely to be paraphrases.

H2: If queries qi and q2 hit the same title t, qi and q2 are likely to be paraphrases.

H3: If a query q hits titles t1 and t2, then ti and t2 are likely to be paraphrases.

expansion terms for new queries. Note that the expansion terms are merely related terms of the queries, not necessarily paraphrases.

There are other studies that use query logs for constructing ontologies (Sekine and Suzuki, 2007), learning named entities (Pa§ca, 2007), building user profiles (Richardson, 2008), correct­ing spelling errors (Ahmad and Kondrak, 2005), and so forth.

Nowadays, more and more users tend to search long queries with search engines. Many users even directly search questions to get exact an­swers. By analyzing our query log that records rich information including user queries, clicked urls, titles, etc., we find that most titles of clicked documents are highly related with search queries. Especially, paraphrases can be easily found from long queries and the corresponding clicked ti­tles. This motivates us to extract paraphrases from query-title pairs. Here we introduce a concept hit that will be frequently used: given a query q, a web document d, and d's title t, if there exist some users that click on d when searching q, then we say q hits t.

The hypothesis for extracting paraphrase Q-T is shown in Table 1 (H1). In addition, we find that when several queries hit the same title, the queries are likely to be paraphrases of each other. The other way round, when a query hits several titles, paraphrases can also be found among the ti­tles. We therefore further extract paraphrase Q-Q and T-T from the paraphrase Q-T. The underly­ing hypotheses can be found in Table 1 (H2 and INPUT: Q: query space, T: title space OUTPUT: Pqt: the set of paraphrase Q-T, Pqq : the set of paraphrase Q-Q, Pit: the set of paraphrase T-T, ParaSet: the set of paraphrases

Our paraphrase extraction algorithm is shown in Table 2. In particular, lines 1~7 extract paraphrase Q-T from the query log. Lines 8^14 and 15~21 extract paraphrase Q-Q and T-T, respec­tively. Line 22 combines the paraphrase Q-T, Q-Q, and T-T together. To filter noise, the extracted Q-T, Q-Q, and T-T pairs are all validated using a function IsParaphrase(si, s2). In this work, we recast paraphrase validation as a binary clas­sification problem. Any pair of (si; s2) is classi­fied as 1 (paraphrase) or 0 (non-paraphrase) with a support vector machine (SVM) classifier. The features used for classification will be detailed in Section 3.3.

In practice, we exploit a query log that contains 287 million Q-T pairs, which are then filtered us­ing the following constraints: (1) exclude Q-T pairs that are too short, i.e., either query q or tittle t contains less than three terms; (2) exclude Q-T pairs where q subsumes t or vice versa, e.g., "4 W (beef)" and "4W W tëfâ (cooking method of beef)"; (3) exclude Q-T pairs in which the similar­ity between q and t is below a predefined threshold T; (4) exclude Q-T pairs whose t contains fre­quent internet terms, such as "+J5 (homepage)", "Nèè (web site)", "iï^t (online)", since such ti­tles are mostly organization home pages, online videos, downloadable resources, etc., which are useless for our purpose of paraphrase extraction.

Given a pair of candidate paraphrases (si; s2), in which si and s2 can be either a query or a title, we exploit the following features in the classification-based paraphrase validation.

• Frequency Feature FF. FF is defined based on each (si,s2)'s frequency. We expect that more frequent (si,s2) should be more reliable.

where C(si, s2) denotes the number of times that the ( si , s2) pair occurs in the corpus. C is a nor­malizing factor (C = 10 in our experiments).

• Length Rate Feature FLR:

max{Cw (si ),Cw (s2)} where cw (s) denotes the number of words in s.

• Word Overlap Rate Feature FWOR :

max{Cw (si ),Cw (s2)} where "si n s2" is the intersection of si and s2.

• Character Overlap Rate Feature FeOR. Chi­nese words are composed of characters. It is quite often that words with similar characters share similar meanings, such as "Â'Èft (comfortable)" and "Ä (comfortable)", "äjH (sell)" and

U (sell)". Here we use FeOR to measure the sim­ilarity between si and s2 at the character level. Detailedly, we segment si and s2 into sets of characters and compute the overlap rate based on Equation (3).

• Cosine Similarity Feature Fes. In Fes, both si and s2 are represented as vectors and their co­sine similarity is computed as:

where veCw (s) is the vector of words in s, "■" de­notes the dot product of two vectors, ||veCw(s)|| is the norm of a vector. Here, the weight of each word w in a vector is computed using a heuristic similar to tf-idf:

where tf(w) is the frequency ofw in the given s, C(w) is the number of times that w occurs in the corpus, N = maxw c(w).

• Edit Distance Feature FED. Let £D(si,s2) be the edit distance at the word level between si and s2, we compute FED as follows:

A key feature of our method is that the extracted paraphrases are particularly suitable for generat­ing paraphrase patterns, especially for the hot do­mains that are frequently searched^ For example, there are quite a few paraphrases concerning the therapy of various diseases, from which we can easily induce patterns expressing the meaning of "How to treat [X] disease", such as "[X] M #0 M fë-)7", fàfï [X] M", and "[X] M W

^nff 7P£fe" Therefore, in this work, we try to generate paraphrase patterns using the extracted paraphrases^

In our preliminary experiments, we only induce paraphrase patterns from paraphrases that contain no more than 6 words^ In addition, only one slot is allowed in each pair of paraphrase patterns^ Let s1 and s2 be a pair of paraphrases extracted above^ If there exist words w G s1 and v G s2 that satisfy (1) w = v, (2) w and v are not stop words, then we can induce a pair of paraphrase patterns by re­placing w in s1 and v in s2 with a slot "[X]'\ It is obvious that several pairs of paraphrase patterns may be induced from one pair of paraphrases^

We first evaluate candidate Q-T without valida­tion To this end, we randomly sampled 5000 pairs of candidate Q-T and labeled them manu-alfy Each pair is labeled into one of the 3 classes: SAME - q and t have the same meaning; RELA - q and t have related meanings; DIFF - q and t have clearly different meanings^ The labeling results are listed in Table 3^ We can see that no candidate Q-T is in the DIFF class^ This is not surprising, since users are unlikely to click on web pages un­related to their queries^

To gain a better insight into the data, we ana­lyzed the subtle types of candidate Q-T in both SAME and RELA classes^ In detail, we sampled 1000 pairs of candidate Q-T from the 5000 pairs labeled above, in which 563 are in the SAME class, while the other 437 are in the RELA class. Our analysis suggests that candidate Q-T in the SAME class can be divided into 4 subtle types:

• Trivial change (12.61%): changes of punctu­ation or stop words, such as "%5jf £ 4 and £4 & ? ".

• Word or phrase replacement (68.38%): re­placements of synonymous words or phrases, such as "U KJP I W $7 £ + H (how mach is ...)" and "M I W 1^ Sffl Jü £^ (what is the price of...)".

• Structure change (7.10%): changes of both words and word orders, such as "ÄIE 4 77 Ä W ^ ft" 4 (what fruit can I eat on a diet)" and "n£ If 4 jRH nTW M (what fruit can help loss weight)".

• Others (11.90%): candidate Q-T that cannot be classified into the 3 types above.

The above analysis reveals that more than two thirds of candidate Q-T in the SAME class are in the "word or phrase replacement" type, while the ones with structure changes are slightly more than 7%. We believe this is mainly because queries and titles are relatively short and their structures are simple. Thus structure rewriting can hardly be conducted. This distribution is in line with that reported in (Zhao et al., 2008).

As for the RELA class, we find that 42.33% of such candidate Q-T share a problem of named en­tity mismatch, such as "J H (US) id 7^M IM" and "4 H (China) II ^1 I M". This indicates that the NE similarity feature is necessary in paraphrase validation.

The candidate Q-T extracted above are classified with a SVM classifier under its default setting. To evaluate the classifier, we run 5-fold cross val­idation with the 5000 human annotated data, in which we use 4000 for training and the rest 1000 for testing in each run. The evaluation criteria are precision (P), recall (R), and f-measure (F), which are defined as follows:

where Sa is the set of paraphrases automatically recognized with the classifier, Sm is the set of paraphrases manually annotated. Precision, re­call, and f-measure are averaged over 5 runs in the 5-fold cross validation.

Figure 2 (a) shows the classification results (dark bars). For comparison, we also show the precision, recall, and f-measure of the candidate Q-T (light bars). As can be seen, the precision is improved from 0.5592 to 0.7444 after classifica­tion. F-measure is also evidently enhanced. This result indicates that the classification-based para­phrase validation is effective. We then use all of the 5000 annotated data to train a classifier and classify all the candidate Q-T. Results show that 2,762,291 out of 4,448,347 pairs of candidate Q-T are classified as paraphrases.

From the paraphrase Q-T, we further extracted 934,758 pairs of candidate Q-Q and 438,954 pairs of candidate T-T (without validation). We ran­domly sampled 5000 from each for human an­notation. The results show that the precisions of candidate Q-Q and T-T are 0.4672 and 0.6860, re­spectively. As can be seen, the precision of can­didate Q-Q is much lower than that of candidate T-T. Our analysis reveals that it is mainly because candidate Q-Q are more noisy, since user queries contain quite a lot of spelling mistakes and infor­mal expressions.

The candidate Q-Q and T-T are also refined based on classification. We first evaluate the clas­sification performance using the 5000 human la­beled data. The experimental setups for Q-Q and

(a) Q-T classification (b) Q-Q classification (c) T-T classification

T-T classification are the same as that of Q-T clas­sification, in which we run 5-fold cross validation with a SVM classifier using its default parameters. Figure 2 (b) and (c) give the classification results (dark bars) as well as the precision, recall, and f-measure of the candidates (light bars).

We can see that the precision of Q-Q is signifi­cantly enhanced from 0.4672 to 0.7345 after clas­sification, which suggests that a substantial part of errors and noise are removed. The increase of f-measure demonstrates the effectiveness of clas­sification despite the decrease of recall. Mean­while, the quality of candidate T-T is not clearly improved after classification. The reason should be that the precision of candidate T-T is already pretty high. We then use all 5000 human labeled data to train a classifier for Q-Q and T-T respec­tively and classify all candidate Q-Q and T-T. Re­sults show that 390,920 pairs of paraphrase Q-Q and 415,539 pairs of paraphrase T-T are extracted after classification.

ize that the method currently used for inducing paraphrase patterns is simple. Hence we will im­prove the method in our following experiments. Specifically, multiple slots will be allowed in a pair of patterns. In addition, we will try to ap­ply the alignment techniques in the generation of paraphrase patterns, as Zhao et al. (2008) did.

Using the method introduced in Section 3.4, we have generated 73,484 pairs of paraphrase pat­terns that appear at least two times in the cor­pus. We randomly selected 500 pairs and labeled them manually. The results show that the preci­sion is 78.4%. Two examples are shown in Ta­ble 4, in which Pi and P2 are paraphrase patterns. Some slot fillers are also listed below. We real-

Feature Contribution. To investigate the contri­butions of different features used in classification, we tried different feature combinations for each of our three classifiers. The results are shown in Ta­ble 5, in which "+" means the feature has contri­bution to the corresponding classifier. As can be seen, the character overlap rate feature (Fcor), cosine similarity feature (Fcs), and NE similarity feature (Fne) are the most useful, which play im­portant roles in all the three classifiers. The other features are useful in some of the classifiers ex­cept the word overlap rate feature (Fwor). The classification results reported in prior sections are all achieved with the optimal feature combination.

Analysis of the Paraphrases. We combine the extracted paraphrase Q-T, Q-Q and T-T and get a total of 3,560,257 pairs of unique paraphrases. Statistics show that only 8380 pairs (0.24%) are from more than one source, which indicates that the intersection among the three sets is very small. Further statistics show that the average length of the queries and titles in the paraphrases is 6.69 (words).

To have a detailed analysis of the extracted paraphrases, we randomly selected 1000 pairs and manually labeled the precision, types, and do­mains. It is found that more than 43% of the para­phrases are paraphrase questions, in which how (36%), what (19%), and yes/no (14%) questions are the most common. In addition, we find that the precision of paraphrase questions (84.26%) is evidently higher than non-question paraphrases (65.14%). Those paraphrase questions are useful in question analysis and expansion in QA, which can hardly be extracted from other kinds of cor­pora.

As expected, the paraphrases we extract cover a variety of domains. However, around 50% of them are in the 7 most popular domains, includ­ing: (1) health and medicine, (2) documentary download, (3) entertainment, (4) software, (5) education and study, (6) computer game, (7) econ­omy and finance. This analysis reflects what web users are most concerned about. These domains, especially (4) and (6), are not well covered by the parallel and comparable corpora previously used for paraphrase extraction.