Corpora have been extensively employed in sev­eral NLP tasks as the basis for automatically learning models for language analysis and gener­ation. In theory, data-driven (empirical or statis­tical) approaches are well suited to take intrinsic characteristics of human language into account. In practice, external factors also determine to what extent they will be popular and/or effective for a given task, so that they have shown different per­formances according to the availability of corpora, to the linguistic complexity of the task, etc.

An essential component of most empirical sys­tems is the language model (LM) and, in partic­ular, n-gram language models. It is the LM that tells the system how likely a word or n-gram is in that language, based on the counts obtained from corpora. However, corpora represent a sample of a language and will be sparse, i.e. certain words or expressions will not occur. One alternative to min­imise the negative effects of data sparseness and account for the probability of out-of-vocabulary words is to use discounting techniques, where a constant probability mass is discounted from each n-gram and assigned to unseen n-grams. Another strategy is to estimate the probability of an un­seen n-gram by backing off to the probability of the smaller n-grams that compose it.

In recent years, there has also been some ef­fort in using the web to overcome data sparseness, given that the web is several orders of magnitude larger than any available corpus. However, it is not straightforward to decide whether (a) it is bet­ter to use the web than a standard corpus for a given task or not, and (b) whether corpus and web counts should be combined and how this should be done (e.g. using interpolation or back-off tech­niques). As a consequence there is a strong need for better understanding of the impacts of web fre­quencies in NLP systems and tasks.

More reliable ways of combining word counts could improve the quality of empirical NLP sys­tems. Thus, in this paper we discuss web-based word frequency distributions (§ 2) and investigate to what extent "web-as-a-corpus" approaches can be employed in NLP tasks compared to standard corpora (§ 3). Then, we present the results of two experiments. First, we compare word counts drawn from general-purpose corpora, from spe­cialised corpora and from the web (§ 4). Second, we propose several methods to combine data from heterogeneous corpora (§ 5), and evaluate their ef­fectiveness in the context of a specific multiword expression task: automatic noun compound iden­tification. We close this paper with some conclu­sions and future work (§6).

Conventional and, in particular, domain-specific corpora, are valuable resources which provide a closed-world environment where precise n-gram counts can be obtained. As they tend to be smaller than general purpose corpora, data sparseness can considerably hinder the results of statistical meth­ods. For instance, in the biomedical Genia cor­pus (Ohta et al., 2002), 45% of the words occur only once (so-called hapax legomena), and this is a very poor basis for a statistical method to decide whether this is a significant event or just random noise.

One possible solution is to see the web as a very large corpus containing pages written in sev­eral languages and being representative of a large fraction of human knowledge. However, there are some differences between using regular corpora and the web as a corpus, as discussed by Kilgar-riff (2003). One assumption, in particular, is that page counts can approximate word counts, so that the total number of pages is used as an estimator of the n-gram count, regardless of how many oc­currences of the n-gram they contain.

This simple underlying assumption has been employed for several tasks. For example, Grefen-stette (1999), in the context of example-based ma­chine translation, uses web counts to decide which of a set of possible translations is the most natural one for a given sequence of words (e.g. groupe de travail as work group vs labour collective). Like­wise, Keller and Lapata (2003) use the web to esti­mate the frequencies of unseen nominal bigrams, while Nicholson and Baldwin (2006) look at the interpretation of noun compounds based on the individual counts of the nouns and on the global count of the compound estimated from the web as a large corpus.

Villavicencio et al. (2007) show that the web and the British National Corpus (BNC) could be used interchangeably to identify general-purpose and type-independent multiword expressions. La-pata and Keller (2005) perform a careful and systematic evaluation of the web as a corpus in other general-purpose tasks both for analysis and generation, comparing it with a standard corpus (the BNC) and using two different techniques to combine them: linear interpolation and back-off. Their results show that, while web counts are not as effective for some tasks as standard counts, the combined counts can generate results, for most tasks, that are as good as the results produced by the best individual corpus between the BNC and the web. Nakov (2007) further investigates these tasks and finds that, for many of them, effective attribute selection can produce results that are at least comparable to those from the BNC using counts obtained from the web.

On the one hand, the web can minimise the problem of sparse data, helping distinguish rare from invalid cases. Moreover, a search engine al­lows access to ever increasing quantities of data, even for rare constructions and words, which counts are usually equated to the number of pages in which they occur. On the other hand, n-grams in the highest frequency ranges, such as the words the, up and down, are often assigned the estimated size of the web, uniformly. While this still gives an idea of their massive occur­rence, it does not provide a finer grained distinc­tion among them (e.g. in the BNC, the, down and up occur 6,187,267, 84,446 and 195,426 times, respectively, while in Yahoo! they all occur in 2,147,483,647 pages).

When we compare n-gram counts estimated from the web with counts taken from a well-formed standard corpus, we notice that web counts are "estimated" or "approximated" as page counts, whereas standard corpus counts are the exact number of occurrences of the n-gram. In this way, web counts are dependent on the particular search engine's algorithms and representations, and these may perform approximations to handle the large size of their indexing structures and procedures, such as ignoring punctuation and using stopword lists (Kilgarriff, 2007). This assumption, as well as the following discussion, are not valid for for controlled data sets derived from Web data, such as the Google 1 trillion n-grams. Thus, our re­sults cannot be compared to those using this kind of data (Bergsma et al., 2009).

In data-driven techniques, some statistical mea­sures are based on contingency tables, and the counts for each of the table cells can be straight­forwardly computed from a standard corpus. However, this is not the case for the web, where the occurrences of an n-gram are not precisely calculated in relation to the occurrences of the (n — 1)-grams composing it. For instance, the n-gram the man may appear in 200,000 pages, while the words the and man appear in respec­tively 1,000,000 and 200,000 pages, implying that the word man occurs with no other word than the .

In addition, the distribution of words in a stan­dard corpus follows the well known Zipfian dis­tribution (Baayen, 2001) while, in the web, it is very difficult to distinguish frequent words or n-grams as they are often estimated as the size of the web. For instance, the Yahoo! frequencies plotted in figure 1(a) are flattened in the upper part, giv­ing the same page counts for more than 700 of the most frequent words. Another issue is the size of the corpus, which is an important information, of­ten needed to compute frequencies from counts or to estimate probabilities in n-gram models. Un­like the size of a standard corpus, which is easily obtained, it is very difficult to estimate how many pages exist on the web, especially as this number is always increasing.

But perhaps the biggest advantage of the web is its availability, even for resource-poor languages and domains. It is a free, expanding and easily ac­cessible resource that is representative of language use, in the sense that it contains a great variability of writing styles, text genres, language levels and knowledge domains.

In this section, we describe an experiment to com­pare the probability distribution of the vocabulary of two corpora, Europarl (Koehn, 2005) and Ge-nia (Ohta et al., 2002), that represent a sample of general-purpose and specialised English. In

Vep Vgenia Vinter

addition to both corpora, we also considered the counts from the web as a corpus, using Google and Yahoo! APIs, and these four corpora act as n-gram count sources. To do that, we preprocessed the data (§ 4.1), extracted the vocabularies from each corpus and calculated their counts in our four n-gram count sources (§ 4.2), analysing their rank plots to compare how each of these sources models general-purpose and specialised language (§ 4.3). The experiments described in this sec­tion were implemented in the mwetoolkit and are available at http://sf.net/projects/ mwetoolkit/.

• all-uppercase and mixed case words were normalised to their predominant form, if it accounts for at least 80% of the occurrences;

• uppercase words at the beginning of sentences were lowercased;

• other words were not modified.

In our second experiment, the goal is to propose and to evaluate techniques for the combination of n-gram counts from heterogeneous sources. Therefore, we will use the insights about the vo­cabulary differences presented in the previous sec­tion. In this evaluation, we measure the impact of the suggested techniques in the identification of noun-noun compounds in corpora. Noun com­pounds are very frequent in general-purpose and specialised texts (e.g. bus stop, European Union and gene activation). We extract them automat­ically from ep and from genia using a standard method based on POS patterns and association measures (Evert and Krenn, 2005; Pecina, 2008; Ramisch et al., 2010).

In this work, we presented a detailed evalua­tion of the use of web frequencies as estima­tors of corpus frequencies in general-purpose and specialised tasks, discussing some important as­pects of corpus-based versus web-based n-gram frequencies. The results indicate that they are not only very distinct but they are so in different ways. The importance of domain-specific data for modelling a specialised vocabulary is discussed in terms of using ep to get Vgenia counts. Further­more, the web corpora were more similar to genia than to ep, which can be explained by the fact that "similar" is different from "good", i.e. they might be equally bad in modelling genia while they are distinctly better for ep.

We also proposed heuristics to combine count sources inspired by standard interpolation and back-off techniques. Results show that we can­not use web-based or combined counts to identify specialised noun compounds, since they do not help minimise data sparseness. However, general­purpose extraction is improved with the use of web counts instead of counts drawn from standard corpora.

Future work includes extending this research to other languages and domains in order to es­timate how much of these results depend on the corpora sizes. Moreover, as current interpolation techniques usually combine two corpora, weights are estimated in a more or less ad hoc proce­dure (Lapata and Keller, 2005). Interpolating sev­eral corpora would need a more controlled learn­ing technique to obtain optimal weights for each frequency function. Additionally, the evaluation shows that corpora perform differently according to the frequency range. This insight could be used to define weight functions for interpolation.

This research was partly supported by CNPq (Projects 479824/2009-6 and 309569/2009-5), FINEP and SEBRAE (COMUNICA project FINEP/SEBRAE 1194/07). Special thanks to Flâvio Brun for his thorough work as volunteer proofreader.