Corporaare collections ofdocuments, generallyinelec-tronic form, used mainly as a source of different kinds of linguistic information. In the last decade, the availabil­ity of many texts in machine-readable form and the devel­opment of powerful tools for exploiting them make cor­pora the basic resource for many areas of language re­search (Lexicography, Linguistics, Psycholinguistics, Nat­ural Language Processing). When a study performed on a particular corpus obtains interesting results, the possibility of extending them to a larger population is very tempting. But only corpora built according to explicit design crite­ria, which constitute a representative sample of a defined language variety, can allow the result of the study to be ex­tended without major bias errors. It follows that character­izations of existing corpora and the design of new ones are now receiving more attention: it becomes essential to be able to describe a corpus, compare it with others, and pro­duce new corpora that are representative samples of partic­ular language varieties.

The criteria that one might use to describe or design a corpus can be external or internal. External criteria are essentially non-linguistic, and therefore not present in the document itself; they cover the document's topic, genre and socio-cultural aspects (e.g., age and occupation of the au­thor) and are standardly assigned by people. By contrast, internal criteria are based on linguistic features, which are more or less directly present inside a text (e.g., words are directly present in a text, while POS tags can be exploited only after further analysis).

Corpus descriptions and corpus design techniques are usually based on external criteria (e.g., the 'Wall Street Journal corpus'). The main problem with external features is that they are not always available and, when they are, not always reliable (e.g., you can not always use the title of a text to identify its topic). Moreover, corpora produced us­ing external features can contain wide variations in internal features, which can cause problems when used by an NLP system. The decision to classify texts only on the basis of external criteria is motivated when the users are human be­ings, who can cope without any problem with differences in linguistic features. But when the user is a system, as in NLP, the performance of any task can be degraded by the presence of different linguistic features, as shown in Biber (1993; Sekine (1997; Roland and Jurafsky (1998; Folch et al. (2000).

The problem of describing and comparing corpora in relation to their internal features is becoming important. Work on corpora comparison started in the early '80s (Hofland and Johansson, 1982) with the study of the dif­ferences between British and American English and then extended to the opposition between spoken and written English (Biber, 1988) and later to differences in register (Biber, 1993; Kessler et al., 1997; Dewdney et al., 2001).

More recent developments focus on corpus homogene­ity and similarity. Both homogeneity and similarity are complex and multi-dimensional issues: a corpus can be ho­mogeneous, and two or more corpora can be similar, in re­lation to aspects such as lexis, syntax, semantics but also in relationto the structure of the texts or the presence of extra-textual information. Because we are mainly interested in textual information, we restrict our analysis to lexical, se­mantic and syntactic aspects, and we label the corpus pro­filing we are interested in as "linguistic". We call a corpus "homogeneous" when it does not contain major differences in internal features among its documents.

Kilgarriff (2001) defines corpus similarity as the "like-hood that linguistic findings based on one corpus apply to another". He presents corpus homogeneity as the prelimi­nary step to any quantitative study of corpus similarity: his claim is that without knowledge of corpus homogeneity it is not clear if it would be appropriate to measure similar­ity between, for example, a homogeneous corpus like the PILLs corpus of Patient Information Leaflets (Scott et al., 2001) and a balanced one like the Brown. He also states that ideally the measure used for corpus similarity can be used for corpus homogeneity, and presents an analysis based on word frequency lists. Illouz et al. (2000) present a method­ology for text profiling that aims to produce measures for corpus homogeneity within the different parts of a corpus. Their supervised approach is similar to Biber's work on text classification, but they use a tagger/parser to analyze syn­tactic features.

We present a technique for corpus analysis strictly based on internal features and unsupervised learning tech­niques, together with a set of measures for corpus homo­geneity and similarity.

We propose a stochastic method to describe and com­pare corpora, which is based only on their internal features. This method can be computed for any corpus, and is inde­pendent of any particular theory of language: it uses all the linguistic features of the documents, and not just a special sub-set as for example in Biber's work. The method has four steps:

1. choose the aspect you want to study and the type of feature you want to use;

4. quantify the characteristics of the corpus: we produce both a description of the corpus and measures of its homogeneity and its similarity in relation to other cor­pora.

The aim of our first experiment is to understand which text-similarity measure is most reliable, among the three currently used (-D(p|\q),X and G).

To evaluate the homogeneity of a corpus is difficult ow­ing to the lack of gold-standard judgments with which the measures can be compared. The hypothesis at the base of homogeneity is that a NLP system can reach better results when it uses an homogeneous corpus rather than an het­erogeneous one. In the experiment we run an NLP system using homogeneous and heterogeneous corpora. Then we compare the accuracy that the system achieved on each cor­pus, with the degree of homogeneity that the corpus scores. We expect to find that the accuracy for the homogeneous corpora is higher.

The NLP system we use for the evaluation is Rainbow (McCallum, 1996), which performs text classification. We choose Rainbow because it is freely available, fast, and does not require any particular annotation or linguistic re­source other than the corpus itself. Because Rainbow per­forms text analysis (builds its model) using all words or content words, we have to restrict the evaluation to just these two internal features in this experiment. We collect a set of corpora for which we have a reliable classification, and compute the homogeneity measure for each corpus. For each corpus we measure homogeneity using the three inter­document similarity measures. Then, for each similarity measure, we rank the corpora according to their homogene­ity value in increasing order, so that homogeneous corpora appear at the beginning of the list. For each of the two features, three ranked lists of homogeneity values are pro­duced.

We then use Rainbow to produce similar ranked lists; using both all-words and content words, to use as a gold standard. All the corpora for which we measure the homo­geneity are merged to form a single big corpus. We then use Rainbow to classify the new big corpus using different sizes of training corpus. The task for Rainbow is to rebuild from the merged corpus all the corpora it was made of. Ac­cording to our hypothesis, in order to achieve the same level of accuracy, homogeneous corpora need to be trained on a smaller subcorpus than heterogeneous corpora. The accu­racy of the classification of each class is computed. Classes are then ranked in a descending order, so that the homo­geneous ones appear at the beginning of the list. For each of the two features, a rank list of Rainbow accuracy values is produced. Finally, the Spearman's rho test is employed to identify the correlation between the homogeneity values and Rainbow accuracy values.

The corpus we used for this first experiment is the British National Corpus (BNC). The BNC is a 100 million-word collection of samples of written and spoken language, from a wide range of sources, designed to represent a wide cross-section of current British English (monolingual syn-chronic corpus). Moreover, it is a general corpus which includes many different language varieties, and is not lim­ited to any particular subject field, genre or register. There has been a lot of work on the classification of BNC docu­ments. The BNC Index (Lee, 2001) is an attempt to com­bine and consolidate some of these suggestions. The re­sult is a resource which provides an accurate classification of the documents in the BNC, according to many differ­ent kinds of external criteria such as medium, domain and genre. According to the medium, BNC documents can be classified into six different classes: spoken, written-to-be-spoken, book, periodical, published miscellanea, and un­published miscellanea. For domain, spoken English can be classified into five classes (e.g., transcription of business .

recordings, spontaneous natural conversations), and writ­ten English into nine (e.g., applied science, arts, belief and thought). There are 24 genres for spoken and 46 genres for written English (among the genres for written English there are for example personal letters, university essays, tabloid newspaper, bibliographies and instructional texts).

To avoid comparing classes whose size is too dissimilar:

• each BNC document is divided into chunks of a fixed size. For the first experiment, chunks of 20,000 words were produced. If a document is too small it is dis­charged. If it is big enough to contain more than one chunk, multiple chunks are produced, and considered as individual documents in the analysis;

for each BNC classes from medium, domain and genre, a corpus is created with the same number of chunks. For the experiment we produced corpora of 20 chunks each. If a class does not contains enough documents it is discharged; otherwise 20 chunks are chosen randomly.

This gives 51 corpora with 20 documents of 20,000 words each. Three random corpora made of20 chunks cho­sen randomly from the BNC are also created. We expected random corpora to be less homogeneous than all the other corpora.

The three homogeneity measures (using , and ) for each corpus are calculated, and the corpora are ranked according to their homogeneity score: corpora with a lower score are considered more homogeneous than ones with a higher score. Then the 54 corpora are merged to form one big corpus, and Rainbow is used to see how accu­rately it can recover each of the 54 corpora, using training sets of different sizes. The sizes used were 1, 5, 10,15; e.g., when the training set size was 5, the task for Rainbow was to recover the other 15 same-class documents out ofthe pot of 1080 documents. For each corpus, we compute the accu­racy, the proportion of correctly classified documents, and the standard deviation calculated on 50 trials.

Tables 2 and 3 list the homogeneity measures for the five corpora at the beginning and end of the lists ranked by homogeneity and Rainbow accuracy values, using all words as features. Using Rainbow, the three random corpora ap­pear to be less homogeneous than all the other corpora, as expected. The homogeneity measures based on inter­document distance instead partially failed; in fact, although they all appear somewhere at the bottom of the rank list, just one of them turns up after all the non-random corpora.

We use Spearman's rho test (Owen and Jones, 1977) to compare the ranks obtained using the homogeneity measure and Rainbow. Spearman's correlation is 1 when the two ranks are exacly the same, and when no correlation is found between the two ranked lists. The results, presented in table 4 for all words and in table 5 for content words, are always positive and usually within the significance level of

The Spearman correlation values show that the original distinction between homogeneous and heterogeneous cor­pora is maintained in Rainbow: corpora with a low homo­geneity score need a small training set to achieve a high ac­curacy in the classification task. By contrast, heterogeneous and random corpora need a bigger training set to achieve an accuracy which, however, is smaller than the one obtained by the homogeneous corpora.

None of the three text-similarity measures used to com­pute homogeneity produces a rank which follows exactly the same order identified with Rainbow, even if the constant high values of the standard deviation suggest that the rank order identify by Rainbow is not fixed. Among the three measures, provides the closest rank, expecially when all words are used.

Various reasons may be responsible for the unclarity of the results:

lack of data: chunks, in which we divide the docu­ments, and the number of chunks, we set for each cor­pus, are not big enough. For this experiment we pro­duce chunks of 20,000 words and we use corpora of 20 chunks each. We also try to use chunks of 50,000 words and corpora made of 50 chunks each, but the

Table 5: Spearman's correlation between Rainbow accu­racy values and Homogeneity values using content words

Table 3: The accuracy obtained by Rainbow analyzing all words for homogeneous and heterogeneous subcorpora using training set of different size: 1,5, 10 and 15 document per class respectively number of BNC classes which contain these amounts of data are few and appear to be all quite heteroge­neous; presence of noise in the data: at the moment we use the N most frequent internal features present in each corpus, and for this experiment we set N equal to 500. We would like to consider ways of identifying the fea­tures that seem more likely to show differences among the documents, and of filtering out those which instead can only create noise;

• the use of Rainbow as a gold-standard judgment for homogeneity: to classify texts any system uses a mix of homogeneity and similarity, so the attempt to use its classification to evaluate a homogeneity measure can be misleading ;

the three text similarity measures used may not be the best for studying corpus homogeneity and similarity.

The results obtained from this first attempt to evaluate the homogeneity measure confirm the hypothesis that ho­mogeneous corpora need a smaller training set than hetero­geneous corpora to achieve a certain degree of accuracy.

But the methodology we have used is still too unrefined to produce clear results.

As far as the methodology is concerned, the aspect re­quiring further attention is the use of some kind of feature selection in order to analyze just the more distinctive fea­tures. At the moment we are considering different types of feature selection. We also want to use a fourth text-similarity measure - perplexity.

As far as evaluation is concerned, other experiments to study the validity and reliability of the measures proposed to quantify homogeneity and similarity are needed. Be­cause the main applications of the two measures are in NLP, they should still be tested in relation to a NLP task. We would like to consider a different system from text classi­fication and also possible ways of combining the two mea­sures.