We emphasize that the ability for a corpus to provide accurate estimates of a linguistic parameter depends on the combined influence of domain considerations (coverage bias and selection bias) and distribution considerations (corpus size). By using a series of experimental corpora on the domain of Wikipedia articles, we can demonstrate the impact of corpus size, coverage bias, selection bias, and stratification on representativeness. Empirical results show that robust sampling methods and large sample sizes can only give you a better representation of the operational domain (i.e. overcome selection bias). However, by themselves, these factors cannot help you achieve accurate quantitative-linguistic analyses for the actual domain (i.e. overcome coverage bias) Uncontrolled domain considerations can lead to unpredictable results with respect to accuracy.

We show that attending to domain considerations in corpus design involves three steps: (1) describing the domain as fully as possible; (2) operationalizing the domain; (3) sampling the texts. Describing the domain requires defining the boundaries of the domain: what texts belong within the domain and what do not? Describing the domain requires identifying important internal categories of texts that reflect qualitative variation within the domain. Domain description should be carried out systematically using a range of sources that can be evaluated for quality and triangulated. Operationalizing the domain refers to specifying the set of texts that are available for sampling; operational domains are always precisely bounded and specified. A sampling frame is an itemized list of all texts (from the operational domain) that are available for sampling. A sampling unit is the individual “object” (usually a text) that will be included in the corpus. Stratification is the process of collecting texts according to identified categories within the domain, and is usually desirable in corpus design. Proportionality refers to the relative sizes of strata within the sample. Strata can be proportional or equal-sized. Sampling methods can be broadly categorized as random and nonrandom.