Published online by Cambridge University Press: 05 October 2012
Introduction
In this chapter, we will begin by addressing three central issues involved in test consistency: reliability, dependability, and fit. As we will explain, these issues arise because tests are never perfect, that is, any set of test scores contains error. Estimating just how much that error contributes to the scores of examinees centers on notions of reliability in norm-referenced testing (NRT), dependability in criterion-referenced testing (CRT), and what is termed fit in item response theory (IRT).
NRT reliability estimation relies heavily on correlational approaches. So, we will necessarily have to provide a brief explanation/ review of how the Pearson product-moment correlation coefficient works and how it should be interpreted. Three basic correlational approaches will then be explained: (a) test–retest reliability, (b) equivalent forms reliability, and (c) internal consistency reliabilities (including split-half adjusted by the Spearman-Brown prophecy formula, alpha, K-R20, and K-R21).
CRT dependability will be discussed in terms of two general approaches to consistency estimation: threshold-loss methods and generalizability theory approaches. The threshold-loss methods will include the original agreement and kappa coefficients based on two administrations of a CRT, and Subkoviak's (1980) short-cut methods that allow for estimating the agreement and kappa coefficients from a single test administration. The generalizability theory approaches will include discussion of single and multiple sources of error as well as in terms of the domain score approach, phi (Φ), and squared-error loss approaches including Livingston's statistic and the more effective phi(lambda).
To save this book to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.
Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.
Find out more about the Kindle Personal Document Service.
To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.
To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.