Hostname: page-component-5f745c7db-f9j5r Total loading time: 0 Render date: 2025-01-06T07:16:53.505Z Has data issue: true hasContentIssue false
  • English
  • Français

Bayesian Inference and the Classical Test Theory Model: Reliability and True Scores

Published online by Cambridge University Press:  01 January 2025

Melvin R. Novick ,
Paul H. Jackson  and
Dorothy T. Thayer
Melvin R. Novick
Affiliation:
Educational Testing Service
Paul H. Jackson
Affiliation:
Educational Testing Service
Dorothy T. Thayer
Affiliation:
Educational Testing Service
Get access Rights & Permissions [Opens in a new window]

Abstract

A general one-way analysis of variance components with unequal replication numbers is used to provide unbiased estimates of the true and error score variance of classical test theory. The inadequacy of the ANOVA theory is noted and the foundations for a Bayesian approach are detailed. The choice of prior distribution is discussed and a justification for the Tiao-Tan prior is found in the particular context of the “n-split” technique. The posterior distributions of reliability, error score variance, observed score variance and true score variance are presented with some extensions of the original work of Tiao and Tan. Special attention is given to simple approximations that are available in important cases and also to the problems that arise when the ANOVA estimate of true score variance is negative. Bayesian methods derived by Box and Tiao and by Lindley are studied numerically in relation to the problem of estimating true score. Each is found to be useful and the advantages and disadvantages of each are discussed and related to the classical test-theoretic methods. Finally, some general relationships between Bayesian inference and classical test theory are discussed.

Type
Original Paper
Information
Psychometrika , Volume 36 , Issue 3 , September 1971 , pp. 261 - 288
Copyright
Copyright © 1971 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

*

Supported in part by the National Institute of Child Health and Human Development under Research Grant 1 PO1 HDO1762. Reproduction, translation, use or disposal by or for the United States Government is permitted.

References

Box, G. E. P. and Tiao, G. C. A Bayesian approach to the importance of assumptions applied to the comparison of variances. Biometrika, 1964, 51, 153–167CrossRefGoogle Scholar
Box, G. E. P. and Tiao, G. C. A note on criterion robustness and inference robustness. Biometrika, 1964, 51, 169–173CrossRefGoogle Scholar
Box, G. E. P. and Tiao, G. C. Bayesian estimation of means for the random effect model. J. Amer. Statist. Assoc., 1968, 63, 174–181CrossRefGoogle Scholar
Davies, O. L. et al. Statistical Methods in Research and Production, 3rd edition, Edinburgh: Oliver and Boyd, 1961Google Scholar
Hill, B. M. Inference about variance components in the one-way model. J. Amer. Statist. Assoc., 1965, 60, 806–825CrossRefGoogle Scholar
Hill, B. M. Correlated errors in the random model. J. Amer. Statist. Assoc., 1967, 62, 1385–1385CrossRefGoogle Scholar
James, W. and Stein, C. Estimation with quadratic loss. In Neyman, J. (Eds.), Proceedings of the Fourth Berkeley Symposium on Probability and Statistics. Vol. I, 1961, Berkeley: University of California PressGoogle Scholar
Jeffreys, H. Theory of Probability, 3rd edition, Oxford: The Clarendon Press, 1961Google Scholar
Kelley, T. L. Fundamentals of Statistics, 1927, Cambridge: Harvard University PressGoogle Scholar
Klotz, J. H., Milton, R. C., and Zacks, S. Mean square efficiency of estimators of variance components. J. Amer. Stat. Assoc., 1969, 64, 1383–1402CrossRefGoogle Scholar
Kristof, W. The statistical theory of stepped-up reliability coefficients when a test has been divided into several equivalent parts. Psychometrika, 1963, 28, 221–238CrossRefGoogle Scholar
Kristof, W. Estimation of true score and error variance for tests under various equivalence assumptions. Psychometrika, 1969, 34, 489–508CrossRefGoogle Scholar
Lindley, D. V. Introduction to Probability and Statistics, Part 2, 1965, Cambridge: University PressCrossRefGoogle Scholar
Novick, M. R. Multiparameter Bayesian indifference procedures. J. Royal Statist. Soc., 1969, 31, 29–64 (with discussion)CrossRefGoogle Scholar
Pearson, K. et al. Tables of the Incomplete Beta-Function, 1968, Cambridge: University PressGoogle Scholar
Stein, C. M. Confidence sets for the mean of a multivariate normal distribution. J. Royal Statist. Soc., 1962, 24, 265–296CrossRefGoogle Scholar
Tiao, G. C. and Tan, W. Y. Bayesian analysis of random-effect models in the analysis of variance. I. Posterior distribution of variance-components. Biometrika, 1965, 52, 37–53CrossRefGoogle Scholar