Hostname: page-component-5f745c7db-xx4dx Total loading time: 0 Render date: 2025-01-06T08:34:54.288Z Has data issue: true hasContentIssue false
  • English
  • Français

Randomization-Based Inference about Latent Variables from Complex Samples

Published online by Cambridge University Press:  01 January 2025

Robert J. Mislevy
Robert J. Mislevy*
Affiliation:
Educational Testing Service
*
Requests for reprints should be sent to Robert J. Mislevy, Educational Testing Service, Princeton, NJ 08541.
Get access Rights & Permissions [Opens in a new window]

Abstract

Standard procedures for drawing inferences from complex samples do not apply when the variable of interest θ cannot be observed directly, but must be inferred from the values of secondary random variables that depend on θ stochastically. Examples are proficiency variables in item response models and class memberships in latent class models. Rubin's “multiple imputation” techniques yield approximations of sample statistics that would have been obtained, had θ been observable, and associated variance estimates that account for uncertainty due to both the sampling of respondents and the latent nature of θ. The approach is illustrated with data from the National Assessment for Educational Progress.

Keywords

complex samplesitem response theorylatent structuremissing datamultiple imputationNational Assessment of Educational Progresssample surveys
Type
Original Paper
Information
Psychometrika , Volume 56 , Issue 2 , June 1991 , pp. 177 - 196
Copyright
Copyright © 1991 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

This research was supported by Grant No. NIE-G-83-0011 of the Office for Educational Research and Improvement, Center for Education Statistics, and Contract No. N00014-88-K-0304, R&T 4421552 from the Cognitive Sciences Program, Cognitive and Neural Sciences Division, Office of Naval Research. It does not necessarily reflect the views of either agency. I am grateful to R. Darrell Bock for calling my attention to the applicability of multiple imputation to the assessment setting; to Albert Beaton and Eugene Johnson for enlightening discussions on the topic; and to Henry Braun, Ben King, Debra Kline, Gary Phillips, Paul Rosenbaum, Don Rubin, John Tukey, Ming-Mei Wang, Kentaro Yamamoto, Rebecca Zwick, and two anonymous reviewers for comments on earlier drafts. Example 4 is based on the analysis of the 1984 National Assessment for Educational Progress reading survey, carried out at Educational Testing Service through the tireless efforts of too many people to mention by name, under the direction of Albert Beaton, Director of NAEP Data Analyses. David Freund, Bruce Kaplan, and Jennifer Nelson conducted additional analyses of the 1984 and 1988 data for the example.

References

Andersen, E. B., & Madsen, M. (1977). Estimating the parameters of a latent population distribution. Psychometrika, 42, 357–374.CrossRefGoogle Scholar
Beaton, A. E. (1987). The NAEP 1983/84 technical report, Princeton: Educational Testing Service.Google Scholar
Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: An application of an EM algorithm. Psychometrika, 46, 443–459.CrossRefGoogle Scholar
Bock, R. D., Mislevy, R. J., & Woodson, C. E. M. (1982). The next stage in educational assessment. Educational Researcher, 11, 4–11.CrossRefGoogle Scholar
Cassel, C. M., Särndal, C. E., & Wretman, J. H. (1977). Foundations of inference in survey sampling, New York: Wiley.Google Scholar
Cochran, W. G. (1977). Sampling techniques, New York: Wiley.Google Scholar
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm (with discussion). Journal of the Royal Statistical Society, Series B, 39, 1–38.CrossRefGoogle Scholar
Ford, B. L. (1983). An overview of hot-deck procedures. In Madow, W. G., Olkin, I. & Rubin, D. B. (Eds.), Incomplete data in sample surveys, Volume 2, Theory and bibliographies (pp. 185–207). New York: Academic Press.Google Scholar
Goldstein, H., & McDonald, R. P. (1988). A general model for the analysis of multilevel data. Psychometrika, 53, 455–467.CrossRefGoogle Scholar
Johnson, E. G. (1987). Estimation of uncertainty due to sampling variation. In Beaton, A. E. (Eds.), The NAEP 1983/84 technical report (pp. 505–512). Princeton: Educational Testing Service.Google Scholar
Kelley, T. L. (1923). Statistical method, New York: Macmillan.Google Scholar
Kirsch, I. S., & Jungeblut, A. (1986). Literacy: Profile of America's young adults, Princeton, NJ: National Assessment of Education Progress.Google Scholar
Laird, N. M. (1978). Nonparametric maximum likelihood estimation of a mixing distribution. Journal of the American Statistical Association, 73, 805–811.CrossRefGoogle Scholar
Little, R. J. A., & Rubin, D. B. (1987). Statistical analysis with missing data, New York: Wiley.Google Scholar
Lord, F. M. (1965). A strong true-score theory, with applications. Psychometrika, 30, 239–270.CrossRefGoogle Scholar
Lord, F. M. (1969). Estimating true score distributions in psychological testing (An empirical Bayes problem). Psychometrika, 34, 259–299.CrossRefGoogle Scholar
Mislevy, R. J. (1984). Estimating latent distributions. Psychometrika, 49, 359–381.CrossRefGoogle Scholar
Mislevy, R. J. (1985). Estimation of latent group effects. Journal of the American Statistical Association, 80, 993–997.CrossRefGoogle Scholar
Mislevy, R. J., & Bock, R. D. (1983). BILOG: Item analysis and test scoring with binary logistic models [computer program], Mooresville, IN: Scientific Software.Google Scholar
Mislevy, R. J., & Sheehan, K. M. (1987). Marginal estimation procedures. In Beaton, A. E. (Eds.), The NAEP 1983/84 technical report (pp. 293–360). Princeton: Educational Testing Service.Google Scholar
Muthén, B., & Satorra, A. (1989). Multilevel aspects of varying parameters in structural models. In Bock, R. D. (Eds.), Multilevel analysis of educational data (pp. 87–99). San Diego: Academic Press.Google Scholar
National Assessment of Educational Progress (1985). The reading report card: Progress toward excellence in our schools, Princeton: Educational Testing Service.Google Scholar
Rigdon, S. E., & Tsutakawa, R. K. (1983). Parameter estimation in latent trait models. Psychometrika, 48, 567–574.CrossRefGoogle Scholar
Rubin, D. B. (1977). Formalizing subjective notions about the effect of nonrespondents in sample surveys. Journal of the American Statistical Association, 72, 538–543.CrossRefGoogle Scholar
Rubin, D. B. (1987). Multiple imputation for nonresponse in surveys, New York: Wiley.CrossRefGoogle Scholar
Sanathanan, L., & Blumenthal, N. (1978). The logistic model and estimation of latent structure. Journal of the American Statistical Association, 73, 794–798.CrossRefGoogle Scholar
Sheehan, K. M. (1985). M-GROUP: Estimation of group effects in multivariate models [computer program], Princeton: Educational Testing Service.Google Scholar
Tanner, M., & Wong, W. (1987). The calculation of posterior distributions by data augmentation (with discussion). Journal of the American Statistical Association, 82, 528–550.CrossRefGoogle Scholar
Zwick, R. (1987). Assessing the dimensionality of NAEP reading data. Journal of Educational Measurement, 24, 293–308.CrossRefGoogle Scholar