Hostname: page-component-5f745c7db-xx4dx Total loading time: 0 Render date: 2025-01-06T08:52:46.987Z Has data issue: true hasContentIssue false
  • English
  • Français

The Role of Secondary Covariates when Estimating latent Trait Population Distributions

Published online by Cambridge University Press:  01 January 2025

Neal Thomas
Neal Thomas*
Affiliation:
Datametrics, Inc. and Bristol-Myers Squibb
*
Requests for reprints should be sent to Neal Thomas, 61 Dream Lake Drive, Madison CT 06443-1600. E-Mail: snthomas99@yahoo.com
Get access Rights & Permissions [Opens in a new window]

Abstract

The U.S. National Assessment of Educational Progress (NAEP), the Third International Mathematics and Science Study (TIMSS), and the U.S. Adult Literacy Survey collect probability samples of students (or adults) who are administered brief examinations in subject areas such as mathematics and reading (cognitive variables), along with background demographic (primary) and educational environment (secondary) questions. The demographic questions are used in the primary reporting, while the numerous “explanatory” secondary variables, or “covariates”, are only directly utilized in subsequent secondary analyses. The covariates are also used indirectly to create the plausible values (multiple imputations) that are an integral part of analyses because of the use of sparse matrix sampling of cognitive items. The improvement in the precision of the primary reporting due to the inclusion of the covariates is assessed here and contrasted with the precision of reporting using plausible values created using only the primary demographic variables.

The results demonstrate that the improvement in precision depends on the matrix sampling designs for the cognitive assessments. The improvements range from essentially none for the most common designs, to moderate for some less common designs. Consequently, two potential changes in the reporting procedures that could improve the statistical and operational efficiency of primary reporting are (a) eliminate or reduce the collection of covariates and increase the number of cognitive items, (b) to avoid delays, eliminate the covariates from the creation of plausible values used for the primary reports, but include them later when creating public-use files for secondary analyses. The potential improvements in statistical and operational efficiency must be weighed against the intrinsic interest in the covariates, and the potential for small discrepancies in the primary and secondary reporting.

Keywords

national assessment (NAEP)item response theorymatrix samplingmultiple imputationsample surveyscovariate design
Type
Articles
Information
Psychometrika , Volume 67 , Issue 1 , March 2002 , pp. 33 - 48
Copyright
Copyright © 2002 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Thanks to Donald Rubin, Robert Mislevy, and John Barnard for their helpful comments and computing assistance. This work was supported by NCES Grant 84.902B980011.

References

Beaton, A., & Zwick, R. (1992). Overview of the National Assessment of Educational Progress. Journal of Educational Statistics, 17, 95–109.Google Scholar
Chang, H., & Stout, W. (1993). The asymptotic posterior normality of the latent trait in an IRT model. Psychometrika, 58, 37–52.CrossRefGoogle Scholar
Cochran, W.G. (1977). Sampling techniques. New York, NY: Wiley.Google Scholar
Forsyth, R., Hambleton, R., Linn, R., Mislevy, R., & Yen, W. (1996). 1996 Design/Feasibility Team: Report to the National Governing Board. Washington, D.C.: National Governing Board.Google Scholar
Johnson, E.G., & Carlson, J. (1994). The NAEP 1992 Technical Report. Washington D.C.: National Center for Educational Statistics.Google Scholar
Johnson, E.G., Mislevy, R., & Thomas, N. (1994). Scaling procedures. In Johnson, E.G., & Allen, N.L. (Eds.), The NAEP 1992 Technical Report (pp. 241–256). Washington D.C.: National Center for Educational Statistics.Google Scholar
Johnson, E.G., & Rust, K. (1992). Population inferences & variance estimation for NAEP Data. Journal of Educational Statistics, 17, 175–190.Google Scholar
Johnson, J.G. (1984). Econometric Methods. New York, NY: McGraw-Hill.Google Scholar
Little, R., & Rubin, D. (1987). Statistical analysis with missing data. New York, NY: Wiley.Google Scholar
Mislevy, R. (1991). Randomization-based inferences about latent variables from complex samples. Psychometrika, 56, 177–196.CrossRefGoogle Scholar
Mislevy, R. (1992). Scaling procedures. In Johnson, E.G., & Allen, N.L. (Eds.), The NAEP 1990 Technical Report (pp. 199–214). Washington D.C.: National Center for Educational Statistics.Google Scholar
Mislevy, R., & Bock, D. (1982). BILOG: Item analysis & test scoring with binary logistic models. Mooresville, IN: Scientific Software.Google Scholar
Mislevy, R., Johnson, E.G., & Muraki, E. (1992). Estimating population characteristics from sparse matrix samples of item responses. Journal of Educational Statistics, 17, 131–154.Google Scholar
Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurements, 16, 159–176.CrossRefGoogle Scholar
Rao, C.R. (1965). Linear statistical inference and its applications. New York, NY: Wiley.Google Scholar
Rubin, D. (1987). Multiple imputation for nonresponse in surveys. New York, NY: Wiley.CrossRefGoogle Scholar
Thomas, N. (1993). Asymptotic corrections for multivariate posterior moments with factored likelihood functions. Journal of Computational and Graphical Statistics, 2, 309–322.CrossRefGoogle Scholar
Thomas, N. (2000). The role of secondary covariates in NAEP primary reporting. Washington D.C.: National Center for Educational Statistics.Google Scholar
Thomas, N., & Gan, N. (1998). Generating multiple imputations for matrix sampling data analyzed with item response models. Journal of Educational and Behavioral Statistics, 23, 425–445.Google Scholar
Zeger, L., & Thomas, N. (1997). Efficient matrix sampling instruments for correlated latent traits: Examples from the National Assessment of Educational Progress. Journal of the American Statistical Association, 92, 416–438.CrossRefGoogle Scholar