Hostname: page-component-5f745c7db-8qdnt Total loading time: 0 Render date: 2025-01-06T21:14:59.575Z Has data issue: true hasContentIssue false
  • English
  • Français

On the Asymptotic Distribution of Pearson’s X2 in Cross-Validation Samples

Published online by Cambridge University Press:  01 January 2025

Harry Joe  and
Albert Maydeu-Olivares
Harry Joe
Affiliation:
University of British Columbia
Albert Maydeu-Olivares*
Affiliation:
University of Barcelona and Instituto de Empresa
*
Requests for reprints should be sent to Albert Maydeu-Olivares, Faculty of Psychology, University of Barcelona, P. Valle de Hebrón, 171, 0835 Barcelona, Spain. E-mail: amaydeu@ub.edu.
Get access Rights & Permissions [Opens in a new window]

Abstract

In categorical data analysis, two-sample cross-validation is used not only for model selection but also to obtain a realistic impression of the overall predictive effectiveness of the model. The latter is of particular importance in the case of highly parametrized models capable of capturing every idiosyncracy of the calibrating sample. We show that for maximum likelihood estimators or other asymptotically efficient estimators Pearson’s X2 is not asymptotically chi-square in the two-sample cross-validation framework due to extra variability induced by using different samples for estimation and goodness-of-fit testing. We propose an alternative test statistic, X2xval, obtained as a modification of X2 which is asymptotically chi-square with C - 1 degrees of freedom in cross-validation samples. Stochastically, X2xval≤ X2. Furthermore, the use of X2 instead of X2xval with a χ2C - 1 reference distribution may provide an unduly poor impression of fit of the model in the cross-validation sample.

Keywords

contingency tablesitem response theory modelinglatent class analysisquadratic form statisticsgoodness-of-fit
Type
Original Paper
Information
Psychometrika , Volume 71 , Issue 3 , September 2006 , pp. 587 - 592
Copyright
Copyright © 2006 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

This paper is dedicated to the memory of Michael V. Levine.

References

Agresti, A. (2002). Categorical data dnalysis, (2nd ed.). Dordrecht: Wiley.CrossRefGoogle Scholar
Bishop, Y.M.M., Fienberg, S.E., Holland, P.W. (1975). Discrete multivariate analysis, Cambridge, MA: MIT Press.Google Scholar
Bock, R.D., Lieberman, M. (1970). Fitting a response model for n dichotomously scored items. Psychometrika, 35, 179–197.CrossRefGoogle Scholar
Browne, M.W. (2000). Cross-validation methods. Journal of Mathematical Psychology, 44, 108–132.CrossRefGoogle ScholarPubMed
Chernyshenko, O.S., Stark, S., Chan, K.-Y., Drasgow, F., Williams, B. (2001). Fitting item response theory models to two personality inventories: Issues and insights. Multivariate Behavioral Research, 36, 523–562.CrossRefGoogle ScholarPubMed
Collins, L.M., Graham, J.W., Long, J.D., Hansen, W.B. (1994). Crossvalidation of latent class models of early substance use onset. Multivariate Behavioral Research, 29, 165–183.CrossRefGoogle ScholarPubMed
Drasgow, F., Levine, M.V., Tsien, S., Williams, B., Mead, A. (1995). Fitting polytomous item response theory models to multiple-choice tests. Applied Psychological Measurement, 19, 143–165.CrossRefGoogle Scholar
Du Toit, M. (2003). IRT from SSI, Lincolnwood, IL: Scientific Software International.Google Scholar
Koehler, K., Larntz, K. (1980). An empirical investigation of goodness-of-fit statistics for sparse multinomials. Journal of the American Statistical Association, 75, 336–344.CrossRefGoogle Scholar
Levine, M.V. (1984). An introduction to multilinear formula score theory. Measurement series 84-4. Champaign, IL: Model Based Measurement Laboratory.Google Scholar
Lord, F.M., Novick, M.R. (1968). Statistical theories of mental test scores, Reading, MA: Addison-Wesley.Google Scholar
Maydeu-Olivares, A. (2005). Further empirical results on parametric vs. non-parametric IRT modeling of Likert-type personality data. Multivariate Behavioral Research, 40, 275–293.CrossRefGoogle Scholar
Thissen, D., Chen, W.-H., Bock, R.D. (2003). Multilog (version 7) [Computer software], Lincolnwood, IL: Scientific Software International.Google Scholar
Zucchini, W. (2000). An introduction to model selection. Journal of Mathematical Psychology, 44, 41–61.CrossRefGoogle ScholarPubMed