Hostname: page-component-cd9895bd7-7cvxr Total loading time: 0 Render date: 2025-01-05T20:58:30.141Z Has data issue: false hasContentIssue false
  • English
  • Français

Hypothesis Testing of the Q-matrix

Published online by Cambridge University Press:  01 January 2025

Yuqi Gu ,
Jingchen Liu ,
Gongjun Xu  and
Zhiliang Ying
Yuqi Gu
Affiliation:
University of Michigan
Jingchen Liu
Affiliation:
Columbia University
Gongjun Xu*
Affiliation:
University of Michigan
Zhiliang Ying
Affiliation:
Columbia University
*
Correspondence should be made to Gongjun Xu, Department of Statistics, University of Michigan, 456 West Hall,1085 South University, Ann Arbor, MI 48109, USA. Email: gongjun@umich.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

The recent surge of interests in cognitive assessment has led to the development of cognitive diagnosis models. Central to many such models is a specification of the Q-matrix, which relates items to latent attributes that have natural interpretations. In practice, the Q-matrix is usually constructed subjectively by the test designers. This could lead to misspecification, which could result in lack of fit of the underlying statistical model. To test possible misspecification of the Q-matrix, traditional goodness of fit tests, such as the Chi-square test and the likelihood ratio test, may not be applied straightforwardly due to the large number of possible response patterns. To address this problem, this paper proposes a new statistical method to test the goodness fit of the Q-matrix, by constructing test statistics that measure the consistency between a provisional Q-matrix and the observed data for a general family of cognitive diagnosis models. Limiting distributions of the test statistics are derived under the null hypothesis that can be used for obtaining the test p-values. Simulation studies as well as a real data example are presented to demonstrate the usefulness of the proposed method.

Keywords

Q-matrixdiagnostic classification modelshypothesis testing
Type
Original Paper
Information
Psychometrika , Volume 83 , Issue 3 , September 2018 , pp. 515 - 537
Copyright
Copyright © The Psychometric Society 2018

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Electronic supplementary material The online version of this article (https://doi.org/10.1007/s11336-018-9629-6) contains supplementary material, which is available to authorized users.

References

Bartholomew, D. J., &Tzamourani, P. (1999.The goodness of fit of latent trait models in attitude measurement.)Sociological Methods & Research, 27(4),525–546.CrossRefGoogle Scholar
Cai, L.,Maydeu-Olivares, A.,Coffman, D. L., &Thissen, D. (2006). Limited-information goodness-of-fit testing of item response theory models for sparse 2p\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$2^p$$\end{document} tables.British Journal of Mathematical and Statistical Psychology, 59(1),173–194.CrossRefGoogle Scholar
Chen, Y.,Liu, J.,Xu, G., &Ying, Z. (2015). Statistical analysis of Q\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$Q$$\end{document}-matrix based diagnostic classification models.Journal of the American Statistical Association, 110,850–866.CrossRefGoogle Scholar
Chiu, C-Y. (2013). Statistical refinement of the Q\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$Q$$\end{document}-matrix in cognitive diagnosis.Applied Psychological Measurement, 37,598–618.CrossRefGoogle Scholar
Chiu, C.,Douglas, J., &Li, X. (2009.Cluster analysis for cognitive diagnosis: Theory and applications.)Psychometrika, 74(4),633–665.CrossRefGoogle Scholar
de la Torre, J. (2008). An empirically-based method of Q\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$Q$$\end{document}-matrix validation for the DINA model: Development and applications.Journal of Educational Measurement, 45 343–362.CrossRefGoogle Scholar
de la Torre, J. (2011.The generalized DINA model framework.)Psychometrika, 76(2),179–199.CrossRefGoogle Scholar
de la Torre, J., &Chiu, C.-Y. (2016). A general method of empirical Q\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$Q$$\end{document}-matrix validation.Psychometrika, 81(2),253–273.CrossRefGoogle Scholar
de la Torre, J., &Douglas, J. (2004.Higher order latent trait models for cognitive diagnosis.)Psychometrika, 69 333–353.CrossRefGoogle Scholar
DeCarlo, L. T. (2011.On the analysis of fraction subtraction data: The DINA model, classification, latent class sizes, and the Q-matrix.)Applied Psychological Measurement, 35,8–26.CrossRefGoogle Scholar
DeCarlo, L. T. (2012). Recognizing uncertainty in the Q\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$Q$$\end{document}-matrix via a bayesian extension of the DINA model.Applied Psychological Measurement, 36(6),447–468.CrossRefGoogle Scholar
Dempster, A. P.,Laird, N. M., &Rubin, D. B. (1977.Maximum likelihood from incomplete data via EM algorithm.)Journal of the Royal Statistical Society Series B-Methodological, 39(1),1–38.CrossRefGoogle Scholar
DiBello, L.,Stout, W., &Roussos, L. (1995.Unified cognitive psychometric assessment likelihood-based classification techniques.)Nichols, P. D.,Chipman, S. F., &Brennan, R. L. Cognitively diagnostic assessment, 361–390.Hillsdale, NJ:Erlbaum.Google Scholar
Gu, Y., & Xu, G. (2018). Partial identifiability of restricted latent class models. arXiv preprint arXiv:1803.04353.Google Scholar
Hartz, S. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. Doctoral Dissertation, University of Illinois, Urbana-Champaign.Google Scholar
Henson, R., & Templin, J. (2005). Hierarchical log-linear modeling of the skill joint distribution. Technical report, External Diagnostic Research Group.Google Scholar
Henson, R. A.,Templin, J. L., &Willse, J. T. (2009.Defining a family of cognitive diagnosis models using log-linear models with latent variables.)Psychometrika, 74(2),191–210.CrossRefGoogle Scholar
Junker, B., &Sijtsma, K. (2001.Cognitive assessment models with few assumptions, and connections with nonparametric item response theory.)Applied Psychological Measurement, 25,258–272.CrossRefGoogle Scholar
Lehmann, E. L., &Romano, J. P. (2006)Testing statistical hypotheses.Berlin:Springer.Google Scholar
Leighton, J. P.,Gierl, M. J., &Hunka, S. M. (2004.The attribute hierarchy model for cognitive assessment: A variation on Tatsuoka’s rule-space approach.)Journal of Educational Measurement, 41,205–237.CrossRefGoogle Scholar
Liu, J., Xu, G., &Ying, Z. (2012). Data-driven learning of Q\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$Q$$\end{document}-matrix.Applied Psychological Measurement, 36(7),548–564.CrossRefGoogle Scholar
Liu, J.,Xu, G., &Ying, Z. (2013). Theory of self-learning Q\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$Q$$\end{document}-matrix.Bernoulli, 19(5A),1790–1817.Google Scholar
Maydeu-Olivares, A. (2001.Limited information estimation and testing of thurstonian models for paired comparison data under multiple judgment sampling.)Psychometrika, 66(2),209–227.CrossRefGoogle Scholar
Maydeu-Olivares, A., &Joe, H. (2005). Limited-and full-information estimation and goodness-of-fit testing in 2n\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$2^n$$\end{document} contingency tables: A unified framework.Journal of the American Statistical Association, 100(471),1009–1020.CrossRefGoogle Scholar
Roussos, L. A.,Templin, J. L., &Henson, R. A. (2007.Skills diagnosis using IRT-based latent class models.)Journal of Educational Measurement, 44,293–311.CrossRefGoogle Scholar
Rupp, A. (2002.Feature selection for choosing and assembling measurement models: A building-block-based organization.)Psychometrika, 2,311–360.Google Scholar
Rupp, A., &Templin, J. (2008). Effects of q\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$q$$\end{document}-matrix misspecification on parameter estimates and misclassification rates in the dina model.Educational and Psychological Measurement, 68 78–98.CrossRefGoogle Scholar
Rupp, A., &Templin, J. (2008.Unique characteristics of diagnostic classification models: A comprehensive review of the current state-of-the-art.)Measurement: Interdisciplinary Research and Perspective, 6,219–262.Google Scholar
Rupp, A., Templin, J., &Henson, R. A. (2010)Diagnostic measurement: Theory, methods, and applications.New York City:Guilford Press.Google Scholar
Sen, B.,Banerjee, M., &Woodroofe, M.et.al (2010.Inconsistency of bootstrap: The Grenander estimator.)The Annals of Statistics, 38(4),1953–1977.CrossRefGoogle Scholar
Sen, B., &Xu, G. (2015.Model based bootstrap methods for interval censored data.)Computational Statistics & Data Analysis, 81 121–129.CrossRefGoogle Scholar
Stout, W. (2007.Skills diagnosis using IRT-based continuous latent trait models.)Journal of Educational Measurement, 44,313–324.CrossRefGoogle Scholar
Tatsuoka, K. (1985.A probabilistic model for diagnosing misconceptions in the pattern classification approach.)Journal of Educational Statistics, 12,55–73.CrossRefGoogle Scholar
Tatsuoka, K. (1990). Toward an integration of item-response theory and cognitive error diagnosis. In N. Frederiksen, R. Glaser, A. Lesgold, & M. Shafto (Eds.), Diagnostic monitoring of skill and knowledge acquisition, (pp. 453–488).Google Scholar
Tatsuoka, C. (2002.Data-analytic methods for latent partially ordered classification models.)Applied Statistics (JRSS-C), 51,337–350.Google Scholar
Tatsuoka, C. (2005.Corrigendum: Data analytic methods for latent partially ordered classification models.)Journal of the Royal Statistical Society: Series C (Applied Statistics), 54(2),465–467.Google Scholar
Tatsuoka, K. (2009). Cognitive assessment: An introduction to the rule space method.Boca Raton:CRC Press.CrossRefGoogle Scholar
Templin, J. (2006). CDM: Cognitive diagnosis modeling with Mplus. Available from http://jtemplin.myweb.uga.edu/cdm/cdm.html.Google Scholar
Templin, J., He, X., Roussos, L., & Stout, W. (2003). The pseudo-item method: A simple technique for analysis of polytomous data with the fusion model. Technical report, External Diagnostic Research Group.Google Scholar
Templin, J., &Henson, R. (2006.Measurement of psychological disorders using cognitive diagnosis models.)Psychological Methods, 11,287–305.CrossRefGoogle ScholarPubMed
Tollenaar, N., &Mooijaart, A. (2003.Type I errors and power of the parametric bootstrap goodness-of-fit test: Full and limited information.)British Journal of Mathematical and Statistical Psychology, 56(2),271–288.CrossRefGoogle ScholarPubMed
Van der Vaart, A. W. (2000). Asymptotic statistics.Cambridge:Cambridge university press.Google Scholar
von Davier, M. (2005). A general diagnosis model applied to language testing data. Research report, Educational Testing Service.Google Scholar
von Davier, M. (2008.A general diagnostic model applied to language testing data.)British Journal of Mathematical and Statistical Psychology, 61,287–307.CrossRefGoogle ScholarPubMed
Xu, G. (2017.Identifiability of restricted latent class models with binary responses.)The Annals of Statistics, 45(2),675–707.CrossRefGoogle Scholar
Xu, G., & Shang, Z. (2018). Identifying latent structures in restricted latent class models. Journal of the American Statistical Association. https://doi.org/10.1080/01621459.2017.1340889.CrossRefGoogle Scholar
Zhang, S. S, DeCarlo, L. T., & Ying, Z. (2013). Non-identifiability, equivalence classes, and attribute-specific classification in Q-matrix based cognitive diagnosis models. ArXiv e-prints.Google Scholar
Supplementary material: File

Gu et al. supplementary material

Gu et al. supplementary material
Download Gu et al. supplementary material(File)
File 1.2 MB