Hostname: page-component-745bb68f8f-hvd4g Total loading time: 0 Render date: 2025-01-07T17:53:59.996Z Has data issue: false hasContentIssue false
  • English
  • Français

An Empirical Q-Matrix Validation Method for the Polytomous G-DINA Model

Published online by Cambridge University Press:  01 January 2025

Jimmy de la Torre ,
Xue-Lan Qiu  and
Kevin Carl Santos
Jimmy de la Torre*
Affiliation:
The University of Hong Kong
Xue-Lan Qiu
Affiliation:
The University of Hong Kong
Kevin Carl Santos
Affiliation:
University of the Philippines
*
Correspondence should be made to Jimmy de la Torre, Faculty of Education, The University of Hong Kong, Pokfulam Road, Pok Fu Lam, Hong Kong. Email: j.delatorre@hku.hk
Get access Rights & Permissions [Opens in a new window]

Abstract

A number of empirically based Q-matrix validation methods are available in the literature, all of which were developed for cognitive diagnosis models (CDMs) involving dichotomous attributes. However, in many applications, it is more instructionally relevant to classify students into more than two categories (e.g., no mastery, basic mastery, and advanced mastery). To extend the practical utility of CDMs, methods for validating the Q-matrix for CDMs that measure polytomous attributes are needed. This study focuses on validating the Q-matrix of the generalized deterministic input, noisy, “and” gate model for polytomous attributes (pG-DINA). The pGDI, an extension of the G-DINA model discrimination index, is proposed for polytomous attributes. The pGDI serves as the basis of a validation method that can be used not only to identify potential misspecified q-entries, but also to suggest more appropriate attribute-level specifications. The theoretical properties of the pGDI are underpinned by several mathematical proofs, whereas its practical viability is examined using simulation studies covering various conditions. The results show that the method can accurately identify misspecified q-entries and suggest the correct attribute-level specifications, particularly when high-quality items are involved. The pGDI is applied to a proportional reasoning test that measures several polytomous attributes.

Keywords

cognitive diagnosis modelsG-DINAQ-matrix validationpolytomous attributes
Type
Theory and Methods
Information
Psychometrika , Volume 87 , Issue 2: Special Issue on Forecasting with Intensive Longitudinal Data , June 2022 , pp. 693 - 724
Copyright
Copyright © 2021 The Author(s) under exclusive licence to The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Chen, J., &de la Torre, J.(2013).A general cognitive diagnosis model for expert-defined polytomous attributes.Applied Psychological Measurement,37,419–437.CrossRefGoogle Scholar
Chen, Y.,Culpepper, S. A.,Chen, Y., &Douglas, J.(2018).Bayesian estimation of the DINA Q.Psychometrika,83,89–108.CrossRefGoogle ScholarPubMed
Chen, Y.,Liu, J.,Xu, G., &Ying, Z.(2015).Statistical analysis of Q-matrix based diagnostic classification models.Journal of the American Statistical Association,110,850–866.CrossRefGoogle Scholar
Chiu, C.-Y.(2013).Statistical refinement of the Q-matrix in cognitive diagnosis.Applied Psychological Measurement,37,598–618.CrossRefGoogle Scholar
Culpepper, S. A.(2019).Estimating the cognitive diagnosis Q matrix with expert knowledge: Application to the fraction-subtraction dataset.Psychometrika,84,333–357.CrossRefGoogle Scholar
DeCarlo, L.(2012).Recognizing uncertainty in the Q-Matrix via a Bayesian extension of the DINA Model.Applied Psychological Measurement,36,447–468.CrossRefGoogle Scholar
de la Torre, J.(2008).An empirically-based method of Q-matrix validation for the DINA model: Development and applications.Journal of Educational Measurement,45,343–362.CrossRefGoogle Scholar
de la Torre, J.(2011).The generalized DINA model framework.Psychometrika,76,179–199.CrossRefGoogle Scholar
de la Torre, J., &Chiu, C.-Y.(2016).A general method of empirical Q-matrix validation.Psychometrika,81,253–273.CrossRefGoogle ScholarPubMed
de la Torre, J., &Douglas, J. A.(2004).Higher-order latent trait models for cognitive diagnosis.Psychometrika,63,333–353.CrossRefGoogle Scholar
de la Torre, J., & Ma, W. (2016). Cognitive diagnosis modeling: A general framework approach and its implementation in R. A Short Course at the Fourth Conference on Statistical Methods in Psychometrics, Columbia University, New York.Google Scholar
Gu, Y., &Xu, G.(2021).Sufficient and necessary conditions for the identifiability of the Q-matrix.Statistica Sinica,31,449–472.Google Scholar
Hartz, S. M. (2002). A Bayesian framework for the Unified Model for assessing cognitive abilities: Blending theory with practicality (Unpublished doctoral dissertation). University of Illinois at Urbana-Champaign.Google Scholar
Junker, B. W., &Sijtsma, K.(2001).Cognitive assessment models with few assumptions, and connections with nonparametric item response theory.Applied Psychological Measurement,25,258–272.CrossRefGoogle Scholar
Liu, J.,Xu, G., &Ying, Z.(2012).Data-driven learning of Q-matrix.Applied Psychological Measurement,36,548–564.CrossRefGoogle ScholarPubMed
Maris, E.(1999).Estimating multiple classification latent class models.Psychometrika,64,187–212.CrossRefGoogle Scholar
Ma, W., &de la Torre, J.(2019).GDINA: The generalized DINA model framework.R Package Version,2(3),2.Google Scholar
Ma, W., &de la Torre, J.(2020).An empirical Q-matrix validation method for the sequential generalized DINA model.British Journal of Mathematical and Statistical Psychology,73,142–163.CrossRefGoogle ScholarPubMed
Nájera, P., Sorrel, M. A., de la Torre, J., & Abad, F. J. (2020). Improving robustness in Q-matrix validation using an iterative and dynamic procedure. Applied Psychological Measurement, 44, 431–446.CrossRefGoogle Scholar
Nájera, P.,Sorrel, M. A., &Abad, F. J.(2019).Reconsidering cutoff points in the general method of empirical Q-matrix validation.Educational and Psychological Measurement,79,727–753.CrossRefGoogle ScholarPubMed
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometric Monograph Supplement No. 17. Richmond, VA: Psychometric Society.Google Scholar
Tatsuoka, K. K.(1983).Rule space: An approach for dealing with misconceptions based on item response theory.Journal of Educational Measurement,20,345–354.CrossRefGoogle Scholar
Templin, J. L., &Henson, R.(2006).Measurement of psychological disorders using cognitive diagnosis models.Psychological Methods,11,287–305.CrossRefGoogle ScholarPubMed
Terzi, R., &de la Torre, J.(2018).An iterative method for empirically-based Q-matrix validation.International Journal of Assessment Tools in Education,5,248–262.CrossRefGoogle Scholar
Tjoe, H., &de la Torre, J.(2013).Designing cognitively-based proportional reasoning problems as an application of modern psychological measurement models.Journal of Mathematics Education,6,17–22.Google Scholar
Tjoe, H.de la Torre, J.(2014).The identification and validation process of proportional reasoning attributes: An application of a cognitive diagnosis modeling framework.Mathematics Education Research Journal,26,237–255.CrossRefGoogle Scholar
Xu, G.(2017).Identifiability of restricted latent class models with binary responses.The Annals of Statistics,45,675–707.CrossRefGoogle Scholar
Xu, G.Shang, Z.(2018).Identifying latent structures in restricted latent class models.Journal of the American Statistical Association,113,1284–1295.CrossRefGoogle Scholar
Xu, G., &Zhang, S.(2016).Identifiability of diagnostic classification models.Psychometrika,81,625–649.CrossRefGoogle ScholarPubMed