Hostname: page-component-745bb68f8f-d8cs5 Total loading time: 0 Render date: 2025-02-06T14:52:10.678Z Has data issue: false hasContentIssue false
  • English
  • Français

Estimating the Cognitive Diagnosis Q\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\varvec{Q}$$\end{document} Matrix with Expert Knowledge: Application to the Fraction-Subtraction Dataset

Published online by Cambridge University Press:  01 January 2025

Steven Andrew Culpepper Open the ORCID record for Steven Andrew Culpepper [Opens in a new window]
Steven Andrew Culpepper*
Affiliation:
University of Illinois at Urbana-Champaign
*
Correspondence should be made to Steven Andrew Culpepper, Department of Statistics, University of Illinois at Urbana-Champaign, 725 South Wright Street, Champaign, IL61820, USA. Email: sculpepp@illinois.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

Cognitive diagnosis models (CDMs) are an important psychometric framework for classifying students in terms of attribute and/or skill mastery. The Q\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\varvec{Q}$$\end{document} matrix, which specifies the required attributes for each item, is central to implementing CDMs. The general unavailability of Q\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\varvec{Q}$$\end{document} for most content areas and datasets poses a barrier to widespread applications of CDMs, and recent research accordingly developed fully exploratory methods to estimate Q. However, current methods do not always offer clear interpretations of the uncovered skills and existing exploratory methods do not use expert knowledge to estimate Q. We consider Bayesian estimation of Q\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\varvec{Q}$$\end{document} using a prior based upon expert knowledge using a fully Bayesian formulation for a general diagnostic model. The developed method can be used to validate which of the underlying attributes are predicted by experts and to identify residual attributes that remain unexplained by expert knowledge. We report Monte Carlo evidence about the accuracy of selecting active expert-predictors and present an application using Tatsuoka’s fraction-subtraction dataset.

Keywords

exploratory cognitive diagnosis modelsgeneral diagnostic modelBayesianmultivariate regressionvariable selectionvalidationspike–slab priors
Type
Original Paper
Information
Psychometrika , Volume 84 , Issue 2 , June 2019 , pp. 333 - 357
Copyright
Copyright © 2018 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Albert, J. (1992). Bayesian estimation of normal ogive item response curves using Gibbs sampling. Journal of Educational and Behavioral Statistics, 17(3), 251–269.CrossRefGoogle Scholar
Albert, J. H., Chib, S. (1993). Bayesian analysis of binary and polychotomous response data. Journal of the American Statistical Association, 88(422), 669–679.CrossRefGoogle Scholar
Béguin, A. A., Glas, C. A. (2001). MCMC estimation and some model-fit analysis of multidimensional IRT models. Psychometrika, 66(4), 541–561.CrossRefGoogle Scholar
Celeux, G., Forbes, F., Robert, C. P., Titterington, D. M. (2006). Deviance information criteria for missing data models. Bayesian Analysis, 1, 651–673.CrossRefGoogle Scholar
Chen, Y., Culpepper, S. A. (2018). A multivariate probit model for learning trajectories with application to classroom assessment, New York: In Paper presentation at the International Meeting of the Psychometric Society.Google Scholar
Chen, Y., Culpepper, S. A., Chen, Y., Douglas, J. (2018). Bayesian estimation of the DINA Q. Psychometrika, 83, 89–108.CrossRefGoogle ScholarPubMed
Chen, Y., Liu, J., Xu, G., Ying, Z. (2015). Statistical analysis of Q-matrix based diagnostic classification models. Journal of the American Statistical Association, 110(510), 850–866.CrossRefGoogle Scholar
Chiu, C.-Y. (2013). Statistical refinement of the Q-matrix in cognitive diagnosis. Applied Psychological Measurement, 37(8), 598–618.CrossRefGoogle Scholar
Chiu, C.-Y., Douglas, J. A., Li, X. (2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74(4), 633–665.CrossRefGoogle Scholar
Chung, M. (2014). Estimating the Q-matrix for cognitive diagnosis models in a Bayesian framework (Unpublished doctoral dissertation). Columbia University.Google Scholar
Culpepper, S. A. (2015). Bayesian estimation of the DINA model with Gibbs sampling. Journal of Educational and Behavioral Statistics, 40(5), 454–476.CrossRefGoogle Scholar
Culpepper, S. A. (2016). Revisiting the 4-parameter item response model: Bayesian estimation and application. Psychometrika, 81(4), 1142–1163.CrossRefGoogle ScholarPubMed
Culpepper, S. A., & Chen, Y. (2018). Development and application of an exploratory reduced reparameterized unified model. Journal of Educational and Behavioral Statistics. https://doi.org/10.3102/1076998618791306.CrossRefGoogle Scholar
Culpepper, S. A., Hudson, A. (2018). An improved strategy for Bayesian estimation of the reduced reparameterized unified model. Applied Psychological Measurement, 42, 99–115.CrossRefGoogle ScholarPubMed
de la Torre, J. (2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45(4), 343–362.CrossRefGoogle Scholar
de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199.CrossRefGoogle Scholar
de la Torre, J., Chiu, C.-Y. (2016). A general method of empirical Q-matrix validation. Psychometrika, 81(2), 253–273.CrossRefGoogle ScholarPubMed
de la Torre, J., Douglas, J. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333–353.CrossRefGoogle Scholar
DeCarlo, L. T. (2012). Recognizing uncertainty in the Q-matrix via a Bayesian extension of the DINA model. Applied Psychological Measurement, 36, 447–468.CrossRefGoogle Scholar
George, E. I., McCulloch, R. E. (1993). Variable selection via Gibbs sampling. Journal of the American Statistical Association, 88(423), 881–889.CrossRefGoogle Scholar
Gershman, S. J., Blei, D. M. (2012). A tutorial on Bayesian nonparametric models. Journal of Mathematical Psychology, 56(1), 1–12.CrossRefGoogle Scholar
Griffiths, T. L., Ghahramani, Z. (2011). The Indian buffet process: An introduction and review. Journal of Machine Learning Research, 12, 1185–1224.Google Scholar
Gu, Y., & Xu, G. (2018). The sufficient and necessary condition for the identifiability and estimability of the DINA model. Psychometrika. https://doi.org/10.1007/s11336-018-9619-8.CrossRefGoogle Scholar
Haertel, E. H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26(4), 301–321.CrossRefGoogle Scholar
Hartz, S. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality (Unpublished doctoral dissertation). University of Illinois at Urbana-Champaign.Google Scholar
Henson, R. A., Templin, J. (2007). Importance of Q-matrix construction and its effects cognitive diagnosis model results, Chicago, IL: In Annual Meeting of the National Council on Measurement in Education.Google Scholar
Henson, R. A., Templin, J. L., Willse, J. T. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74(2), 191–210.CrossRefGoogle Scholar
Ishwaran, H., Rao, J. S. (2005). Spike and slab variable selection: Frequentist and Bayesian strategies. Annals of Statistics, 33, 730–773.CrossRefGoogle Scholar
Junker, B. W., Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258–272.CrossRefGoogle Scholar
Klein, M. F., Birenbaum, M., Standiford, S. N., & Tatsuoka, K. K. (1981). Logical error analysis and construction of tests to diagnose student “bugs” in addition and subtraction of fractions. (Tech. Rep.). University of Illinois at Urbana-Champaign, Champaign, IL.Google Scholar
Liu, J. (2017). On the consistency of Q-matrix estimation: A commentary. Psychometrika, 82(2), 523–527.CrossRefGoogle ScholarPubMed
Liu, J., Xu, G., Ying, Z. (2012). Data-driven learning of Q-matrix. Applied Psychological Measurement, 36(7), 548–564.CrossRefGoogle ScholarPubMed
Liu, J., Xu, G., Ying, Z. (2013). Theory of the self-learning Q-matrix. Bernoulli, 19 5A1790–1817.CrossRefGoogle ScholarPubMed
Maris, E. (1999). Estimating multiple classification latent class models. Psychometrika, 64(2), 187–212.CrossRefGoogle Scholar
O’Hara, R. B., Sillanpää, M. J. (2009). A review of Bayesian variable selection methods: What, how and which. Bayesian Analysis, 4(1), 85–117.Google Scholar
Rupp, A. A., Templin, J. L. (2008). The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68(1), 78–96.CrossRefGoogle Scholar
Spiegelhalter, D. J., Best, N. G., Carlin, B. P., Van Der Linde, A. (2002). Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64(4), 583–639.CrossRefGoogle Scholar
Tatsuoka, C. (2002). Data analytic methods for latent partially ordered classification models. Journal of the Royal Statistical Society: Series C (Applied Statistics), 51(3), 337–350.Google Scholar
Tatsuoka, K. K. (1984). Analysis of errors in fraction addition and subtraction problems, Computer-Based Education Research Laboratory: University of Illinois at Urbana-Champaign.Google Scholar
Tatsuoka, K. K. (1990). Toward an integration of item-response theory and cognitive error diagnosis (Tech. Rep.). University of Illinois at Urbana-Champaign, Champaign, IL.Google Scholar
Templin, J. L. & Henson, R. A. (2006). A Bayesian method for incorporating uncertainty into Q-matrix estimation in skills assessment. In Symposium conducted at the meeting of the American Educational Research Association, San Diego, CA.Google Scholar
Templin, J. L., Henson, R. A., Templin, S. E., Roussos, L. (2008). Robustness of hierarchical modeling of skill association in cognitive diagnosis models. Applied Psychological Measurement, 32, 559–574.CrossRefGoogle Scholar
von Davier, M. (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61(2), 287–307.CrossRefGoogle ScholarPubMed
Xu, G. (2017). Identifiability of restricted latent class models with binary responses. Annals of Statistics, 45(2), 675–707.CrossRefGoogle Scholar
Xu, G., & Shang, Z. (2018). Identifying latent structures in restricted latent class models. Journal of the American Statistical Association. https://doi.org/10.1080/01621459.2017.CrossRefGoogle Scholar