Hostname: page-component-745bb68f8f-v2bm5 Total loading time: 0 Render date: 2025-01-07T18:38:25.989Z Has data issue: false hasContentIssue false
  • English
  • Français

Cognitive Diagnosis for Small Educational Programs: The General Nonparametric Classification Method

Published online by Cambridge University Press:  01 January 2025

Chia-Yi Chiu ,
Yan Sun  and
Yanhong Bian
Chia-Yi Chiu*
Affiliation:
Rutgers, The State University of New Jersey
Yan Sun
Affiliation:
Rutgers, The State University of New Jersey
Yanhong Bian
Affiliation:
Rutgers, The State University of New Jersey
*
Correspondence should be made to Chia-Yi Chiu, Rutgers, The State University of New Jersey, New Brunswick, NJ, USA. Email: cychiu@gse.rutgers.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

The focus of cognitive diagnosis (CD) is on evaluating an examinee’s strengths and weaknesses in terms of cognitive skills learned and skills that need study. Current methods for fitting CD models (CDMs) work well for large-scale assessments, where the data of hundreds or thousands of examinees are available. However, the development of CD-based assessment tools that can be used in small-scale test settings, say, for monitoring the instruction and learning process at the classroom level has not kept up with the rapid pace at which research and development proceeded for large-scale assessments. The main reason is that the sample sizes of the small-scale test settings are simply too small to guarantee the reliable estimation of item parameters and examinees’ proficiency class membership. In this article, a general nonparametric classification (GNPC) method that allows for assigning examinees to the correct proficiency classes with a high rate when sample sizes are at the classroom level is proposed as an extension of the nonparametric classification (NPC) method (Chiu and Douglas in J Classif 30:225–250, 2013). The proposed method remedies the shortcomings of the NPC method and can accommodate any CDM. The theoretical justification and the empirical studies are presented based on the saturated general CDMs, supporting the legitimacy of using the GNPC method with any CDM. The results from the simulation studies and real data analysis show that the GNPC method outperforms the general CDMs when samples are small.

Keywords

cognitive diagnosiscognitive diagnostic modelsgeneral nonparametric classification methodLCDMG-DINA
Type
Original Paper
Information
Psychometrika , Volume 83 , Issue 2 , June 2018 , pp. 355 - 375
Copyright
Copyright © 2017 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Ayers, E., Nugent, R., & Dean, N., (2008). Skill set profile clustering based on student capability vectors computed from online tutoring data. In R. S. J. de Baker, T. Barnes, & J. E. Beck (Eds.), Educational data mining 2008: Proceedings of the 1st international conference on educational data mining, Montreal, Quebec, Canada (pp. 210–217). Retrieved from http://www.educationaldatamining.org/EDM2008/uploads/proc/full%20proceedings.pdf.Google Scholar
Chiu, C-Y, Douglas, J.A., (2013). A nonparametric approach to cognitive diagnosis by proximity to ideal response patterns, Journal of Classification, 30, 225–250.CrossRefGoogle Scholar
Chiu, C-Y, Douglas, J.A., Li, X., (2009). Cluster analysis for cognitive diagnosis: Theory and applications, Psychometrika, 74, 633–665.CrossRefGoogle Scholar
Chiu, C-Y, Köhn, H-F, Wu, H-M, (2016). Fitting the Reduced RUM with Mplus: A Tutorial, International Journal of Testing, 16, 331–351.CrossRefGoogle Scholar
de la Torre, J., (2011). The generalized DINA model framework, Psychometrika, 76, 179–199.CrossRefGoogle Scholar
DiBello, L.V., Roussos, L.A., Stout, W.F., (2007). Review of cognitively diagnostic assessment and a summary of psychometric models. In Rao, C.R., Sinharay, S. (Eds.), Handbook of statistics: Psychometrics. (pp 979–1030). Amsterdam:Elsevier.Google Scholar
Fu, J., & Li, Y., (2007). An integrative review of cognitively diagnostic psychometric models. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Chicago, IL.Google Scholar
Garrett, E., Zeger, S.L., (2000). Latent class model diagnosis, Biometrics, 56, 1055–1067.CrossRefGoogle ScholarPubMed
Hartz, S. M., (2002). A Bayesian framework for the Unified Model for assessing cognitive abilities: Blending theory with practicality (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 3044108).Google Scholar
Hartz, S.M., Roussos, L.A., (2008). The fusion model for skill diagnosis: blending theory with practicality. Princeton, NJ:Educational Testing Service(Research report No. RR-08-71).Google Scholar
Henson, R.A., Templin, J.L., Willse, J.T., (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables, Psychometrika, 74, 191–210.CrossRefGoogle Scholar
Junker, B.W., Sijtsma, K., (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory, Applied Psychological Measurement, 25, 258–272.CrossRefGoogle Scholar
Köhn, H-F, Chiu, C-Y, (2017). A procedure for assessing the completeness of the Q-matrices of cognitively diagnostic tests, Psychometrika, 82, 112–132.CrossRefGoogle ScholarPubMed
Macready, G.B., Dayton, C.M., (1977). The use of probabilistic models in the assessment of mastery, Journal of Educational Statistics, 2, 99–120.CrossRefGoogle Scholar
Maris, E., (1999). Estimating multiple classification latent class models, Psychometrika, 64, 187–212.CrossRefGoogle Scholar
Richardson, S., Green, P.J., (1997). On Bayesian analysis of mixtures with an unknown number of components, Journal of Royal Statistical Society Series B, 59, 731–792.CrossRefGoogle Scholar
Rupp, A.A., Templin, J.L., Henson, R.A., (2010). Diagnostic measurement. Theory, methods, and applications. New York:Guilford.Google Scholar
Stephens, M., (2000). Dealing with label switching in mixture models, Journal of Royal Statistical Society Series B, 62, 795–809.CrossRefGoogle Scholar
Stout, W., (2002). Psychometrics: From practice to theory and back, Psychometrika, 67, 485–518.CrossRefGoogle Scholar
Tatsuoka, K.K., (1985). A probabilistic model for diagnosing misconception by the pattern classification approach, Journal of Educational Statistics, 10, 55–73.CrossRefGoogle Scholar
Templin, J.L., Henson, R.A., (2006). Measurement of psychological disorders using cognitive diagnosis models, Psychological Methods, 11, 287–305.CrossRefGoogle ScholarPubMed
Tueller, S.J., Drotar, S., Lubke, G.H., (2011). Addressing the problem of switched class labels in latent variable mixture model simulation studies, Structural Equation Modeling A Multidisciplinary Journal, 18, 110–131.CrossRefGoogle Scholar
von Davier, M., (2005). A general diagnostic model applied to language testing data. Princeton, NJ:Educational Testing Service(Research report No. RR-05-16).CrossRefGoogle Scholar
von Davier, M., (2008). A general diagnostic model applied to language testing data, British Journal of Mathematical and Statistical Psychology, 61, 287–301.CrossRefGoogle ScholarPubMed
von Davier, M., (2014). The DINA model as a constrained general diagnostic model: Two variants of a model equivalency, British Journal of Mathematical and Statistical Psychology, 67, 49–71.CrossRefGoogle Scholar
Wang, S., Douglas, J., (2015). Consistency of nonparametric classification in cognitive diagnosis, Psychometrika, 80, 85–100.CrossRefGoogle ScholarPubMed
Willse, J., Henson, R., & Templin, J., (2007). Using sum scores or IRT in place of cognitive diagnosis models: Can existing or more familiar models do the job? Paper presented at the Annual Meeting of the National Council on Measurement in Education, Chicago, IL..Google Scholar
Zhang, S., & Culpepper, S. A., (2017). Bayesian estimation of restricted latent class models. Paper presented at the annual meeting of the National Council on Measurement in Education, San Antonio, Texas..Google Scholar