Hostname: page-component-745bb68f8f-v2bm5 Total loading time: 0 Render date: 2025-01-07T18:50:36.323Z Has data issue: false hasContentIssue false
  • English
  • Français

Advances in CD-CAT: The General Nonparametric Item Selection Method

Published online by Cambridge University Press:  01 January 2025

Chia-Yi Chiu  and
Yuan-Pei Chang
Chia-Yi Chiu*
Affiliation:
Rutgers, The State University of New Jersey
Yuan-Pei Chang
Affiliation:
Rutgers, The State University of New Jersey
*
Correspondence should be made to Chia-Yi Chiu, Rutgers, The State University of New Jersey, NewBrunswick, NJ, USA. Email: chia-yi.chiu@gse.rutgers.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

Computerized adaptive testing (CAT) is characterized by its high estimation efficiency and accuracy, in contrast to the traditional paper-and-pencil format. CAT specifically for cognitive diagnosis (CD-CAT) carries the same advantages and has been seen as a tool for advancing the use of cognitive diagnosis (CD) assessment for educational practice. A powerful item selection method is the key to the success of a CD-CAT program, and to date, various parametric item selection methods have been proposed and well-researched. However, these parametric methods all require large samples, to secure high-precision calibration of the items in the item bank. Thus, at present, implementation of parametric methods in small-scale educational settings, such as classroom, remains challenging. In response to this issue, Chang, Chiu, and Tsai (Appl Psychol Meas 43:543–561, 2019) proposed the nonparametric item selection (NPS) method that does not require parameter calibration and outperforms the parametric methods for settings with only small or no calibration samples. Nevertheless, the NPS method is not without limitations; extra assumptions are required to guarantee a consistent estimator of the attribute profiles when data conform to complex models. To remedy this shortcoming, the general nonparametric item selection (GNPS) method that incorporates the newly developed general NPC (GNPC) method (Chiu et al. in Psychometrika 83:355–375, 2018) as the classification vehicle is proposed in this study. The inclusion of the GNPC method in the GNPS method relaxes the assumptions imposed on the NPS method. As a result, the GNPS method can be used with any model or multiple models without abandoning the advantage of being a small-sample technique. The legitimacy of using the GNPS method in the CD-CAT system is supported by Theorem 1 proposed in the study. The efficiency and effectiveness of the GNPS method are confirmed by the simulation study that shows the outperformance of the GNPS method over the compared parametric methods when the calibration samples are small.

Keywords

Cognitive diagnosisGeneral nonparametric classification methodComputerized adaptive testingCD-CATNonparametric item selection
Type
Theory and Methods
Information
Psychometrika , Volume 86 , Issue 4: Special Section: Revisiting Cronbach’s Alpha , December 2021 , pp. 1039 - 1057
Copyright
Copyright © 2021 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Chang, Y-.P.,Chiu, C-.Y.,&Tsai, R-C..(2019).Nonparametric CAT for CD in educational settings with small samples.Applied Psychological Measurement,43,543–561CrossRefGoogle ScholarPubMed
Cheng, Y..(2009).When cognitive diagnosis meets computerized adaptive testing: CD-CAT.Psychometrika,74,619–632CrossRefGoogle Scholar
Chiu, C-.Y.,&Douglas, J..(2013).A nonparametric approach to cognitive diagnosis by proximity to ideal response patterns.Journal of Classification,30,225–250CrossRefGoogle Scholar
Chiu, C-.Y.,Douglas, J. A.,&Li, X..(2009).Cluster analysis for cognitive diagnosis: Theory and applications.Psychometrika,74,633–665CrossRefGoogle Scholar
Chiu, C-.Y.,&Köhn, H-.F..(2019).Consistency theory for the general nonparametric classification method.Psychometrika,84,830–845CrossRefGoogle ScholarPubMed
Chiu, C-.Y.,Sun, Y.,&Bian, Y..(2018).Cognitive diagnosis for small educational programs: The general nonparametric classification method.Psychometrika,83,355–375CrossRefGoogle ScholarPubMed
Cover, T.M.,&Thomas, J.A.,(1991).Elements of information theory,New York:WileyGoogle Scholar
de la Torre, J..(2011).The generalized DINA model framework.Psychometrika,76,179–199CrossRefGoogle Scholar
de la Torre, J.,&Chiu, C-Y..(2016).A general method of empirical Q-matrix validation.Psychometrika,81,253–273CrossRefGoogle ScholarPubMed
Henson, R. A.,Templin, J. L.,&Willse, J. T..(2009).Defining a family of cognitive diagnosis models using log-linear models with latent variables.Psychometrika,74,191–210CrossRefGoogle Scholar
Junker, B. W.,&Sijtsma, K..(2001).Cognitive assessment models with few assumptions, and connections with nonparametric item response theory.Applied Psychological Measurement,25,258–272CrossRefGoogle Scholar
Kaplan, M.,de la Torre, J.,&Barrada, J. R..(2015).New item selection methods for cognitive diagnosis computerized adaptive testing.Applied Psychological Measurement,39,167–188CrossRefGoogle ScholarPubMed
Liu, J.,Ying, Z.,&Zhang, S..(2015).A rate function approach to computerized adaptive testing for cognitive diagnosis.Psychometrika,80,468–490CrossRefGoogle ScholarPubMed
Ma, W . &de la Torre, J . (2017). GDINA: The generalized DINA model framework. R package version 1.4.2. Retrived from https://CRAN.R-project.org/package=GDINA.Google Scholar
Macready, G. B.,&Dayton, CM..(1977).The use of probabilistic models in the assessment of mastery.Journal of Educational Statistics,2,99–120CrossRefGoogle Scholar
Shannon, C. E..(1948).A mathematical theory of communication.Bell System Technical Journal,27,379–423CrossRefGoogle Scholar
Sun, Y., &de la Torre, J. (2020). Improving attribute classification accuracy in high dimensional data. In H. Jiao & R. W. Lissitz (Eds.), Innovative psychometric modeling and methods (pp. 17–44). Information age publishing INC.Google Scholar
Tatsuoka, C..(2002).Data analytic methods for latent partially ordered classification models.Applied Statistics, 51, 337–350.Google Scholar
Tatsuoka, K. K..(1985).A probabilistic model for diagnosing misconception by the pattern classification approach.Journal of Educational Statistics,10,55–73CrossRefGoogle Scholar
Templin, J. L.,&Henson, R. A..(2006).Measurement of psychological disorders using cognitive diagnosis models.Psychological Methods,11,287–305CrossRefGoogle ScholarPubMed
von Davier, M . (2005). A general diagnostic model applied to language testing data, Research Report RR-05-16. ETS.Google Scholar
von Davier, M..(2008).A general diagnostic model applied to language testing data.British Journal of Mathematical and Statistical Psychology,61,287–307CrossRefGoogle ScholarPubMed
Wang, S.,&Douglas, J..(2015).Consistency of nonparametric classification in cognitive diagnosis.Psychometrika,80,85–100CrossRefGoogle ScholarPubMed
Xu, G.,Wang, C.,&Shang, Z..(2016).On initial item selection in cognitive diagnostic computerized adaptive testing.British Journal of Mathematical and Statistical Psychology,69,291–315CrossRefGoogle ScholarPubMed
Xu, X ., Chang, H .-H., & Douglas, J . (2003). Computerized adaptive testing strategies for cognitive diagnosis. Paper presented at the annual meeting of National Council on Measurement in Education, Montreal, Canada .Google Scholar
Zheng, C.,&Chang, H-.H..(2016).High-efficiency response distribution-based item selection algorithms for short-length cognitive diagnostic computerized adaptive testing.Applied Psychological Measurement,40,608–624CrossRefGoogle ScholarPubMed