Hostname: page-component-745bb68f8f-lrblm Total loading time: 0 Render date: 2025-02-07T00:58:18.480Z Has data issue: false hasContentIssue false
  • English
  • Français

When Cognitive Diagnosis Meets Computerized Adaptive Testing: CD-CAT

Published online by Cambridge University Press:  01 January 2025

Ying Cheng
Ying Cheng*
Affiliation:
University of Notre Dame
*
Requests for reprints should be sent to Ying Cheng, 118 Haggar Hall, Notre Dame, IN, 46556, USA. E-mail: ycheng4@nd.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

Computerized adaptive testing (CAT) is a mode of testing which enables more efficient and accurate recovery of one or more latent traits. Traditionally, CAT is built upon Item Response Theory (IRT) models that assume unidimensionality. However, the problem of how to build CAT upon latent class models (LCM) has not been investigated until recently, when Tatsuoka (J. R. Stat. Soc., Ser. C, Appl. Stat. 51:337–350, 2002) and Tatsuoka and Ferguson (J. R. Stat., Ser. B 65:143–157, 2003) established a general theorem on the asymptotically optimal sequential selection of experiments to classify finite, partially ordered sets. Xu, Chang, and Douglas (Paper presented at the annual meeting of National Council on Measurement in Education, Montreal, Canada, 2003) then tested two heuristics in a simulation study based on Tatsuoka’s theoretical work in the context of computerized adaptive testing. One of the heuristics was developed based on Kullback–Leibler information, and the other based on Shannon entropy. In this paper, we showcase the application of the optimal sequential selection methodology in item selection of CAT that is built upon cognitive diagnostic models. Two new heuristics are proposed, and are compared against the randomized item selection method and the two heuristics investigated in Xu et al. (Paper presented at the annual meeting of National Council on Measurement in Education, Montreal, Canada, 2003). Finally, we show the connection between the Kullback–Leibler-information-based approaches and the Shannon-entropy-based approach, as well as the connection between algorithms built upon LCM and those built upon IRT models.

Keywords

optimal sequential selectionlatent class modelcomputerized adaptive testingcognitive diagnosisitem response theory
Type
Theory and Methods
Information
Psychometrika , Volume 74 , Issue 4 , December 2009 , pp. 619 - 632
Copyright
Copyright © 2009 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

The author would like to thank the editors, anonymous reviewers, and Drs. Hua-Hua Chang and Jeff Douglas for their constructive suggestions.

References

Chang, H., Ying, Z. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20, 213–229.CrossRefGoogle Scholar
Chang, H., Zhang, J. (2002). Hypergeometric family and item overlap rates in computerized adaptive testing. Psychometrika, 67, 387–398.CrossRefGoogle Scholar
Cover, T.M., Thomas, J.A. (1991). Elements of information theory, New York: Wiley.Google Scholar
Embretson, S.E. (2001). The second century of ability testing: Some predictions and speculations. Retrievable at http://www.ets.org/Media/Research/pdf/PICANG7.pdf.Google Scholar
Haertel, E.H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 301–321.CrossRefGoogle Scholar
Haertel, E.H. (1984). An application of latent class models to assessment data. Applied Psychological Measurement, 8, 333–346.CrossRefGoogle Scholar
Haertel, E.H., Wiley, D.E. (1993). Presentations of ability structures: Implications for testing. In Frederiksen, N., Mislevey, R.J., Bejar, I.I. (Eds.), Test theory for a new generation of tests (pp. 359–384). Hillsdale: Erlbaum.Google Scholar
Hambleton, R., Swaminathan, H. (1985). Item response theory: Principles and applications, Boston: Kluwer-Nijhoff.CrossRefGoogle Scholar
Hartz, S. (2002). A Bayesian framework for the Unified Model for assessing cognitive abilities: blending theory with practice. Unpublished doctoral thesis, University of Illinois at Urbana-Champaign.Google Scholar
Hartz, S., Roussos, L., Stout, W. (2002). Skill diagnosis: Theory and practice [Computer software user manual for Arpeggio software], Princeton: ETS.Google Scholar
Henson, R., Douglas, J. (2005). Test construction for cognitive diagnosis. Applied Psychological Measurement, 29, 262–277.CrossRefGoogle Scholar
Junker, B.W., Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272.CrossRefGoogle Scholar
Karelitz, T.M., & de la Torre, J. (2008). When do measurement models produce diagnostic information? An investigation of the assumptions of cognitive diagnosis modeling. In National Council on Measurement in Education Annual Meeting in New York, NY.Google Scholar
Lord, F.M. (1980). Applications of item response theory to practical testing problems, Hillsdale: Erlbaum.Google Scholar
McGlohen, M.K. (2004). The application of cognitive diagnosis and computerized adaptive testing to a large-scale assessment. Unpublished doctoral thesis, University of Texas at Austin.Google Scholar
McGlohen, M.K., Chang, H. (2008). Combining computer adaptive testing technology with cognitively diagnostic assessment. Behavioral Research Methods, 40, 808–821.CrossRefGoogle ScholarPubMed
Mislevy, R.J., Almond, R.G., Yan, D., Steinberg, L.S. (1999). Bayes nets in educational assessment: Where do the numbers come from?. In Laskey, K.B., Prade, H. (Eds.), Proceedings of the fifteenth conference on uncertainty in artificial intelligence (pp. 437–446). San Mateo: Morgan Kaufmann.Google Scholar
Rupp, A., Templin, J. (2008). The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68, 78–96.CrossRefGoogle Scholar
Shannon, C.E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27, 379–423.CrossRefGoogle Scholar
Stocking, M.L. (1994). Three practical issues for modern adaptive testing item pools (ETS Research Rep. No. 94-5). Princeton: Educational Testing Service.Google Scholar
Tatsuoka, C. (2002). Data analytic methods for latent partially ordered classification models. Journal of the Royal Statistical Society, Series C (Applied Statistics), 51, 337–350.CrossRefGoogle Scholar
Tatsuoka, C., Ferguson, T. (2003). Sequential classification on partially ordered sets. Journal of Royal Statistics, Series B, 65, 143–157.CrossRefGoogle Scholar
Tatsuoka, K.K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345–354.CrossRefGoogle Scholar
Tatsuoka, K.K. (1995). Architecture of knowledge structures and cognitive diagnosis: a statistical pattern classification approach. In Nichols, P., Chipman, S., Brennan, R. (Eds.), Cognitively diagnostic assessments (pp. 327–359). Hillsdale: Erlbaum.Google Scholar
Templin, J. (2006). CDM: cognitive diagnosis modeling using Mplus, user guide. Retrievable at: http://www.iqgrads.net/jtemplin/downloads/CDM_user_guide.pdf.Google Scholar
Templin, J., Henson, R. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287–305.CrossRefGoogle ScholarPubMed
Thissen, D., Mislevy, R.J.et al. (2000). Testing algorithms. In Wainer, H.et al. (Eds.), Computerized adaptive testing: A primer (pp. 101–133). Hillsdale: Erlbaum.Google Scholar
van der Linden, W.J. (1998). Bayesian item-selection criteria for adaptive testing. Psychometrika, 63, 201–216.CrossRefGoogle Scholar
von Davier, M. (2005). A general diagnostic model applied to language testing data (ETS Research Rep. No. RR-05-16). Princeton: ETS.Google Scholar
Xu, X., Chang, H., & Douglas, J. (2003). Computerized adaptive testing strategies for cognitive diagnosis. Paper presented at the annual meeting of National Council on Measurement in Education, Montreal, Canada.Google Scholar
Xu, X., Douglas, J. (2006). Computerized adaptive testing under nonparametric IRT models. Psychometrika, 71, 121–137.CrossRefGoogle Scholar