Hostname: page-component-745bb68f8f-kw2vx Total loading time: 0 Render date: 2025-01-07T18:33:02.816Z Has data issue: false hasContentIssue false
  • English
  • Français

A Rate Function Approach to Computerized Adaptive Testing for Cognitive Diagnosis

Published online by Cambridge University Press:  01 January 2025

Jingchen Liu ,
Zhiliang Ying  and
Stephanie Zhang
Jingchen Liu*
Affiliation:
Columbia University
Zhiliang Ying
Affiliation:
Columbia University
Stephanie Zhang
Affiliation:
Columbia University
*
Requests for reprints should be sent to Jingchen Liu, Columbia University, 1255 Amsterdam Avenue, New York, NY 10027, USA. E-mail: jcliu@stat.columbia.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

Computerized adaptive testing (CAT) is a sequential experiment design scheme that tailors the selection of experiments to each subject. Such a scheme measures subjects’ attributes (unknown parameters) more accurately than the regular prefixed design. In this paper, we consider CAT for diagnostic classification models, for which attribute estimation corresponds to a classification problem. After a review of existing methods, we propose an alternative criterion based on the asymptotic decay rate of the misclassification probabilities. The new criterion is then developed into new CAT algorithms, which are shown to achieve the asymptotically optimal misclassification rate. Simulation studies are conducted to compare the new approach with existing methods, demonstrating its effectiveness, even for moderate length tests.

Keywords

computerized adaptive testingcognitive diagnosislarge deviationclassification
Type
Original Paper
Information
Psychometrika , Volume 80 , Issue 2 , June 2015 , pp. 468 - 490
Copyright
Copyright © 2013 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Chang, H.-H., & Ying, Z. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20, 213–229.CrossRefGoogle Scholar
Cheng, Y. (2009). When cognitive diagnosis meets computerized adaptive testing CD-CAT. Psychometrika, 74, 619–632.CrossRefGoogle Scholar
Chiu, C., Douglas, J., & Li, X. (2009). Cluster analysis for cognitive diagnosis: theory and applications. Psychometrika, 74, 633–665.CrossRefGoogle Scholar
Cox, D., & Hinkley, D. (2000). Theoretical statistics. London: Chapman & Hall.Google Scholar
de la Torre, J., & Douglas, J. (2004). Higher order latent trait models for cognitive diagnosis. Psychometrika, 69, 333–353.CrossRefGoogle Scholar
DiBello, L.V., Stout, W.F., & Roussos, L.A. (1995). Unified cognitive psychometric diagnostic assessment likelihood-based classification techniques. In Nichols, P.D., Chipman, S.F., & Brennan, R.L. (Eds.), Cognitively diagnostic assessment (pp. 361–390). Hillsdale: Erlbaum Associates.Google Scholar
Edelsbrunner, H., & Grayson, D.R. (2000). Edgewise subdivision of a simplex. Discrete & Computational Geometry, 24, 707–719.CrossRefGoogle Scholar
Hartz, S.M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: blending theory with practicality. Unpublished doctoral dissertation, University of Illinois, Urbana-Champaign.Google Scholar
Junker, B. (2007). Using on-line tutoring records to predict end-of-year exam scores: experience with the ASSISTments project and MCAS 8th grade mathematics. In Lissitz, R.W. (Ed.), Assessing and modeling cognitive development in school: intellectual growth and standard settings. Maple Grove: JAM Press.Google Scholar
Junker, B., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272.CrossRefGoogle Scholar
Leighton, J.P., Gierl, M.J., & Hunka, S.M. (2004). The attribute hierarchy model for cognitive assessment: a variation on Tatsuoka’s rule-space approach. Journal of Educational Measurement, 41, 205–237.CrossRefGoogle Scholar
Lord, F.M. (1971). Robbins–Monro procedures for tailored testing. Educational and Psychological Measurement, 31, 3–31.CrossRefGoogle Scholar
Lord, F.M. (1980). Applications of item response theory to practical testing problems. Hillsdale: Erlbaum.Google Scholar
Owen, R.J. (1975). Bayesian sequential procedure for quantal response in context of adaptive mental testing. Journal of the American Statistical Association, 70, 351–356.CrossRefGoogle Scholar
Rupp, A.A., Templin, J., & Henson, R.A. (2010). Diagnostic measurement: theory, methods, and applications. New York: Guilford Press.Google Scholar
Serfling, R.J. (1980). Approximation theorems of mathematical statistics. New York: Wiley-Interscience (W. Shewhart & S. Wilks (Eds.)).CrossRefGoogle Scholar
Tatsuoka, K.K. (1983). Rule space: an approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345–354.CrossRefGoogle Scholar
Tatsuoka, K.K. (1985). A probabilistic model for diagnosing misconceptions in the pattern classification approach. Journal of Educational Statistics, 12, 55–73.CrossRefGoogle Scholar
Tatsuoka, K. (1991). Boolean algebra applied to determination of the universal set of misconception states (ONR-Technical Report No. RR-91-44). Princeton: Educational Testing Services.Google Scholar
Tatsuoka, C. (1996). Sequential classification on partially ordered sets. Doctoral dissertation, Cornell University.Google Scholar
Tatsuoka, C. (2002). Data-analytic methods for latent partially ordered classification models. Applied Statistics, 51, 337–350.Google Scholar
Tatsuoka, K.K. (2009). Cognitive assessment: an introduction to the rule space method. New York: Routledge.CrossRefGoogle Scholar
Tatsuoka, C., & Ferguson, T. (2003). Sequential classification on partially ordered sets. Journal of the Royal Statistical Society, Series B, Statistical Methodology, 65, 143–157.CrossRefGoogle Scholar
Templin, J., & Henson, R.A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287–305.CrossRefGoogle ScholarPubMed
Templin, J., He, X., Roussos, L.A., & Stout, W.F. (2003). The pseudo-item method: a simple technique for analysis of polytomous data with the fusion model (External Diagnostic Research Group Technical Report).Google Scholar
Thissen, D., & Mislevy, R.J. et al (2000). Testing algorithms. In Wainer, H. et al. (Eds.), Computerized adaptive testing: a primer (2nd ed., pp. 101–133). Mahwah: Lawrence Erlbaum Associates.Google Scholar
van der Linden, W.J. (1998). Bayesian item selection criteria for adaptive testing. Psychometrika, 63, 201–216.CrossRefGoogle Scholar
von Davier, M. (2005). A general diagnosis model applied to language testing data (Research report). Princeton: Educational Testing Service.Google Scholar
Xu, X., Chang, H.-H., & Douglas, J. (2003). A simulation study to compare CAT strategies for cognitive diagnosis. Paper presented at the annual meeting of the American Educational Research Association, Chicago, April 2003.Google Scholar