Hostname: page-component-745bb68f8f-v2bm5 Total loading time: 0 Render date: 2025-01-07T19:02:48.888Z Has data issue: false hasContentIssue false
  • English
  • Français

Model Evaluation and Multiple Strategies in Cognitive Diagnosis: An Analysis of Fraction Subtraction Data

Published online by Cambridge University Press:  01 January 2025

Jimmy de la Torre  and
Jeffrey A. Douglas
Jimmy de la Torre*
Affiliation:
Department of Educational Psychology, Rutgers, The State University of New Jersey
Jeffrey A. Douglas
Affiliation:
Department of Statistics, University of Illinois
*
Requests for reprints should be sent to Jimmy de la Torre, Graduate School of Education, Rutgers, The State University of New Jersey, 10 Seminary Place, New Brunswick, NJ 08901, USA. E-mail: j.delatorre@rutgers.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

This paper studies three models for cognitive diagnosis, each illustrated with an application to fraction subtraction data. The objective of each of these models is to classify examinees according to their mastery of skills assumed to be required for fraction subtraction. We consider the DINA model, the NIDA model, and a new model that extends the DINA model to allow for multiple strategies of problem solving. For each of these models the joint distribution of the indicators of skill mastery is modeled using a single continuous higher-order latent trait, to explain the dependence in the mastery of distinct skills. This approach stems from viewing the skills as the specific states of knowledge required for exam performance, and viewing these skills as arising from a broadly defined latent trait resembling the θ of item response models. We discuss several techniques for comparing models and assessing goodness of fit. We then implement these methods using the fraction subtraction data with the aim of selecting the best of the three models for this application. We employ Markov chain Monte Carlo algorithms to fit the models, and we present simulation results to examine the performance of these algorithms.

Keywords

cognitive diagnosisitem response theorylatent class modelMarkov chain Monte Carlogoodness-of-fit
Type
Theory and Methods
Information
Psychometrika , Volume 73 , Issue 4 , December 2008 , pp. 595 - 624
Copyright
Copyright © 2008 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

The work reported here was performed under the auspices of the External Diagnostic Research Team funded by Educational Testing Service. Views expressed in this paper does not necessarily represent the views of Educational Testing Service.

References

Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In Petrov, B.N., & Csaki, F. (Eds.), Proceedings of the Second International Symposium on Information Theory (pp. 267–281). Budapest: Akad. Kiado.Google Scholar
Brooks, S.P., & Gelman, A. (1998). General methods of monitoring convergence of iterative simulations. Journal of Computational and Graphical Statistics, 7, 434–455.CrossRefGoogle Scholar
Casella, G., & George, E.I. (1992). Explaining the Gibbs sampler. The American Statistician, 46, 167–174.CrossRefGoogle Scholar
Celeux, G., Forbers, F., Robert, C.P., & Titterington, D.M. (2006). Deviance information criteria for missing data models. Bayesian Analysis, 1, 651–674.CrossRefGoogle Scholar
Chib, S., & Greenberg, E. (1995). Understanding the Metropolis-Hastings algorithm. The American Statistician, 49, 327–335.CrossRefGoogle Scholar
de la Torre, J. (2008, in press). An empirically-based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement.CrossRefGoogle Scholar
de la Torre, J., & Douglas, J. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333–353.CrossRefGoogle Scholar
DiBello, L.V., Stout, W.F., & Roussos, L.A. (1995). Unified cognitive/psychometric diagnostic assessment likelihood-based classification techniques. In Nichols, P.D., Chipman, S.F., & Brennan, R.L. (Eds.), Cognitively diagnostic assessment (pp. 361–389). Hillsdale, NJ: Erlbaum.Google Scholar
Doignon, J.P., & Falmagne, J.C. (1999). Knowledge spaces, New York: Springer.CrossRefGoogle Scholar
Doornik, J.A. (2003). Object-Oriented Matrix Programming using Ox (Version 3.1) (Computer software), London: Timberlake Consultants Press.Google Scholar
Embretson, S. (1984). A general latent trait model for response processes. Psychometrika, 49, 175–186.CrossRefGoogle Scholar
Embretson, S. (1997). Multicomponent response models. In van der Linden, W.J., & Hambleton, R.K. (Eds.), Handbook of modern item response theory (pp. 305–321). New York: Springer.CrossRefGoogle Scholar
Geman, S., & Geman, D. (1984). Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6, 721–741.CrossRefGoogle ScholarPubMed
Haertel, E.H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 333–352.CrossRefGoogle Scholar
Junker, B.W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272.CrossRefGoogle Scholar
Kass, R.E. (1993). Bayes factor in practice. The Statistician, 42, 551–560.CrossRefGoogle Scholar
Kass, R.E., & Raftery, A.E. (1995). Bayes factor. Journal of the American Statistical Association, 430, 773–795.CrossRefGoogle Scholar
LeFevre, J., Bisanz, J., Daley, K., Buffone, L., Greenham, S.L., & Sadesky, G.S. (1996). Multiple routes to solution of single-digit multiplication problems. Journal of Experimental Psychology: General, 125, 284–306.CrossRefGoogle Scholar
Macready, G.B., & Dayton, C.M. (1977). The use of probabilistic models in the assessment of mastery. Journal of Educational Statistics, 33, 379–416.Google Scholar
Maris, E. (1999). Estimating multiple classification latent class models. Psychometrika, 64, 187–212.CrossRefGoogle Scholar
Mislevy, R.J. (1996). Test theory reconceived. Journal of Education al Measurement, 33, 379–416.CrossRefGoogle Scholar
Opfer, J.E., & Siegler, R.S. (2007). Representational change and children’s numerical estimation. Cognitive Psychology, 55, 169–195.CrossRefGoogle ScholarPubMed
Patz, R.J., & Junker, B.W. (1999). A straightforward approach to Markov chain Monte Carlo methods for item response theory. Journal of Educational and Behavioral Statistics, 24, 146–178.CrossRefGoogle Scholar
Patz, R.J., & Junker, B.W. (1999). Applications and extensions of MCMC in IRT: Multiple item types, missing data, and rated responses. Journal of Educational and Behavioral Statistics, 24, 342–366.CrossRefGoogle Scholar
Raftery, A.E. (1996). Hypothesis testing and model selection. In Gilks, R.W., Richardson, S., & Spiegelhalter, D.J. (Eds.), Markov chain Monte Carlo in practice (pp. 163–187). London: Chapman & Hall.Google Scholar
Reder, L.M. (1987). Strategy selection in question answering. Cognitive Psychology, 19, 90–138.CrossRefGoogle Scholar
Shultz, T., Fisher, G., Pratt, C., & Rulf, S. (1986). Selection of causal rules. Child Development, 57, 143–152.CrossRefGoogle Scholar
Siegler, R.S. (1988). Individual differences in strategy choices: Good students, not-so-good students, and perfectionists. Child Development, 59, 833–851.CrossRefGoogle ScholarPubMed
Siegler, R.S., Adolph, K.E., & Lemaire, P. (1996). Strategy choices across the lifespan. In Reder, L. (Eds.), Implicit memory and metacognition (pp. 79–121). Hillsdale, NJ: Erlbaum.Google Scholar
Siegler, R.S., & Shrager, J. (1984). Strategy choices in addition and subtraction: How do children know what to do?. In Sophian, C. (Eds.), The origins of cognitive skills (pp. 229–293). Hillsdale, NJ: Erlbaum.Google Scholar
Spiegelhalter, D.J., Best, N.G., Carlin, B.P., & van der Linde, A. (2002). Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society Series B, 64, 583–639.CrossRefGoogle Scholar
Stout, W., Roussos, L., & Hartz, S. (2003). A demonstration of the Fusion Model skills diagnostic system: An analysis of mixed-number subtraction data. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Chicago, IL.Google Scholar
Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461–464.CrossRefGoogle Scholar
Tatsuoka, K. (1985). A probabilistic model for diagnosing misconceptions in the pattern classification approach. Journal of Educational Statistics, 12, 55–73.CrossRefGoogle Scholar
Tatsuoka, C. (2002). Data-analytic methods for latent partially ordered classification models. Journal of the Royal Statistical Society Series C (Applied Statistics), 51, 337–350.CrossRefGoogle Scholar
Tatsuoka, K. (1990). Toward an integration of item-response theory and cognitive error diagnosis. In Frederiksen, N., Glaser, R., Lesgold, A., & Safto, M. (Eds.), Monitoring skills and knowledge acquisition (pp. 453–488). Hillsdale, NJ: Erlbaum.Google Scholar
Yan, D., Almond, R., Mislevy, R., & Simpson, M.A. (2003). A rule space approach to modeling skills-pattern of two different pedagogical groups in mathematics assessment. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Chicago, IL.Google Scholar