Hostname: page-component-745bb68f8f-g4j75 Total loading time: 0 Render date: 2025-02-11T20:30:11.608Z Has data issue: false hasContentIssue false
  • English
  • Français

Multidimensional Adaptive Testing with Optimal Design Criteria for Item Selection

Published online by Cambridge University Press:  01 January 2025

Joris Mulder  and
Wim J. van der Linden
Joris Mulder*
Affiliation:
University of Twente
Wim J. van der Linden
Affiliation:
University of Twente
*
Requests for reprints should be sent to Joris Mulder, Department of Research Methodology, Measurement, and Data Analysis, Twente University, P.O. Box 217, 7500 AE Enschede, The Netherlands. E-mail: j.mulder3@uu.nl
Rights & Permissions [Opens in a new window]

Abstract

Core share and HTML view are not available for this content. However, as you have access to this content, a full PDF is available via the ‘Save PDF’ action button.

Several criteria from the optimal design literature are examined for use with item selection in multidimensional adaptive testing. In particular, it is examined what criteria are appropriate for adaptive testing in which all abilities are intentional, some should be considered as a nuisance, or the interest is in the testing of a composite of the abilities. Both the theoretical analyses and the studies of simulated data in this paper suggest that the criteria of A-optimality and D-optimality lead to the most accurate estimates when all abilities are intentional, with the former slightly outperforming the latter. The criterion of E-optimality showed occasional erratic behavior for this case of adaptive testing, and its use is not recommended. If some of the abilities are nuisances, application of the criterion of As-optimality (or Ds-optimality), which focuses on the subset of intentional abilities is recommended. For the measurement of a linear combination of abilities, the criterion of c-optimality yielded the best results. The preferences of each of these criteria for items with specific patterns of parameter values was also assessed. It was found that the criteria differed mainly in their preferences of items with different patterns of values for their discrimination parameters.

Keywords

adaptive testingFisher information matrixmultidimensional IRToptimal design
Type
Original Paper
Information
Psychometrika , Volume 74 , Issue 2 , June 2009 , pp. 273 - 296
Creative Commons
Creative Common License - CCCreative Common License - BYCreative Common License - NC
This article distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
Copyright
Copyright © 2008 The Author(s)

Footnotes

The first author is now at the Department of Methodology and Statistics, Statistics, Faculty of Social Sciences, Utrecht University, Heidelberglaan 1, 3854 Utrecht, The Netherlands. The second author is now at Research Department, CTB/McGraw-Hill, Monterey, CA, USA.

References

Berger, M.P.F., Wong, W.K. (2005). Applied optimal design, London: Wiley.CrossRefGoogle Scholar
Bernaerts, K., Servaes, R.D., Kooyman, S., Versyck, K.J., Van Impe, J.F. (2002). Optimal temperature input designs for estimation of the square root model parameters: parameter accuracy and model validity restrictions. International Journal of Food Microbiology, 73, 145–157.CrossRefGoogle ScholarPubMed
Bloxom, B., & Vale, C.D. (1987). Multidimensional adaptive testing: An approximate procedure for updating. In Meeting of the psychometric society. Montreal, Canada, June.Google Scholar
Boughton, K.A., Yao, L., & Lewis, D.M. (2006). Reporting diagnostic subscale scores for tests composed of complex structure. In Meeting of the national council on measurement in education. San Francisco, CA, April.Google Scholar
Chang, H.-H. (2004). Understanding computerized adaptive testing: from Robbins-Monro to Lord and beyond. In Kaplan, D. (Eds.), Handbook of quantitative methods for the social sciences (pp. 117–133). Thousand Oaks: Sage.Google Scholar
Chang, H.-H., Ying, Z. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20, 213–229.CrossRefGoogle Scholar
Fan, M., & Hsu, Y. (1996). Multidimensional computer adaptive testing. In Annual meeting of the American educational research association. New York City, NY, April.Google Scholar
Lehmann, E.L. (1999). Elements of large-sample theory, New York: Springer.CrossRefGoogle Scholar
Luecht, R.M. (1996). Multidimensional computer adaptive testing. Applied Psychological Measurement, 20, 389–404.CrossRefGoogle Scholar
McDonald, R.P. (1967). Nonlinear factor analysis. Psychometric Monographs No. 15.CrossRefGoogle Scholar
McDonald, R.P. (1997). Normal-ogive multidimensional model. In van der Linden, W.J., Hambleton, R.K. (Eds.), Handbook of modern item response theory (pp. 258–270). New York: Springer.Google Scholar
Owen, R.J. (1969). A Bayesian approach to tailored testing (Research Report 69-92). Princeton, NJ: Educational Testing Service.Google Scholar
Owen, R.J. (1975). A Bayesian sequential procedure for quantal response in the context of adaptive mental testing. Journal of the American Statistical Association, 70, 351–356.CrossRefGoogle Scholar
Reckase, M.D. (1985). The difficulty of test items that measure more than one ability. Applied Psychological Measurement, 9, 401–412.CrossRefGoogle Scholar
Reckase, M.D. (1997). A linear logistic multidimensional model for dichotomous items response data. In van der Linden, W.J., Hambleton, R.K. (Eds.), Handbook of modern item response theory (pp. 271–286). New York: Springer.CrossRefGoogle Scholar
Samejima, F. (1974). Normal ogive model for the continuous response level in the multidimensional latent space. Psychometrika, 39, 111–121.CrossRefGoogle Scholar
Segall, D.O. (1996). Multidimensional adaptive testing. Psychometrika, 61, 331–354.CrossRefGoogle Scholar
Segall, D.O. (2000). Principles of multidimensional adaptive testing. In van der Linden, W.J., Glas, C.A.W. (Eds.), Computerized adaptive testing: Theory and practice (pp. 53–73). Boston: Kluwer Academic.CrossRefGoogle Scholar
Silvey, S.D. (1980). Optimal design, London: Chapman & Hall.CrossRefGoogle Scholar
Sympson, J.B., & Hetter, R.D. (1985). Controlling item-exposure rates in computerized adaptive testing. In Proceedings of the 27th annual meeting of the military testing association (pp. 973–977). San Diego, CA: Navy Personnel Research and Development Center.Google Scholar
Tanner, M.A. (1993). Tools for statistical inference, New York: Springer.CrossRefGoogle Scholar
van der Linden, W.J. (1996). Assembling tests for the measurement of multiple traits. Applied Psychological Measurement, 20, 373–388.CrossRefGoogle Scholar
van der Linden, W.J. (1999). Multidimensional adaptive testing with a minimum error-variance criterion. Journal of Educational and Behavioral Statistics, 24, 398–412.CrossRefGoogle Scholar
van der Linden, W.J. (2005). Linear models for optimal test design, New York: Springer.CrossRefGoogle Scholar
van der Linden, W.J., Glas, C.A.W. (2000). Computerized adaptive testing: Theory and practice, Boston: Kluwer Academic.CrossRefGoogle Scholar
van der Linden, W.J., Glas, C.A.W. (2007). Statistical aspects of adaptive testing. In Rao, C.R., Sinharay, S. (Eds.), Psychometrics (pp. 801–838). Amsterdam: North-Holland.Google Scholar
van der Linden, W.J., Veldkamp, B.P. (2007). Conditional item-exposure control in adaptive testing using item-ineligibility probabilities. Journal of Educational and Behavioral Statistics, 32, 398–418.CrossRefGoogle Scholar
Veldkamp, B.P., van der Linden, W.J. (2002). Multidimensional adaptive testing with constraints on test content. Psychometrika, 67, 575–588.CrossRefGoogle Scholar
Wainer, H. (2000). Computerized adaptive testing: A primer, Hillsdale: Lawrence Erlbaum Associates.CrossRefGoogle Scholar
Yao, L., Boughton, K.A. (2007). A multidimensional item response modeling approach for improving subscale proficiency estimation and classification. Applied Psychological Measurement, 31, 83–105.CrossRefGoogle Scholar