Hostname: page-component-745bb68f8f-b95js Total loading time: 0 Render date: 2025-01-07T11:30:26.519Z Has data issue: false hasContentIssue false
  • English
  • Français

Noncompensatory MIRT For Passage-Based Tests

Published online by Cambridge University Press:  01 January 2025

Nana Kim Open the ORCID record for Nana Kim [Opens in a new window] ,
Daniel M. Bolt  and
James Wollack
Nana Kim*
Affiliation:
University of Wisconsin-Madison
Daniel M. Bolt
Affiliation:
University of Wisconsin-Madison
James Wollack
Affiliation:
University of Wisconsin-Madison
*
Correspondence should be made to Nana Kim, University of Wisconsin-Madison, 1025 W. Johnson St, Madison, WI53706, USA. Email: nkim84@wisc.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

We consider a multidimensional noncompensatory approach for binary items in passage-based tests. The passage-based noncompensatory model (PB-NM) emphasizes two underlying components in solving passage-based test items: a passage-related component and a passage-independent component. An advantage of the PB-NM model over commonly applied compensatory models (e.g., bifactor model) is that the two components are parameterized in relation to difficulty as opposed to discrimination parameters. As a result, while simultaneously accounting for passage-related local item dependence, the model permits the assessment of how items based on the same passage may require varying levels of passage comprehension (as well as varying levels of passage-independent proficiency) to obtain a correct response. Through a simulation study, we evaluate the comparative fit of the PB-NM against the bifactor model and also illustrate the relationship between the difficulty parameters of the PB-NM and the discrimination parameters of the bifactor model. We further apply the PB-NM to an actual reading comprehension test to demonstrate the relevance of the model in understanding variation in the relative difficulty of the two components across different item types.

Keywords

multidimensional item response theory (MIRT)noncompensatory MIRTconjunctive Rasch modelbifactor modelpassage-based tests
Type
Application Reviews and Case Studies
Information
Psychometrika , Volume 87 , Issue 3 , September 2022 , pp. 992 - 1009
Copyright
Copyright © 2021 The Author(s) under exclusive licence to The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Supplementary Information The online version contains supplementary material available at https://doi.org/10.1007/s11336-021-09826-6.

References

Ackerman, T., (1996). Graphical representation of multidimensional item response theory analyses Applied Psychological Measurement 20(4) 311–329CrossRefGoogle Scholar
Bolt, D. M., & Lall, V. F., (2003). Estimation of compensatory and noncompensatory multidimensional IRT models using Markov chain Monte Carlo Applied Psychological Measurement 27(6) 395–514CrossRefGoogle Scholar
Bolt, D. M. (2019). Bifactor MIRT as an appealing and related alternative to CDMs in the presence of skill attribute continuity. In M. von Davier & Y.-S. Lee (Eds.), Handbook of Diagnostic Classification Models (pp. 395–417). Springer. https://doi.org/10.1007/978-3-030-05584-4.CrossRefGoogle Scholar
Bonifay, W., & Cai, L., (2017). On the complexity of item response theory models Multivariate Behavioral Research 52(4) 465–484CrossRefGoogle ScholarPubMed
Bonifay, W.,Lane, S. P., Reise, S. P., (2017). Three concerns with applying a bifactor model as a structure of psychopathology Clinical Psychological Science 5(1) 184–186CrossRefGoogle Scholar
Bradlow, E. T.,Wainer, H., Wang, X., (1999). A Bayesian random effects model for testlets Psychometrika 64(2) 153–168CrossRefGoogle Scholar
Cai, L.,Yang, J. S., Hansen, M., (2011). Generalized full-information item bifactor analysis Psychological Methods 16(3) 221–248CrossRefGoogle ScholarPubMed
Chalmers, R. P., (2012). mirt: A multidimensional item response theory package for the R environment Journal of Statistical Software 48(6) 1–29CrossRefGoogle Scholar
Chen, W-H Thissen, D., (1997). Local dependence indexes for item pairs using item response theory Journal of Educational and Behavioral Statistics 22(3) 265–289CrossRefGoogle Scholar
De Boeck, P., & Partchev, I., (2012). IRTrees: Tree-based item response models of the GLMM family Journal of Statistical Software 48(1) 1–28CrossRefGoogle Scholar
DeMars, C. E., (2006). Application of the bi-factor multidimensional item response theory model to testlet-based tests Journal of Educational Measurement 43(2) 145–168CrossRefGoogle Scholar
DeMars, C. E., (2016). Partially compensatory multidimensional item response theory models: Two alternate model forms Educational and Psychological Measurement 76(2) 231–257CrossRefGoogle ScholarPubMed
Edwards, M. C.,Houts, C. R., Cai, L., (2018). A diagnostic procedure to detect departures from local independence in item response theory models Psychological Methods 23(1) 138–149CrossRefGoogle ScholarPubMed
Embretson, S. E., (1984). A general multicomponent latent trait model for response processes Psychometrika 49 175–186CrossRefGoogle Scholar
Gelman, A., & Rubin, D., (1992). Inference from iterative simulation using multiple sequences Statistical Science 7(4) 457–472CrossRefGoogle Scholar
Gibbons, R. D., & Hedeker, D. R., (1992). Full-information item bifactor analysis Psychometrika 57 423–436CrossRefGoogle Scholar
Holzinger, K. J., & Swineford, F., (1937). The bi-factor method Psychometrika 2 41–54CrossRefGoogle Scholar
Hoskens, M., & De Boeck, P., (1997). A parametric model for local dependence among test items Psychological Methods 2(3) 261–277CrossRefGoogle Scholar
Ip, E. H., (2000). Adjusting for information inflation due to local dependency in moderately large item clusters Psychometrika 65(1) 73–91CrossRefGoogle Scholar
Ip, E. H., (2002). Locally dependent latent trait model and the Dutch identity revisited Psychometrika 67(3) 367–386CrossRefGoogle Scholar
Ip, E. H., (2010). Empirically indistinguishable multidimensional IRT and locally dependent unidimensional item response models British Journal of Mathematical and Statistical Psychology 63 395–415CrossRefGoogle ScholarPubMed
Ip, E. H.,Molenberghs, G., Chen, S.,Goegebeur, Y., De Boeck, P., (2013). Functional unidimensional item response models for multivariate binary data Multivariate Behavioral Research 48 534–562CrossRefGoogle ScholarPubMed
Jeon, M., & De Boeck, P., (2016). A generalized item response tree model for psychological assessments Behavior Research Methods 48 1070–1085CrossRefGoogle ScholarPubMed
Jiao, H., Lissitz, R. W., & Zhan, P. (2017). A noncompensatory testlet model for calibrating innovative items embedded in multiple contexts. In Jiao, H. & Lissitz, R. W. (Eds.), Technology enhanced innovative assessment: Development, modeling, and scoring from an interdisciplinary perspective (pp. 117–138). Information Age Publishing.Google Scholar
Kellner, K. (2019). jagsUI: A wrapper arouns rjags to streamline JAGS analyses (R package Version 1.5.1.). https://github.com/kenkellner/jagsUI.Google Scholar
Li, Y.,Bolt, D. M., Fu, J., (2006). A comparison of alternative models for testlets Applied Psychological Measurement 30(1) 3–21CrossRefGoogle Scholar
Luecht, R., & Ackerman, T. A., (2018). A technical note on IRT simulation studies: Dealing with truth, estimates, observed data, and residuals Educational Measurement: Issues and Practice 37(3) 65–76CrossRefGoogle Scholar
Maris, E., (1995). Psychometric latent response models Psychometrika 60(4) 523–547CrossRefGoogle Scholar
Plummer, M. (2017). JAGS version 4.3.0 user manual [Computer software manual]. https://sourceforge.net/projects/mcmc-jags.Google Scholar
R core team. (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. http://www.R-project.org.Google Scholar
Reckase, M. (2009). Multidimensional item response theory. Springer. https://doi.org/10.1007/978-0-387-89976-3.CrossRefGoogle Scholar
Rijmen, F. (2009). Efficient full information maximum likelihood estimation for multidimensional IRT models. ETS Research Report Series, 2009(1), i-31. https://doi.org/10.1002/j.2333-8504.2009.tb02160.xCrossRefGoogle Scholar
Sireci, S. G.,Thissen, D., Wainer, H., (1991). On the reliability of testlet-based tests Journal of Educational Measurement 28(3) 237–247CrossRefGoogle Scholar
Thissen, D.,Steinberg, L., Mooney, J. A., (1989). Trace lines for testlets: A use of multiple-categorical-response models Journal of Educational Measurement 26(3) 247–260CrossRefGoogle Scholar
Wainer, H., Bradlow, E. T., & Wang, X. (2007). Testlet response theory and its applications. Cambridge University Press. https://doi.org/10.1017/CBO9780511618765.CrossRefGoogle Scholar
Wainer, H., & Kiely, G., (1987). Item clusters and computerized adaptive testing: A case for testlets Journal of Educational Measurement 24(3) 185–201CrossRefGoogle Scholar
Wainer, H., & Thissen, D., (1996). How is reliability related to the quality of test scores? What is the effect of local dependence on reliability? Educational Measurement: Issues and Practice 15(1) 22–29CrossRefGoogle Scholar
Whitely, S. E., (1980). Multicomponent latent trait models for ability tests Psychometrika 45 479–494CrossRefGoogle Scholar
Yen, W. M., (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model Applied Psychological Measurement 8(2) 125–145CrossRefGoogle Scholar
Yen, W. M., (1993). Scaling performance assessments: Strategies for managing local item dependence Journal of Educational Measurement 30(3) 187–213CrossRefGoogle Scholar
Supplementary material: File

Kim et al. supplementary material

Supplementary Information The online version contains supplementary material available at https://doi.org/10.1007/s11336-021-09826-6.
Download Kim et al. supplementary material(File)
File 643.9 KB