Hostname: page-component-5f745c7db-96s6r Total loading time: 0 Render date: 2025-01-06T21:45:27.013Z Has data issue: true hasContentIssue false
  • English
  • Français

Modeling Omitted and Not-Reached Items in IRT Models

Published online by Cambridge University Press:  01 January 2025

Norman Rose Open the ORCID record for Norman Rose [Opens in a new window] ,
Matthias von Davier  and
Benjamin Nagengast
Norman Rose*
Affiliation:
University of Tübingen
Matthias von Davier
Affiliation:
Educational Testing Service
Benjamin Nagengast
Affiliation:
University of Tübingen
*
Correspondence should be made to Norman Rose, Hector Research Institute of Education Sciences and Psychology, University of Tübingen, Europastrasse 6, 72072 Tübingen, Germany. Email: norman.rose@uni-tuebingen.de
Get access Rights & Permissions [Opens in a new window]

Abstract

Item nonresponse is a common problem in educational and psychological assessments. The probability of unplanned missing responses due to omitted and not-reached items may stochastically depend on unobserved variables such as missing responses or latent variables. In such cases, missingness cannot be ignored and needs to be considered in the model. Specifically, multidimensional IRT models, latent regression models, and multiple-group IRT models have been suggested for handling nonignorable missing responses in latent trait models. However, the suitability of the particular models with respect to omitted and not-reached items has rarely been addressed. Missingness is formalized by response indicators that are modeled jointly with the researcher’s target model. We will demonstrate that response indicators have different statistical properties depending on whether the items were omitted or not reached. The implications of these differences are used to derive a joint model for nonignorable missing responses with ability to appropriately account for both omitted and not-reached items. The performance of the model is demonstrated by means of a small simulation study.

Keywords

nonignorableitem nonresponsesomitted and not-reached itemsIRT
Type
Original paper
Information
Psychometrika , Volume 82 , Issue 3 , September 2017 , pp. 795 - 819
Copyright
Copyright © 2016 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Electronic supplementary material The online version of this article (doi:10.1007/s11336-016-9544-7) contains supplementary material, which is available to authorized users.

Parts of this paper are based on the unpublished dissertation of the first author. We thank Andreas Frey and Rolf Steyer who served as members on the thesis committee. We thank the anonymous reviewers for their careful reading of our manuscript and their many insightful comments and suggestions.

References

Adams, R. J., Wilson, M., & Wang, W.. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21(1), 1–23. doi:10.1177/0146621697211001.CrossRefGoogle Scholar
Adams, R. J., Wilson, M., & Wu, M.. (1997). Multilevel item response models: An approach to errors in variables regression. Journal of Educational and Behavioral Statistics, 22(1), 47–76. doi:10.3102/10769986022001047.CrossRefGoogle Scholar
Antal, T. (2007). On the latent regression model of item response theory. Research Report No. RR-07-12. Princeton, NJ: Educational Testing Service..Google Scholar
Baker, F. B., & Kim, S.-H. (2004). Item response theory: Parameter estimation techniques (2nd ed.). Boca Raton, FL: CRC. doi:10.2307/2532822.CrossRefGoogle Scholar
Blossfeld, H.-P., Roßbach, H.-G., & von Maurice, J. (2011). Education as a lifelong process—The German national educational panel study (NEPS) [Special issue]. In Zeitschrift für Erziehungswissenschaft, 14. Wiesbaden: Springer VS..Google Scholar
Bock, R. D., & Aitkin, M.. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46, 443–459. doi:10.1007/BF02293801.CrossRefGoogle Scholar
Cai, L.. (2010). Metropolis-Hastings Robbins–Monro algorithm for confirmatory item factor analysis. Journal of Educational and Behavioral Statistics, 35(3), 307–335. doi:10.3102/1076998609353115.CrossRefGoogle Scholar
Cai, L., & Thissen, D.Reise, S. P., & Revicki, D. A.. (2015). Modern approaches to parameter estimation in item response theory. Handbook of item response theory modeling: Applications to typical performance assessment. New York: Routledge.Google Scholar
Culbertson, M. (2011, April). Is it wrong? Handling missing responses in IRT. Speech presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA..Google Scholar
De Boeck, P.. (2008). Random item IRT models. Psychometrika, 73(4), 533–559. doi:10.1007/s11336-008-9092-x.CrossRefGoogle Scholar
DeMars, C.. (2002). Incomplete data and item parameter estimates under JMLE and MML. Applied Measurement in Education, 15, 15–31. doi:10.1207/S15324818AME1501_02.CrossRefGoogle Scholar
Diggle, P., & Kenward, M. G.. (1994). Informative drop-out in longitudinal data analysis. Applied Statistics, 43(1), 49–93. doi:10.2307/2986113.CrossRefGoogle Scholar
Enders, C. K., (2010). Applied missing data analysis. New York: The Guilford Press.Google Scholar
Fisher, R. A.. (1925). Theory of statistical estimation. Proceedings of the Cambridge Philosophical Society, 22, 700–725. doi:10.1017/S0305004100009580.CrossRefGoogle Scholar
Frey, A., Hartig, J., & Rupp, A. A.. (2009). An NCME instructional module on booklet designs in large-scale assessments of student achievement: Theory and practice. Educational Measurement: Issues and Practice, 28(3), 39–53. doi:10.1111/j.1745-3992.2009.00154.x.CrossRefGoogle Scholar
Glas, C. A. W., & Pimentel, J. L.. (2008). Modeling nonignorable missing data in speeded tests. Educational and Psychological Measurement, 68(6), 907–922. doi:10.1177/0013164408315262.CrossRefGoogle Scholar
Glynn, R. J., Laird, N. M., & Rubin, D. B. (1986). Selection modeling versus mixture modeling with nonignorable nonresponses. In H. Wainer (Ed.), Drawing inferences from self-selected samples. New York: Springer. doi:10.1007/978-1-4612-4976-4_10.CrossRefGoogle Scholar
Harwell, M. R., Baker, F. B., & Zwarts, M.. (1988). Item parameter estimation via marginal maximum likelihood and an em algorithm: A didactic. Journal of Educational and Behavioral Statistics, 13(3), 243–271. doi:10.3102/10769986013003243.CrossRefGoogle Scholar
Heckman, J.. (1976). The common structure of statistical models of truncation, sample selection and limited dependent variables and a simple estimator for such models. The Annals of Economic and Social Measurement, 5, 475–492.Google Scholar
Heckman, J.. (1979). Sample selection bias as a specification error. Econometrica, 47(1), 153–161. doi:10.2307/1912352.CrossRefGoogle Scholar
Holman, R., & Glas, C. A. W.. (2005). Modelling non-ignorable missing-data mechanisms with item response theory models. British Journal of Mathematical and Statistical Psychology, 58(1), 1–17. doi:10.1111/j.2044-8317.2005.tb00312.x.Google ScholarPubMed
Hsu, Y.. (2000). On the Bock–Aitkin procedure—From an EM algorithm perspective. Psychometrika, 65, 547–549. doi:10.1007/BF02296345.CrossRefGoogle Scholar
Huisman, M.. (2000). Imputation of missing item responses: Some simple techniques. Quality & Quantity, 34, 331–351. doi:10.1023/A:1004782230065.CrossRefGoogle Scholar
Kiefer, T., Robitzsch, A., & Wu, M. (2015). Tam: Test analysis modules [Computer software manual]. R package version 1.14-0. Retrieved from http://CRAN.R-project.org/package=TAM.Google Scholar
Korobko, O. K., Glas, C. A. W., Bosker, R. J., Luyten, J. W.. (2008). Comparing the difficulty of examination subjects with item response theory. Journal of Educational Measurement, 45, 137–155. doi:10.1111/j.1745-3984.2007.00057.x.CrossRefGoogle Scholar
Little, R. J. A.. (1993). Pattern-mixture models for multivariate incomplete data. Journal of the American Statistical Association, 88, 125–134.CrossRefGoogle Scholar
Little, R. J. A.Fitzmaurice, G., Davidian, M., Verbeke, G., Molenberghs, G.. (2008). Selection and pattern-mixture models. Longitudinal data analysis. Boca Raton, FL: Chapman & Hall/CRC 409–432. doi:10.1201/9781420011579.ch18.CrossRefGoogle Scholar
Little, R. J. A., Rubin, D. B., (2002). Statistical analysis with missing data. 2New York: Wiley.CrossRefGoogle Scholar
Lord, F. M.. (1974). Estimation of latent ability and item parameters when there are omitted responses. Psychometrika, 39, 247–264. doi:10.1007/BF02291471.CrossRefGoogle Scholar
Lord, F. M.. (1983). Maximum likelihood estimation of item response parameters when some responses are omitted. Psychometrika, 48, 477–482. doi:10.1007/BF02293689.CrossRefGoogle Scholar
Ludlow, L. H., O’Leary, M.. (1999). Scoring omitted and not-reached items: Practical data analysis implications. Educational and Psychological Measurement, 59(4), 615–630. doi:10.1177/0013164499594004.CrossRefGoogle Scholar
Mislevy, R. J.. (1984). Estimating latent distributions. Psychometrika, 49(3), 359–381. doi:10.1007/BF02306026.CrossRefGoogle Scholar
Mislevy, R. J.. (1985). Estimation of latent group effects. Journal of the American Statistical Association, 80(392), 993–997. doi:10.1080/01621459.1985.10478215.CrossRefGoogle Scholar
Mislevy, R. J.. (1987). Exploiting auxiliary information about examinees in the estimation of item parameters. Applied Psychological Measurement, 11(1), 81–91. doi:10.1177/014662168701100106.CrossRefGoogle Scholar
Mislevy, R. J.. (1988). Exploiting auxiliary information about items in the estimation of Rasch item diffculty parameters. Applied Psychological Measurement, 12(3), 281–296. doi:10.1177/014662168801200306.CrossRefGoogle Scholar
Mislevy, R. J., & Wu, P. K. (1996). Missing responses and IRT ability estimation: Omits, choice, time limits, and adaptive testing. Research Report No. RR-96-30. Princeton, NJ: Educational Testing Service..Google Scholar
Molenberghs, G., Kenward, M. G., Lesaffre, E.. (1997). The analysis of longitudinal ordinal data with nonrandom drop-out. Biometrika, 84(1), 33–44. doi:10.1093/biomet/84.1.33.CrossRefGoogle Scholar
Moustaki, I., Knott, M.. (2000). Weighting for item non-response in attitude scales by using latent variable models with covariates. Journal of the Royal Statistical Society, Series A, 163, 445–459. doi:10.1111/1467-985X.00177.CrossRefGoogle Scholar
Muthén, B. O., & Muthén, L. K. (1998–2012). Mplus User’s Guide (Version 7) [Computer software manual]. Los Angeles, CA: Muthn and Muthn..Google Scholar
OECD. (2009). PISA 2009. Technical Report. Paris: OECD..Google Scholar
O’Muircheartaigh, C., Moustaki, I.. (1999). Symmetric pattern models: A latent variable approach to item non-response in attitudes scales. Journal of the Royal Statistic Society, 162, 177–194. doi:10.1111/1467-985X.00129.CrossRefGoogle Scholar
Pohl, S., Gräfe, L., Rose, N.. (2014). Dealing with omitted and not-reached items in competence tests: Evaluating approaches accounting for missing responses in item response theory models. Educational and Psychological Measurement, 74, 423–452. doi:10.1177/0013164413504926.CrossRefGoogle Scholar
Pohl, S., Haberkorn, K., Hardt, K., & Wiegand, E. (2012). NEPS technical report for reading ? scaling results of starting cohort 3 in fifth grade. NEPS Working Paper No. 15. Bamberg: Otto-Friedrich-Universitt, Nationales Bildungspanel. Retrieved from https://www.neps-data.de/de-de/projekt%C3%BCbersicht/publikationen/nepsworkingpapers.aspx.Google Scholar
Robitzsch, A., & Ldtke, O. (2015). Ein genereller Ansatz zur Modellierung fehlender Item Responses in IRT-Modellen [A general approach for modeling item nonresponses in IRT models]. Speech presented at the 12. Tagung der Fachgruppe Methoden und Evaluation, Jena, September 2015..Google Scholar
Rose, N. (2013). Item nonresponses in educational and psychological measurement. Doctoral Thesis, Friedrich-Schiller-University, Jena. Retrieved from http://d-nb.info/1036873145/34.Google Scholar
Rose, N.. (2015). Commonalities and differences in IRT-based methods for nonignorable item-nonresponses. Psychological Test and Assessment Modeling, 57, 472–478.Google Scholar
Rose, N., von Davier, M., & Xu, X. (2010). Modeling nonignorable missing data with IRT. Research Report No. RR-10-11. Princeton, NJ: Educational Testing Service..Google Scholar
Rubin, D. B.. (1976). Inference and missing data. Biometrika, 63(3), 581–592. doi:10.1093/biomet/63.3.581.CrossRefGoogle Scholar
Schafer, J. L., (1997). Analysis of incomplete multivariate data. London: Chapman & Halldoi:10.1201/9781439821862.CrossRefGoogle Scholar
Schafer, J. L., Graham, J. W.. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147–177. doi:10.1037/1082-989X.7.2.147.CrossRefGoogle ScholarPubMed
Schilling, S., Bock, R. D.. (2005). High-dimensional maximum marginal likelihood item factor analysis by adaptive quadrature. Psychometrika, 70(3), 533–555.Google Scholar
Van der Linden, W. J., Veldkamp, B. P., Carlson, J. E.. (2004). Optimizing balanced incomplete block designs for educational assessments. Applied Psychological Measurement, 28(5), 317–331. doi:10.1177/0146621604264870.CrossRefGoogle Scholar
von Davier, M.. (2009). Some notes on the reinvention of latent structure models as diagnostic classification models. Measurement: Interdisciplinary Research and Perspectives, 7, 67–74.Google Scholar
von Davier, M., DiBello, L., Yamamoto, K.Hartig, J., Klieme, E., Leutner, D.. (2008). Reporting test outcomes using models for cognitive diagnosis. Assessment of competencies in educational contexts. Cambridge, MA: Hogrefe & Huber 151–176.Google Scholar
von Davier, M., Sinharay, S.. (2010). Stochastic approximation methods for latent regression item response models. Journal of Educational and Behavioral Statistics, 35(2), 174–193. doi:10.3102/1076998609346970.CrossRefGoogle Scholar
Winship, C., & Mare, R. (1992). Models for sample selection bias. Annual Review of Sociology, 327–350. doi:10.1146/annurev.so.18.080192.001551.CrossRefGoogle Scholar
Wu, M. L., Adams, R. J., & Wilson, M. R. (1998). ACER ConQest: Generalised item response modelling software [Computer software manual]. Mebourne: Australia..Google Scholar
Supplementary material: File

Rose et al. supplementary material

Rose et al. supplementary material
Download Rose et al. supplementary material(File)
File 68.9 KB