Hostname: page-component-cd9895bd7-fscjk Total loading time: 0 Render date: 2025-01-06T03:09:37.078Z Has data issue: false hasContentIssue false
  • English
  • Français

Using Response Times to Detect Aberrant Responses in Computerized Adaptive Testing

Published online by Cambridge University Press:  01 January 2025

Wim J. van der Linden  and
Edith M. L. A. van Krimpen-Stoop
Wim J. van der Linden*
Affiliation:
University of Twente
Edith M. L. A. van Krimpen-Stoop
Affiliation:
University of Twente
*
Requests for reprints should be sent to W.J. van der Linden, Department of Research Methodology, Measurement and Data Analysis, University of Twente, P.O. Box 217, 7500 AE Enschede, THE NETHERLANDS. E-Mail: w.j.vanderlinden@utwente.nl
Get access Rights & Permissions [Opens in a new window]

Abstract

A lognormal model for response times is used to check response times for aberrances in examinee behavior on computerized adaptive tests. Both classical procedures and Bayesian posterior predictive checks are presented. For a fixed examinee, responses and response times are independent; checks based on response times offer thus information independent of the results of checks on response patterns. Empirical examples of the use of classical and Bayesian checks for detecting two different types of aberrances in response times are presented. The detection rates for the Bayesian checks outperformed those for the classical checks, but at the cost of higher false-alarm rates. A guideline for the choice between the two types of checks is offered.

Keywords

aberrant response patternscomputerized adaptive testingposterior predictive checksperson misfitresidual analysisresponse times
Type
Articles
Information
Psychometrika , Volume 68 , Issue 2 , June 2003 , pp. 251 - 265
Copyright
Copyright © 2003 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

This study received funding from the Law School Admission Council (LSAC). The opinions and conclusions contained in this paper are those of the authors and do not necessarily reflect the policy and position of LSAC. The authors are most indebted to Wim M. M. Tielen for his computational assistance and to the US Defense Manpower Data Center for the permission to use the ASVAB data set in the empirical examples.

References

Bradlow, E.T., Weiss, R. E., Cho, M. (1998). Bayesian detection of outliers in computerized adaptive tests. Journal of the American Statistical Association, 93, 910–919.CrossRefGoogle Scholar
Drasgow, F., Levine, M.V., Williams, E.A. (1985). Appropriateness measurement with polytomous item reponse models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38, 67–86.CrossRefGoogle Scholar
Gelman, A., Carlin, J.B., Stern, H., Rubin, D.B. (1995). Bayesian data analysis. London, U.K.: Chapman & Hall.CrossRefGoogle Scholar
Johnson, V.E., Albert, J.H. (1999). Ordinal data modeling. New York, NY: Springer-Verlag.CrossRefGoogle Scholar
Levine, M.V., Rubin, D.B. (1979). Measuring the appropriateness of multiple-choice test scores. Journal of Educational Statistics, 4, 269–290.CrossRefGoogle Scholar
Meijer, R.R., Sijtsma, K. (1995). Detection of aberrant item response patterns: A review of recent developments. Applied Measurement in Education, 8, 261–272.CrossRefGoogle Scholar
Mislevy, R.J., Chang, H. (2000). Does adaptive testing violate local independence?. Psychometrika, 65, 149–156.CrossRefGoogle Scholar
Mislevy, R.J., Wu, P.-K. (1996). Missing responses and Bayesian IRT estimation: Omits, choice, time limits, and adaptive testing. Princeton, NJ: Educational Testing Service.Google Scholar
Molenaar, I.W., Hoijtink, H. (1990). The many null distributions of person-fit statistics. Psychometrika, 55, 75–106.CrossRefGoogle Scholar
Neter, J., Wasserman, W., Kutner, M.H. (1985). Applied linear statistical models: Regression, analysis of variance, and experimental designs. Homewood, IL: Richard D. Irwin.Google Scholar
Segall, D.O., Moreno, K.E., Hetter, D.H. (1997). In Sands, W.A., Waters, B.K., McBride, J.R. (Eds.), Computerized adaptive testing: From inquiry to operation (pp. 117–130). Washington, DC: American Psychological Association.CrossRefGoogle Scholar
Schnipke, D.L., Scrams, D.J. (1997). Representing response time information in item banks. Newtown, PA: Law School Admission Council.Google Scholar
Thissen, D. (1983). Timed testing: An approach using item resonse theory. In Weiss, D.J. (Eds.), New horizons in testing: Latent trait test theory and computerized adaptive testing (pp. 179–203). New York, NY: Academic Press.Google Scholar
Trabin, T.E., Weiss, D.J. (1983). The person response curve: Fit of individuals to item response theory models. In Weiss, D.J. (Eds.), New horizons in testing: Latent trait theory and computerized adaptive testing. New York, NY: Academic Press.Google Scholar
van der Linden, W.J. (2002). A model for speed and accuracy on tests. Unpublished manuscipt.Google Scholar
van der Linden, W.J., Pashley, P.J. (2000). Item selection and ability estimation in adaptive testing. In van der Linden, W.J., Glas, C.A.W. (Eds.), Computerized adaptive testing: Theory and practice (pp. 1–25). Norwell, MA: Kluwer Academic Publishers.CrossRefGoogle Scholar
van der Linden, W.J., Scrams, D.J., Schnipke, D.L. (1999). Using response-time constraints to control for speededness in computerized adaptive testing. Applied Psychological Measurement, 23, 195–210.CrossRefGoogle Scholar
van Krimpen-Stoop, E.M.L.A., Meijer, R.R. (1999). Simulating the null distribution of person-fit statistics for conventional and adaptive tests. Applied Psychological Measurement, 23, 327–345.CrossRefGoogle Scholar
van Krimpen-Stoop, E.M.L.A., Meijer, R.R. (2000). Detecting person misfit in adaptive testing using statistical process control techniques. In van der Linden, W.J., Glas, C.A.W. (Eds.), Computerized adaptive testing: Theory and practice (pp. 221–219). Norwell, MA: Kluwer Academic Publishers.Google Scholar