Hostname: page-component-5f745c7db-hj587 Total loading time: 0 Render date: 2025-01-06T21:20:56.308Z Has data issue: true hasContentIssue false
  • English
  • Français

Modelling Conditional Dependence Between Response Time and Accuracy

Published online by Cambridge University Press:  01 January 2025

Maria Bolsinova ,
Paul de Boeck  and
Jesper Tijmstra
Maria Bolsinova*
Affiliation:
Utrecht University CITO, Dutch National Institute for Educational Measurement University of Amsterdam
Paul de Boeck
Affiliation:
Ohio State University KU Leuven
Jesper Tijmstra
Affiliation:
Tilburg University
*
Correspondence should be made to Maria Bolsinova, Department of Psychology, University of Amsterdam, Nieuweachtergracht 129, 1018 WS Amsterdam, The Netherlands. Email: m.a.bolsinova@uva.nl
Get access Rights & Permissions [Opens in a new window]

Abstract

The assumption of conditional independence between response time and accuracy given speed and ability is commonly made in response time modelling. However, this assumption might be violated in some cases, meaning that the relationship between the response time and the response accuracy of the same item cannot be fully explained by the correlation between the overall speed and ability. We propose to explicitly model the residual dependence between time and accuracy by incorporating the effects of the residual response time on the intercept and the slope parameter of the IRT model for response accuracy. We present an empirical example of a violation of conditional independence from a low-stakes educational test and show that our new model reveals interesting phenomena about the dependence of the item properties on whether the response is relatively fast or slow. For more difficult items responding slowly is associated with a higher probability of a correct response, whereas for the easier items responding slower is associated with a lower probability of a correct response. Moreover, for many of the items slower responses were less informative for the ability because their discrimination parameters decrease with residual response time.

Keywords

response timeshierarchical modelconditional independenceitem response theory
Type
Original Paper
Information
Psychometrika , Volume 82 , Issue 4 , December 2017 , pp. 1126 - 1148
Copyright
Copyright © 2016 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Electronic supplementary material The online version of this article (doi:10.1007/s11336-016-9537-6) contains supplementary material, which is available to authorized users.

References

Birnbaum, A. Lord, F. M. & Novick, M. R. (1968). Some latent trait models and their use in inferring an examinee’s ability. Statistical theories of mental test scores Reading: Addison-Wesley 395–479Google Scholar
Bloxom, B. (1985). Considerations in psychometric modeling of response time. Psychometrika 50 (4), 383–397CrossRefGoogle Scholar
Bolsinova, M. & Maris, G. (2016). A test for conditional independence between response time and accuracy. British Journal of Mathematical and Statistical Psychology 69, 62–79CrossRefGoogle ScholarPubMed
Bolsinova, M. & Tijmstra, J. (2016). Posterior predictive checks for conditional independence between response time and accuracy. Journal of Educational and Behavioural Statistics 41, 123–145Google Scholar
Brooks, S. & Gelman, A. (1998). General methods for monitoring convergence of iterative simulations. Journal of Computational and Graphical Statistics 7 (4), 434–455CrossRefGoogle Scholar
Casella, G. & George, E. (1992). Explaining the Gibbs sampler. The American Statistician 43 (3), 167–174CrossRefGoogle Scholar
Coyle, T. (2003). A review of the worst performance rule: Evidence, theory, and alternative hypotheses. Intelligence 31 (6), 567–587CrossRefGoogle Scholar
Gelman, A. Meng, X.-L. & Stern, H. (1996). Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica 6, 733–807Google Scholar
Gelman, A. & Rubin, D. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science 7 (4), 457–472CrossRefGoogle Scholar
Geman, S. & Geman, D. (1984). Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence 6, 721–741CrossRefGoogle ScholarPubMed
Goldhammer, F. & Klein Entink, R. (2011). Speed of reasoning and its relation to reasoning ability. Intelligence 39, 108–119CrossRefGoogle Scholar
Goldhammer, F. Naumann, J. Stelter, A. Tóth, K. Rölke, H. & Klieme, E. (2014). The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment. Journal of Educational Psychology 106 (3), 608–626CrossRefGoogle Scholar
Goldhammer, F. Naumann, J. & Greiff, S. (2015). More is not always better: The relation between item response and item response time in Raven’s matrices.Journal of. Intelligence 3 (1), 21–40CrossRefGoogle Scholar
Hoff, P. D. (2009). A first course in Bayesian statistical methods New York: SpringerCrossRefGoogle Scholar
Huang, A. & Wand, M. (2013). Simple marginally noninformative prior distributions for covariance matrices. Bayesian Analysis 8 (2), 439–452CrossRefGoogle Scholar
Klein Entink, R. Kuhn, J. Hornke, L. & Fox, J. P. (2009). Evaluating cognitive theory: A joint modeling approach using responses and response times. Psychological methods 14 (1), 54–75CrossRefGoogle ScholarPubMed
Loeys, T. Rossel, Y. & Baten, K. (2011). A joint modelling approach for reaction time and accuracy in psycholinguistic experiments. Psychometrika 76 (3), 487–503CrossRefGoogle Scholar
Luce, R. D. (1986). Response times: Their role in inferring elementary mental organization New York: Oxford University PressGoogle Scholar
Marsman, M., Maris, G., Bechger, T., & Glas, C. A. (2014). Composition algorithms for conditional distributions. Manuscript submitted for publication.Google Scholar
Meng, X. L. (1994). Posterior predictive p-values. The Annals of Statistics 22 (3), 1142–1160CrossRefGoogle Scholar
Metropolis, N. Rosenbluth, A. Rosenbluth, M. Teller, A. & Teller, E. (1953). Equations of state calculations by fast computing machines. Journal of Chemical Physics 21, 1087–1092CrossRefGoogle Scholar
Partchev, I. & De Boeck, P. (2012). Can fast and slow intelligence be differentiated?. Intelligence 40, 23–32CrossRefGoogle Scholar
Petscher, Y. Mitchell, A. & Foorman, B. (2015). Improving the reliability of student scores from speeded assessments: An illustration of conditional item response theory using a computer-administered measure of vocabulary. Reading and writing 28 (1), 31–56CrossRefGoogle ScholarPubMed
R Development Core Team. (2006). R: A language and environment for statistical computing. Vienna: Austria R Foundation for Statistical Computing.Google Scholar
Ranger, J. & Ortner, T. (2012). The case of dependency of responses and response times: A modeling approach based on standard latent trait models. Psychological Test and Assessment Modeling 54 (2), 128–148Google Scholar
Roskam, E. E. Roskam, E. E. & Suck, R. (1987). Toward a psychometric theory of intelligence. Progress in mathematical psychology Amsterdam: North-Holland 151–171Google Scholar
Scherer, R. Greiff, S. & Hautamäki, J. (2015). Exploring the relation between time on task and ability in complex problem solving. Intelligence 48, 37–50CrossRefGoogle Scholar
Sinharay, S. Johnson, M. & Stern, H. (2006). Posterior predictive assessment of item response theory models. Applied Psychological Measurement 30, 298–321CrossRefGoogle Scholar
Spiegelhalter, D. J. Best, N. G. Carlin, B. & van der Linde, A. (2002). Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society Series B (Statistical Methodology) 64, 583–639CrossRefGoogle Scholar
van Breukelen, GJP (2005). Psychometric modeling of response speed and accuracy with mixed and conditional regression. Psychometrika 70 (2), 359–376CrossRefGoogle Scholar
van Breukelen, G. Roskam, E. Doignon, J. & Falmagne, R. J. (1991). A Rasch model for the speed-accuracy trade of in time-limited tests. Mathematical psychology: Current developments New York: Springer 251–271CrossRefGoogle Scholar
van der Linden, W. J. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics 31, 181–204CrossRefGoogle Scholar
van der Linden, W. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika 72, 287–308CrossRefGoogle Scholar
van der Linden, W. J. (2008). Using response times for item selection in adaptive testing. Journal of Educational and Behavioral Statistics 33 (1), 5–20CrossRefGoogle Scholar
van der Linden, W. J. (2009). Conceptual issues in response-time modeling. Journal of Educational Measurement 46 (3), 247–272CrossRefGoogle Scholar
van der Linden, W. J. & Glas, CAW (2010). Statistical tests of conditional independence between responses and/or response times on test items. Psychometrika 75, 120–139CrossRefGoogle Scholar
van der Linden, W. J. & Guo, F. (2008). Bayesian procedures for identifying aberrant response-time patterns in adaptive testing. Psychometrika 73 (3), 365–384CrossRefGoogle Scholar
Verhelst, N. D. Verstralen, HHFM Jansen, M. G. van der Linden, W. J. & Hambleton, R. K. (1997). A logistic model for time-limit tests. Handbook of modern item response theory New York: Springer 169–185CrossRefGoogle Scholar
Wang, T. (2006). A model for the joint distribution of item response and response time using one-parameter Weibull distribution (CASMA Research Report 20). Iowa City: IA Center for Advanced Studies in Measurement and Assessment.Google Scholar
Wang, T. & Hanson, B. A. (2005). Development and calibration of an item response model that incorporates response time. Applied Psychological Measurement 29, 323–339CrossRefGoogle Scholar
Supplementary material: File

Bolsinova et al. supplementary material

Bolsinova et al. supplementary material
Download Bolsinova et al. supplementary material(File)
File 51.2 KB