Hostname: page-component-745bb68f8f-lrblm Total loading time: 0 Render date: 2025-01-07T22:03:01.871Z Has data issue: false hasContentIssue false
  • English
  • Français

The Asymptotic Distribution of Average Test Overlap Rate in Computerized Adaptive Testing

Published online by Cambridge University Press:  01 January 2025

Edison M. Choe Open the ORCID record for Edison M. Choe [Opens in a new window]  and
Hua-Hua Chang
Edison M. Choe*
Affiliation:
Graduate Management Admission Council™ (GMAC™)
Hua-Hua Chang
Affiliation:
Purdue University
*
Correspondence should be made to Edison M. Choe, Graduate Management Admission Council™ (GMAC™), 11921 Freedom Drive, Suite 300, Reston, VA 20190, USA. Email: echoe@gmac.com
Get access Rights & Permissions [Opens in a new window]

Abstract

The average test overlap rate is often computed and reported as a measure of test security risk or item pool utilization of a computerized adaptive test (CAT). Despite the prevalent use of this sample statistic in both literature and operations, its sampling distribution has never been known nor studied in earnest. In response, a proof is presented for the asymptotic distribution of a linear transformation of the average test overlap rate in fixed-length CAT. The theoretical results enable the estimation of standard error and construction of confidence intervals. Moreover, a practical simulation study demonstrates the statistical comparison of average test overlap rates between two CAT designs with different exposure control methods.

Keywords

test overlapitem exposuretest securitypool utilizationcomputerized adaptive testingasymptotic theory
Type
Original Paper
Information
Psychometrika , Volume 84 , Issue 4 , December 2019 , pp. 1129 - 1151
Copyright
Copyright © 2019 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Baker, F. B.,Kim, S-. H.. (2004) Item response theory: Parameter estimation techniques, 2 New York, NY: Marcel Dekker. CrossRefGoogle Scholar
Barrada, J., Olea, J., Ponsoda, V., & Abad, F. J.. (2009). Test overlap rate and item exposure as indicators of test security in CATs. In D. J. Weiss (Ed.),Proceedings of the 2009 GMAC conference on computerized adaptive testing.Google Scholar
Chang, H. -H.. (2015). Psychometrics behind computerized adaptive testing. Psychometrika, 80 (1) 1– 20. CrossRefGoogle ScholarPubMed
Chang, H. -H.,Qian, J., &Ying, Z.(2001). a-Stratified multistage computerized adaptive testing with b-blocking. Applied Psychological Measurement, 25 (4) 333– 341. CrossRefGoogle Scholar
Chang, H. -H.,Ying, Z.(1999). a-Stratified multistage computerized adaptive testing. Applied Psychological Measurement, 23 (3) 211– 222. CrossRefGoogle Scholar
Chang, H-. H.,Ying, Z..(2009). Nonlinear sequential designs for logistic item response theory models with applications to computerized adaptive tests. The Annals of Statistics, 37 (3) 1466– 1488. CrossRefGoogle Scholar
Chang, H-. H.,Zhang, J.(2002). Hypergeometric family and item overlap rates in computerized adaptive testing. Psychometrika, 67 (3) 387– 398. CrossRefGoogle Scholar
Chen, S-. Y.,Ankenmann, R. D., &Spray, J. A.(2003). The relationship between item exposure and test overlap in computerized adaptive testing. Journal of Educational Measurement, 40 (2) 129– 145. CrossRefGoogle Scholar
Choe, E. M.,Kern, J. L., &Chang, H-. H.(2018). Optimizing the use of response times for item selection in computerized adaptive testing. Journal of Educational and Behavioral Statistics, 43 (2) 135– 158. CrossRefGoogle Scholar
Cochran, W. G..(1977). Sampling techniques, 3 New York, NY: Wiley. Google Scholar
Han, K. T..(2016). Maximum likelihood score estimation method with fences for short-length tests and computerized adaptive tests. Applied Psychological Measurement, 40 (4) 289– 301. CrossRefGoogle ScholarPubMed
Hau, K-. T.,Chang, H-. H..(2001). Item selection in computerized adaptive testing: Should more discriminating items be used first?. Journal of Educational Measurement, 38 (3) 249– 266. CrossRefGoogle Scholar
Kingsbury, G. G.,Zara, A. R.(1989). Procedures for selecting items for computerized adaptive tests. Applied Measurement in Education, 2 (4) 359– 375. CrossRefGoogle Scholar
Lord, F. M.(1980). Applications of item response theory to practical testing problems, Hillsdale, NJ: Erlbaum. Google Scholar
Lord, F. M.,Novick, M. R..(1968). Statistical theories of mental test scores, Reading, MA: Addison-Wesley. Google Scholar
Mislevy, R.J.,Chang, H-. H..(2000). Does adaptive testing violate local independence?. Psychometrika, 65 (2) 149– 156. CrossRefGoogle Scholar
Rencher, A. C.,Schaalje, G. B..(2008). Linear models in statistics, 2 Hoboken, NJ: Wiley-Interscience. Google Scholar
Revuelta, J.,Ponsoda, V..(1998). A comparison of item exposure control methods in computerized adaptive testing. Journal of Educational Measurement, 35 (4) 311– 327. CrossRefGoogle Scholar
Sympson, J. B., & Hetter, R. D. (1985). Controlling item-exposure rates in computerized adaptive testing. In Proceedings of the 27th annual meeting of the Military Testing Association. San Diego, CA: Navy Personnel Research and Development Center.Google Scholar
van der Linden, W. J.,Veldkamp, B.P..(2007). Conditional item-exposure control in adaptive testing using item-ineligibility probabilities. Journal of Educational and Behavioral Statistics, 32 (4) 398– 418. CrossRefGoogle Scholar
Wang, C.,Zheng, Y., &Chang, H-H..(2014). Does standard deviation matter? Using “standard deviation” to quantify security of multistage testing. Psychometrika, 79 (1) 154– 174. 10.1007/s11336-013-9356-y CrossRefGoogle ScholarPubMed
Way, W. D..(1998). Protecting the integrity of computerized testing item pools. Educational Measurement: Issues and Practice, 17 17– 27. CrossRefGoogle Scholar