Hostname: page-component-5f745c7db-hj587 Total loading time: 0 Render date: 2025-01-06T21:18:30.208Z Has data issue: true hasContentIssue false
  • English
  • Français

Some Remarks on Applications of Tests for Detecting A Change Point to Psychometric Problems

Published online by Cambridge University Press:  01 January 2025

Sandip Sinharay Open the ORCID record for Sandip Sinharay [Opens in a new window]
Sandip Sinharay*
Affiliation:
Pacific Metrics Corporation
*
Correspondence should be made to Sandip Sinharay, Pacific Metrics Corporation, Princeton, NJ USA. Email: ssinharay@ets.org
Get access Rights & Permissions [Opens in a new window]

Abstract

Tests for a change point (e.g., Chen and Gupta, Parametric statistical change point analysis (2nd ed.). Birkhuser, Boston, 2012; Hawkins et al., J Qual Technol 35:355–366, 2003) have recently been brought into the spotlight for their potential uses in psychometrics. They have been successfully applied to detect an unusual change in the mean score of a sequence of administrations of an international language assessment (Lee and von Davier, Psychometrika 78:557–575, 2013) and to detect speededness of examinees (Shao et al., Psychometrika, 2015). The differences in the type of data used, the test statistics, and the manner in which the critical values were obtained in these papers lead to questions such as “what type of psychometric problems can be solved by tests for a change point?” and “what test statistics should be used with tests for a change point in psychometric problems?” This note attempts to answer some of these questions by providing a general overview of tests for a change point with a focus on application to psychometric problems. A discussion is provided on the choice of an appropriate test statistic and on the computation of a corresponding critical value for tests for a change point. Then, three real data examples are provided to demonstrate how tests for a change point can be used to make important inferences in psychometric problems. The examples include some clarifications and remarks on the critical values used in Lee and von Davier (Psychometrika, 78:557–575, 2013) and Shao et al. (Psychometrika, 2015). The overview and the examples provide insight on tests for a change point above and beyond Lee and von Davier (Psychometrika, 78:557–575, 2013) and Shao et al. (Psychometrika, 2015). Thus, this note extends the research of Lee and von Davier (Psychometrika, 78:557–575, 2013) and Shao et al. (Psychometrika, 2015) on tests for a change point.

Keywords

likelihood ratio teststatistical process controlstatistical quality control
Type
Original Paper
Information
Psychometrika , Volume 82 , Issue 4 , December 2017 , pp. 1149 - 1161
Copyright
Copyright © 2016 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Allalouf, A. (2007). Quality control procedures in the scoring, equating, and reporting of test scores. Educational Measurement: Issues and Practice 26 (1), 36–46CrossRefGoogle Scholar
Andrews, D. (1993). Tests for parameter instability and structural change with unknown change point. Econometrica 6 (61), 821–856CrossRefGoogle Scholar
Benjamini, Y. & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B (Methodological) 57, 289–300CrossRefGoogle Scholar
Bradlow, E. Weiss, R. E. & Cho, M. (1998). Bayesian detection of outliers in computerized adaptive tests. Journal of the American Statistical Association 93, 910–919CrossRefGoogle Scholar
Chen, J. & Gupta, A. K. (2012). Parametric statistical change point analysis 2Boston, MA: BirkhuserCrossRefGoogle Scholar
Cizek, G. J., & Wollack, J. A. (2017). Handbook of detecting cheating on tests. Washington, DC: Routledge.Google Scholar
Csorgo, M. & Horvath, L. (1997). Limit theorems in change-point analysis New York, NY: WileyGoogle Scholar
Estrella, A., & Rodrigues, A. (2005). One-sided test for an unknown breakpoint: Theory, computation, and application to monetary theory (Staff Reports No. 232). Federal Reserve Bank of New York.Google Scholar
Gombay, E. & Horvath, L. (1996). On the rate of approximations for maximum likelihood tests in change-point models. Journal of Multivariate Analysis 56, 120–152CrossRefGoogle Scholar
Hawkins, D. Qiu, P. & Kang, C. (2003). The changepoint model for statistical process control. Journal of Quality Technology 35, 355–366CrossRefGoogle Scholar
Kingsbury, G. G. & Zara, A. R. (1989). Procedures for selecting items for computerized adaptive tests. Applied Measurement in Education 2, 359–375CrossRefGoogle Scholar
Lee, Y.-H. & Haberman, S. J. (2013). Harmonic regression and scale stability. Psychometrika 78, 557–575CrossRefGoogle ScholarPubMed
Lee, Y.-H. & von Davier, A. A. (2013). Monitoring scale scores over time via quality control charts, model-based approaches, and time series techniques. Psychometrika 78, 557–575CrossRefGoogle ScholarPubMed
Meijer, R. R. (2002). Outlier detection in high-stakes certification testing. Journal of Educational Measurement 39, 219–233CrossRefGoogle Scholar
Montgomery, D. C. (2013). Introduction to statistical quality control New York, NY: WileyGoogle Scholar
Olson, J. F. & Fremer, J. (2013). TILSA test security guidebook: Preventing, detecting, and investigating test securities irregularities Washington, DC: Council of Chief State School OfficersGoogle Scholar
R Core Team. (2016). R: A language and environment for statistical computing. Vienna: Austria.Google Scholar
Sen, A. K. & Srivastava, M. S. (1975). On tests for detecting a change in mean. Annals of Statistics 3, 98–108CrossRefGoogle Scholar
Shao, C., Kim, D., Cheng, Y., & Luo, X. (2015, April). Detection of warm-up effect in cat using change-point analysis. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.Google Scholar
Shao, C., Li, J., & Cheng, Y. (2015). A change point based method for test speededness detection. Psychometrika. (Advance online publication. doi:10.1007/s11336-015-9476-7).CrossRefGoogle Scholar
Sinharay, S. (2016). Person fit analysis in computerized adaptive testing using tests for a change point. Journal of Educational and Behavioral Statistics 41, 521–549CrossRefGoogle Scholar
Sinharay, S. (in press). Detection of item preknowledge using likelihood ratio test and score test. Journal of Educational and Behavioral Statistics.Google Scholar
Sullivan, J. H. & Woodall, W. H. (1996). A control chart for preliminary analysis of individual observations. Journal of Quality Technology 28, 265–278CrossRefGoogle Scholar
Thisted, R. A. (1988). Elements of statistical computing: Numerical computation London: Chapman and HallGoogle Scholar
van Krimpen-Stoop, EMLA Meijer, R. R. van der Linden, W. J. & Glas, C. A. (2000). Detecting person misfit in adaptive testing using statistical process control techniques. Computerized adaptive testing: Theory and practice Netherlands: Springer 201–219CrossRefGoogle Scholar
von Davier, A. A. (2012). The use of quality control and data mining techniques for monitoring scaled scores: An overview (ETS Research Report No. RR-12-20). Princeton, NJ: ETS.Google Scholar
Woodall, W. H. & Montgomery, D. C. (1999). Research issues and ideas in statistical process control. Journal of Quality Technology 31, 376–386CrossRefGoogle Scholar
Worsley, K. J. (1979). On the likelihood ratio test for a shift in location of normal populations. Journal of the American Statistical Association 74, 365–367Google Scholar