Hostname: page-component-5f745c7db-nc56l Total loading time: 0 Render date: 2025-01-06T07:00:58.591Z Has data issue: true hasContentIssue false
  • English
  • Français

Using the Criterion-Predictor Factor Model to Compute the Probability of Detecting Prediction Bias with Ordinary Least Squares Regression

Published online by Cambridge University Press:  01 January 2025

Steven Andrew Culpepper
Steven Andrew Culpepper*
Affiliation:
Department of Statistics, University of Illinois at Urbana-Champaign
*
Requests for reprints should be sent to Steven Andrew Culpepper, Department of Statistics, University of Illinois at Urbana-Champaign, 101 Illini Hall, MC-374, 725 South Wright Street, Champaign, IL 61820, USA. E-mail: sculpepp@illinois.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

The study of prediction bias is important and the last five decades include research studies that examined whether test scores differentially predict academic or employment performance. Previous studies used ordinary least squares (OLS) to assess whether groups differ in intercepts and slopes. This study shows that OLS yields inaccurate inferences for prediction bias hypotheses. This paper builds upon the criterion-predictor factor model by demonstrating the effect of selection, measurement error, and measurement bias on prediction bias studies that use OLS. The range restricted, criterion-predictor factor model is used to compute Type I error and power rates associated with using regression to assess prediction bias hypotheses. In short, OLS is not capable of testing hypotheses about group differences in latent intercepts and slopes. Additionally, a theorem is presented which shows that researchers should not employ hierarchical regression to assess intercept differences with selected samples.

Keywords

selectionprediction biasmeasurement biastype I errorpower
Type
Original Paper
Information
Psychometrika , Volume 77 , Issue 3 , July 2012 , pp. 561 - 580
Copyright
Copyright © 2012 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

AERA APA NCME (1999). Standards for educational and psychological testing, Washington: American Educational Research AssociationGoogle Scholar
Aguinis, H., Boik, R.J., Pierce, C.A. (2001). A generalized solution for approximating the power to detect effects of categorical moderator variables using multiple regression. Organizational Research Methods, 4, 291–323CrossRefGoogle Scholar
Aguinis, H., Boik, R.J., Pierce, C.A., Beaty, J.C. (2005). Effect size and power in assessing moderating effects of categorical variables using multiple regression: a 30-year review. Journal of Applied Psychology, 90, 94–107CrossRefGoogle ScholarPubMed
Aguinis, H., Culpepper, S.A., Pierce, C.A. (2010). Revival of test bias research in preemployment testing. Journal of Applied Psychology, 95, 648–680CrossRefGoogle ScholarPubMed
Aguinis, H., Stone-Romero, E.F. (1997). Methodological artifacts in moderated multiple regression and their effects on statistical power. Journal of Applied Psychology, 82, 192–206CrossRefGoogle Scholar
Aitkin, M. (1964). Correlation in a singly truncated bivariate normal distribution. Psychometrika, 29, 263–270CrossRefGoogle Scholar
Arnold, B.C., Beaver, R.J., Groeneveld, R.A., Meeker, W.Q. (1993). The nontruncated marginal of a truncated bivariate normal distribution. Psychometrika, 58, 471–488CrossRefGoogle Scholar
Balakkrishnan, N., Lai, C. (2009). Continuous bivariate distributions, New York: SpringerGoogle Scholar
Barr, D.R., Sherrill, E.T. (1999). Mean and variance of truncated normal distributions. The American Statistician, 53, 357–361CrossRefGoogle Scholar
Bartlett, C.J., Bobko, P., Mosier, S.B., Hannan, R. (1978). Testing for fairness with a moderated regression strategy: an alternative to differential analysis. Personnel Psychology, 31, 233–241CrossRefGoogle Scholar
Birnbaum, Z.W., Paulson, E., Andrews, F.C. (1950). On the effect of selection performed on some coordinates of a multi-dimensional population. Psychometrika, 15, 191–204CrossRefGoogle ScholarPubMed
Borsboom, D., Romeijn, J., Wicherts, J.M. (2008). Measurement invariance versus selection invariance: is fair selection possible?. Psychological Methods, 13, 75–98CrossRefGoogle ScholarPubMed
Cleary, T.A. (1968). Test bias: prediction of grades of Negro and White students in integrated colleges. Journal of Educational Measurement, 5, 115–124CrossRefGoogle Scholar
Culpepper, S.A. (2010). Studying individual differences in predictability with gamma regression and nonlinear multilevel models. Multivariate Behavioral Research, 45, 153–185CrossRefGoogle ScholarPubMed
Culpepper, S.A. (in press). Evaluating EIV, OLS, and SEM estimators of group slope differences in the presence of measurement error: the single indicator case. Applied Psychological Measurement.Google Scholar
Culpepper, S.A., Aguinis, H. (2011). Analysis of covariance (ANCOVA) with fallible covariates. Psychological Methods, 16, 166–178CrossRefGoogle ScholarPubMed
Culpepper, S.A., Davenport, E.C. (2009). Assessing differential prediction of college grades by race/ethnicity with a multilevel model. Journal of Educational Measurement, 46, 220–242CrossRefGoogle Scholar
DeCorte, W., Lievens, F., Sackett, P.R. (2007). Combining predictors to achieve optimal trade-offs between selection quality and adverse impact. Journal of Applied Psychology, 92, 1380–1393CrossRefGoogle Scholar
Dorans, N.J. (2004). Using subpopulation invariance to assess test score equity. Journal of Educational Measurement, 41, 43–68CrossRefGoogle Scholar
Finch, D.M., Edwards, B.D., Wallace, J.C. (2009). Multistage selection strategies: simulating the effects on adverse impact and expected performance for various predictor combinations. Journal of Applied Psychology, 94, 318–340CrossRefGoogle ScholarPubMed
Gratz, v. Bollinger (2003). 539 U.S. 244.Google Scholar
Grutter, v. Bollinger (2003). 539 U.S. 306.Google Scholar
Holmes, D.J. (1990). The robustness of the usual correction for restriction in range due to explicit selection. Psychometrika, 55, 19–32CrossRefGoogle Scholar
Humphreys, L. (1952). Individual differences. Annual Review of Psychology, 3, 131–150CrossRefGoogle ScholarPubMed
Humpreys, L.G. (1986). An analysis and evaluation of test and item bias in the prediction context. Journal of Applied Psychology, 71, 327–333CrossRefGoogle Scholar
Kuncel, N.R., Hezlett, S.A., Ones, D.S. (2001). A comprehensive meta-analysis of the predictive validity of the graduate record examinations: implications for graduate student selection and performance. Psychological Bulletin, 127, 162–181CrossRefGoogle Scholar
Lautenschlager, G.J., Mendoza, J.L. (1986). A step-down hierarchical multiple regression analysis for examining hypotheses about test bias in prediction. Applied Psychological Measurement, 10, 133–139CrossRefGoogle Scholar
Lawley, D.N. (1943). A note on Karl Pearson’s selection formulae. Proceedings of the Royal Society of Edinburgh. Section A, Mathematical and Physical Sciences, 62, 28–30CrossRefGoogle Scholar
Linn, R.L. (1983). Pearson selection formulas: implications for studies of predictive bias and estimates of educational effects in selected samples. Journal of Educational Measurement, 20, 1–15CrossRefGoogle Scholar
Linn, R.L. (1984). Selection bias: multiple meanings. Journal of Educational Measurement, 21, 33–47CrossRefGoogle Scholar
Linn, R.L., Werts, C.E. (1971). Considerations for studies of test bias. Journal of Educational Measurement, 8, 1–4CrossRefGoogle Scholar
Meredith, W. (1993). Measurement invariance, factor analysis, and factorial invariance. Psychometrika, 58, 525–543CrossRefGoogle Scholar
Millsap, R.E. (1997). Invariance in measurement and prediction: their relationship in the single-factor case. Psychological Methods, 2, 248–260CrossRefGoogle Scholar
Millsap, R.E. (1998). Group differences in regression intercepts: implications for factorial invariance. Multivariate Behavioral Research, 33, 403–424CrossRefGoogle ScholarPubMed
Millsap, R.E. (2007). Invariance in measurement and prediction revisited. Psychometrika, 72, 461–473CrossRefGoogle Scholar
Millsap, R.E., Kwok, O.M. (2004). Evaluating the impact of partial factorial invariance on selection in two populations. Psychological Methods, 9, 93–115CrossRefGoogle ScholarPubMed
Muthen, B. (1990). Moments of the censored and truncated bivariate normal distribution. British Journal of Mathematical & Statistical Psychology, 43, 131–143CrossRefGoogle Scholar
Pearson, K. (1903). Mathematical contributions to the theory of evolution—xi. On the influence of natural selection on the variability and correlation of organs. Philosophical Transactions, CC.–A 321, 1–66Google Scholar
Ricci, v. DeStefano (2009). 557 U.S. 244.Google Scholar
Rosenbaum, S. (1961). Moments of a truncated bivariate normal distribution. Journal of the Royal Statistical Society. Series B. Methodological, 23, 405–408CrossRefGoogle Scholar
Saad, S., Sackett, P.R. (2002). Investigating differential prediction by gender in employment-oriented personality measures. Journal of Applied Psychology, 87, 667–674CrossRefGoogle ScholarPubMed
SIOP (2003). Principles for the validation and use of personnel selection procedures, (4th ed.). Bowling Green: Society for Industrial and Organizational Psychology, IncGoogle Scholar
Wicherts, J.M., Millsap, R.E. (2009). The absence of underprediction does not imply the absence of measurement bias. The American Psychologist, 64, 281–283CrossRefGoogle Scholar
Young, J.W. (2001). Differential validity, differential prediction, and college admission testing: a comprehensive review and analysis (Research Report No. 6). College Entrance Examination Board.Google Scholar