Hostname: page-component-5f745c7db-6bmsf Total loading time: 0 Render date: 2025-01-06T06:56:57.204Z Has data issue: true hasContentIssue false
  • English
  • Français

An Improved Correction for Range Restricted Correlations Under Extreme, Monotonic Quadratic Nonlinearity and Heteroscedasticity

Published online by Cambridge University Press:  01 January 2025

Steven Andrew Culpepper
Steven Andrew Culpepper*
Affiliation:
University of Illinois at Urbana-Champaign
*
Correspondence should be made to Steven Andrew Culpepper, Department of Statistics, University of Illinois at Urbana-Champaign, 725 South Wright Street, Champaign, IL 61820 USA. Email: sculpepp@illinois.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

Standardized tests are frequently used for selection decisions, and the validation of test scores remains an important area of research. This paper builds upon prior literature about the effect of nonlinearity and heteroscedasticity on the accuracy of standard formulas for correcting correlations in restricted samples. Existing formulas for direct range restriction require three assumptions: (1) the criterion variable is missing at random; (2) a linear relationship between independent and dependent variables; and (3) constant error variance or homoscedasticity. The results in this paper demonstrate that the standard approach for correcting restricted correlations is severely biased in cases of extreme monotone quadratic nonlinearity and heteroscedasticity. This paper offers at least three significant contributions to the existing literature. First, a method from the econometrics literature is adapted to provide more accurate estimates of unrestricted correlations. Second, derivations establish bounds on the degree of bias attributed to quadratic functions under the assumption of a monotonic relationship between test scores and criterion measurements. New results are presented on the bias associated with using the standard range restriction correction formula, and the results show that the standard correction formula yields estimates of unrestricted correlations that deviate by as much as 0.2 for high to moderate selectivity. Third, Monte Carlo simulation results demonstrate that the new procedure for correcting restricted correlations provides more accurate estimates in the presence of quadratic and heteroscedastic test score and criterion relationships.

Keywords

range restrictionpersonnel selectioncollege admissionsnonlinearityheteroscedasticity
Type
Article
Information
Psychometrika , Volume 81 , Issue 2 , June 2016 , pp. 550 - 564
Copyright
Copyright © 2015 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Electronic supplementary material The online version of this article (doi:10.1007/s11336-015-9466-9) contains supplementary material, which is available to authorized users.

References

Aguinis, H., Whitehead, R. (1997). Sampling variance in the correlation coefficient under indirect range restriction: Implications for validity generalization. Journal of Applied Psychology, 82(4), 528–538CrossRefGoogle Scholar
Alexander, R.A., Alliger, G.M., Hanges, P.J. (1984). Correcting for range restriction when the population variance is unknown. Applied Psychological Measurement, 8(4), 431–437CrossRefGoogle Scholar
Arneson, J.J., Sackett, P.R., Beatty, A.S. (2011). Ability-performance relationships in education and employment settings critical tests of the more-is-better and the good-enough hypotheses. Psychological Science, 22(10), 1336–1342CrossRefGoogle ScholarPubMed
Berry, C.M., Sackett, P.R., Sund, A. (2013). The role of range restriction and criterion contamination in assessing differential validity by race/ethnicity. Journal of Business and Psychology, 28, 345–359CrossRefGoogle Scholar
Brillinger, D. (1969). The calculation of cumulants via conditioning. Annals of the Institute of Statistical Mathematics, 21, 215–218CrossRefGoogle Scholar
Chan, W., Chan, D. (2004). Bootstrap standard error and confidence intervals for the correlation corrected for range restriction: A simulation study. Psychological Methods, 9, 369–385CrossRefGoogle ScholarPubMed
Chernyshenko, O.S., Ones, D.S. (1999). How selective are psychology graduate programs? The effect of the selection ratio on GRE score validity. Educational and Psychological Measurement, 59(6), 951–961CrossRefGoogle Scholar
Coward, W.M., Sackett, P.R. (1990). Linearity of ability-performance relationships: A reconfirmation. Journal of Applied Psychology, 75(3), 297–300CrossRefGoogle Scholar
Cullen, M.J., Hardison, C.M., Sackett, P.R. (2004). Using SAT-grade and ability-job performance relationships to test predictions derived from stereotype threat theory. Journal of Applied Psychology, 89(2), 220–230CrossRefGoogle ScholarPubMed
Culpepper, S.A. (2010). Studying individual differences in predictability with gamma regression and nonlinear multilevel models. Multivariate Behavioral Research, 45(1), 153–185CrossRefGoogle ScholarPubMed
Dobson, P. (1988). The correction of correlation coefficients for restriction of range when restriction results from the truncation of a normally distributed variable. British Journal of Mathematical and Statistical Psychology, 41(2), 227–234CrossRefGoogle Scholar
Fife, D.A., Mendoza, J.L., Terry, R. (2013). Revisiting case IV: A reassessment of bias and standard errors of case IV under range restriction. British Journal of Mathematical and Statistical Psychology, 66(3), 521–542CrossRefGoogle Scholar
Greener, J., Osburn, H. (1979). An empirical study of the accuracy of corrections for restriction in range due to explicit selection. Applied Psychological Measurement, 3, 31–41CrossRefGoogle Scholar
Greener, J., Osburn, H. (1980). Accuracy of corrections for restriction in range due to explicit selection in heteroscedastic and nonlinear distributions. Educational and Psychological Measurement, 40, 337–346CrossRefGoogle Scholar
Gross, A. (1982). Relaxing the assumptions underlying corrections for restriction of range. Educational and Psychological Measurement, 42(3), 795–801CrossRefGoogle Scholar
Gross, A. (1990). A maximum likelihood approach to test validation with missing and censored dependent variables. Psychometrika, 55(3), 533–549CrossRefGoogle Scholar
Gross, A. (1997). Interval estimation of bivariate correlations with missing data on both variables: A Bayesian approach. Journal of Educational and Behavioral Statistics, 22(4), 407–424CrossRefGoogle Scholar
Gross, A. (2000). Bayesian interval estimation of multiple correlations with missing data: A Gibbs sampling approach. Multivariate Behavioral Research, 35(2), 201–227CrossRefGoogle Scholar
Gross, A., Fleischman, L. (1983). Restriction of range corrections when both distribution and selection assumptions are violated. Applied Psychological Measurement, 7, 227–237CrossRefGoogle Scholar
Gross, A., Fleischman, L. (1987). The correction for restriction of range and nonlinear regressions: An analytic study. Applied Psychological Measurement, 11, 211–217CrossRefGoogle Scholar
Gross, A., Perry, P. (1983). Validating a selection test, a predictive probability approach. Psychometrika, 48(1), 113–127CrossRefGoogle Scholar
Gross, A., Torres-Quevedo, R. (1995). Estimating correlations with missing data, a Bayesian approach. Psychometrika, 60(3), 341–354CrossRefGoogle Scholar
Hagglund, G., Larsson, R. (2006). Estimation of the correlation coefficient based on selected data. Journal of Educational and Behavioral Statistics, 31, 377–411CrossRefGoogle Scholar
Harvey, A. (1976). Estimating regression models with multiplicative heteroscedasticity. Econometrica, 44, 461–465CrossRefGoogle Scholar
Held, J., Foley, P. (1994). Explanations for accuracy of the general multivariate formulas in correcting for range restriction. Applied Psychological Measurement, 518, 355–367CrossRefGoogle Scholar
Holmes, D. (1990). The robustness of the usual correction for restriction in range due to explicit selection. Psychometrika, 55, 19–32CrossRefGoogle Scholar
Hsu, J. (1995). Sampling behaviour in estimating predictive validity in the context of selection and latent variable modelling: A Monte Carlo study. British Journal of Mathematical and Statistical Psychology, 48, 75–97CrossRefGoogle Scholar
Hunter, J., Schmidt, F., Le, H. (2006). Implications of direct and indirect range restriction for meta-analysis methods and findings. Journal of Applied Psychology, 91, 594–612CrossRefGoogle ScholarPubMed
Kobrin, J., Sinharay, S., Haberman, S., & Chajewski, M. (2011). An investigation of the fit of linear regression models to data from an SAT validity study. Tech. Rep. 2011–3, College Board Research Report.CrossRefGoogle Scholar
Lawley, D. (1944). A note on Karl Pearson’s selection formulæ. Proceedings of the Royal Society of Edinburgh Section A Mathematical and Physical Sciences, 62(01), 28–30CrossRefGoogle Scholar
Le, H., Oh, I., Robbins, S., Ilies, R., Holland, E., Westrick, P. (2011). Too much of a good thing: Curvilinear relationships between personality traits and job performance. Journal of Applied Psychology, 96, 113–133CrossRefGoogle ScholarPubMed
Lee, R., Foley, P. (1986). Is the validity of a test constant throughout the test score range?. Journal of Applied Psychology, 71, 641–644CrossRefGoogle Scholar
Li, J., Chan, W., Cui, Y. (2011). Bootstrap standard error and confidence intervals for the correlations corrected for indirect range restriction. British Journal of Mathematical and Statistical Psychology, 64(3), 367–387CrossRefGoogle ScholarPubMed
Li, J., Cui, Y., Chan, W. (2013). Bootstrap confidence intervals for the mean correlation corrected for case IV range restriction: A more adequate procedure for meta-analysis. Journal of Applied Psychology, 98(1), 183–193CrossRefGoogle ScholarPubMed
Linn, R. (1983). Pearson selection formulas: Implications for studies of predictive bias and estimates of educational effects in selected samples. Journal of Educational Measurement, 20, 1–15CrossRefGoogle Scholar
Linn, R. (1984). Selection bias: Multiple meanings. Journal of Educational Measurement, 21, 33–47CrossRefGoogle Scholar
Little, R.J., Rubin, D.B. (2014). Statistical analysis with missing data, Hoboken, NJ: John Wiley & Sons IncGoogle Scholar
Mendoza, J. (1993). Fisher transformations for correlations corrected for selection and missing data. Psychometrika, 58(4), 601–615CrossRefGoogle Scholar
Mendoza, J., Bard, D.E., Mumford, M., Ang, S.C. (2004). Criterion-related validity in multiple-hurdle designs: Estimation and bias. Organizational Research Methods, 7(4), 418–441CrossRefGoogle Scholar
Mendoza, J., Mumford, M. (1987). Corrections for attenuation and range restriction on the predictor. Journal of Educational Statistics, 12, 282–293CrossRefGoogle Scholar
Mendoza, J., Stafford, K.L., Stauffer, J.M. (2000). Large-sample confidence intervals for validity and reliability coefficients. Psychological Methods, 5(3), 356–369CrossRefGoogle ScholarPubMed
Muthen, B., Hsu, J. (1993). Selection and predictive validity with latent variable structures. British Journal of Mathematical and Statistical Psychology, 46, 255–271CrossRefGoogle Scholar
Muthen, B., Kaplan, D., Hollis, M. (1987). On structural equation modeling with data that are not missing completely at random. Psychometrika, 52, 431–462CrossRefGoogle Scholar
Raju, N.S., Brand, P.A. (2003). Determining the significance of correlations corrected for unreliability and range restriction. Applied Psychological Measurement, 27(1), 52–71CrossRefGoogle Scholar
Raju, N.S., Lezotte, D.V., Fearing, B.K., Oshima, T. (2006). A note on correlations corrected for unreliability and range restriction. Applied Psychological Measurement, 30(2), 145–149CrossRefGoogle Scholar
Robie, C., Ryan, A. (1999). Effect of nonlinearity and heteroscedasticity on the validity of conscientiousness in predicting overall job performance. International Journal of Selection and Assessment, 7, 157–169CrossRefGoogle Scholar
Sackett, P., Yang, H. (2000). Correction for range restriction: An expanded typology. Journal of Applied Psychology, 85, 112–118CrossRefGoogle ScholarPubMed
Stauffer, J., Mendoza, J. (2001). The proper sequence for correcting correlation coefficients for range restriction and unreliability. Psychometrika, 66, 63–68CrossRefGoogle Scholar
Yang, H., Sackett, P., Nho, Y. (2004). Developing a procedure to correct for range restriction that involves both institutional selection and applicants’ rejection of job offers. Organizational Research Methods, 7, 442–455CrossRefGoogle Scholar
Supplementary material: File

Culpepper supplementary material

Culpepper supplementary material
Download Culpepper supplementary material(File)
File 2.5 KB