Hostname: page-component-cd9895bd7-jn8rn Total loading time: 0 Render date: 2024-12-28T20:22:09.962Z Has data issue: false hasContentIssue false
  • English
  • Français

The Relationship Between the Number of Raters and the Validity of Performance Ratings

Published online by Cambridge University Press:  04 July 2016

Matt C. Howard
Matt C. Howard*
Affiliation:
Mitchell College of Business, Department of Management, University of South Alabama, and Department of Psychology, Pennsylvania State University
*
Correspondence concerning this article should be addressed to Matt C. Howard, Mitchell College of Business, Department of Management, University of South Alabama, 5811 USA Drive South, Room 346, Mobile, AL 36688-0002. E-mail: mhoward@southalabama.edu
Get access Rights & Permissions [Opens in a new window]

Extract

In the focal article “Getting Rid of Performance Ratings: Genius or Folly? A Debate,” two groups of authors argued the merits of performance ratings (Adler et al., 2016). Despite varied views, both sides noted the importance of including multiple raters to obtain more accurate performance ratings. As the pro side noted, “if ratings can be pooled across many similarly situated raters, it should be possible to obtain quite reliable assessments” (Adler et al., p. 236). Even the con side noted, “In theory, it is possible to obtain ratings from multiple raters and pool them to eliminate some types of interrater agreement” (Adler et al., p. 225), although this side was certainly less optimistic about the merits of multiple raters. In the broader industrial–organizational psychology literature, authors have repeatedly heralded the benefits of adding additional raters for performance ratings, some even treating it as a panacea for inaccurate ratings. Although these authors extol the virtues of multiple raters, an important question is often omitted from relevant discussions of performance ratings: To what extent do additional raters actually improve performance ratings? Does adding an additional rater double the validity of performance ratings? Does an additional rater increase the validity of performance ratings by a constant value? Or is the answer something else altogether?

Type
Commentaries
Information
Industrial and Organizational Psychology , Volume 9 , Issue 2 , June 2016 , pp. 361 - 367
Copyright
Copyright © Society for Industrial and Organizational Psychology 2016 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Adler, S., Campion, M., Colquitt, A., Grubb, A., Murphy, K., Ollander-Krane, R., & Pulakos, E. D. (2016). Getting rid of performance ratings: Genius or folly? A debate. Industrial and Organizational Psychology: Perspectives on Science and Practice, 9 (2), 219–252.Google Scholar
Ghiselli, E. E. (1964). Theory of psychological measurement. New York, NY: McGraw-Hill.Google Scholar
Hoffman, B., Lance, C. E., Bynum, B., & Gentry, W. A. (2010). Rater source effects are alive and well after all. Personnel Psychology, 63, 119–151.Google Scholar
Hogarth, R. M. (1978). A note on aggregating opinions. Organizational Behavior and Human Performance, 21, 40–46.Google Scholar
Murphy, K. R., Cleveland, J. N., & Mohler, C. (2001). Reliability, validity, and meaningfulness of multisource ratings. In Bracken, D., Timmreck, C., & Church, A. (Eds.), Handbook of multisource feedback (pp. 130–148). San Francisco, CA: Jossey-Bass.Google Scholar
Ones, D. S., Viswesvaran, C., & Schmidt, F. L. (2008). No new terrain: Reliability and construct validity of job performance ratings. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 174–179.Google Scholar
Tsujimoto, R. N., Hamilton, M., & Berger, D. E. (1990). Averaging multiple judges to improve validity: Aid to planning cost-effective clinical research. Psychological Assessment: A Journal of Consulting and Clinical Psychology, 2 (4), 432–437.Google Scholar
Viswesvaran, C., Ones, D. S., & Schmidt, F. L. (1996). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81, 557–574.CrossRefGoogle Scholar