Hostname: page-component-78c5997874-94fs2 Total loading time: 0 Render date: 2024-11-13T04:42:53.710Z Has data issue: false hasContentIssue false
  • English
  • Français

Best Practice Recommendations for Situational Judgment Tests

Published online by Cambridge University Press:  10 September 2015

Stefanie Pollard  and
Helena D. Cooper-Thomas
Stefanie Pollard
Affiliation:
Talegent, Auckland, New Zealand
Helena D. Cooper-Thomas*
Affiliation:
School of Psychology, The University of Auckland, New Zealand
*
Address for correspondence: Helena D. Cooper-Thomas, School of Psychology, The University of Auckland, Private Bag 92019, Auckland 1142, New Zealand. Email: h.cooper-thomas@auckland.ac.nz
Get access

Abstract

Situational judgement tests (SJTs) are increasingly popular as part of selection procedures because of their strong psychometric properties. SJTs present test takers with job-related situations and a series of potential responses. The aim of this review article is to provide evidence-based recommendations for practitioners who are considering implementing SJTs for their clients, as well as offering an analysis of some key issues for academics. This research provides a review of the literature on SJTs to answer three questions: (1) What is the difference between framing questions either as what should you do versus what would you do? (2) What is the difference between multimedia-based and text-based SJTs? (3) What is the best process for SJT creation? Following this, recommendations are provided as follows: That responses are optimally framed in terms of what a respondent thinks s/he should do, not what s/he would do; that multimedia-based SJTs are used in preference to text-based SJTs; and a ‘best practice’ process for creating an SJT is outlined. Finally, the review includes a discussion of implications for creating tests that are inclusive of different ethnic groups, as well as ethical issues relating to the use of SJTs in selection.

Keywords

situational judgment testSJTselectionitem stemvalidity
Type
Articles
Information
The Australasian Journal of Organisational Psychology , Volume 8 , 2015 , e7
Copyright
Copyright © Australian Psychological Society Ltd 2015 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Anderson, N., Herriot, P., & Hodgkinson, G.P. (2001). The practitioner‐researcher divide in Industrial, Work and Organizational (IWO) psychology: Where are we now, and where do we go from here? Journal of Occupational and Organizational Psychology, 74, 391–411. doi:10.1348/096317901167451CrossRefGoogle Scholar
Anderson, N., Salgado, J.F., & Hülsheger, U.R. (2010). Applicant reactions in selection: Comprehensive meta-analysis into reaction generalization versus situational specificity. International Journal of Selection and Assessment, 18, 291–304. doi:10.1111/j.1468-2389.2010.00512.xCrossRefGoogle Scholar
Arthur, W. Jr, Glaze, R.M., Jarrett, S.M., White, C.D., Schurig, I., & Taylor, J.E. (2014). Comparative evaluation of three situational judgment test response formats in terms of construct-related validity, subgroup differences, and susceptibility to response distortion. Journal of Applied Psychology, 99, 535–545. doi:10.1037/a0035788CrossRefGoogle ScholarPubMed
Avery, D.R. (2003). Reactions to diversity in recruitment advertising — Are differences black and white? Journal of Applied Psychology, 88, 672–679. doi:10.1037/0021-9010.88.4.672CrossRefGoogle ScholarPubMed
Bauer, T.N., & Truxillo, D.M. (2006). Applicant reactions to situational judgment tests: Research and related practical issues. In Weekley, J.A. & Ployhart, R.E. (Eds.), Situational judgment tests: Theory, measurement, and application (pp. 233–252). Mahwah, NJ: Erlbaum.Google Scholar
Briner, R., & Rousseau, D.M. (2011). Evidence-based I/O Psychology: Not there yet. Industrial and Organizational Psychology, 4, 3–22. doi:1754-9426/11CrossRefGoogle Scholar
*Brooks, M.E., & Highhouse, S. (2006). Can good judgment be measured? In Weekley, J.A. & Ployhart, R.E. (Eds.), Situational judgment tests: Theory, measurement, and application (pp. 39–57). Mahwah, NJ: Erlbaum.Google Scholar
Campion, M.C., Ployhart, R.E., & MacKenzie, W.I. Jr. (2014). The state of research on situational judgment tests: A content analysis and directions for future research. Human Performance, 27, 283–310. doi:10.1080/08959285.2014.929693CrossRefGoogle Scholar
Chan, D. (2006). Interactive effects of situational judgment effectiveness and proactive personality on work perceptions and work outcomes. Journal of Applied Psychology, 91, 475–481. doi:10.1037/0021-9010.91.2.475CrossRefGoogle ScholarPubMed
*Chan, D., & Schmitt, N. (1997). Video-based versus paper-and-pencil method of assessment in situational judgment tests: Subgroup differences in test performance and face validity perceptions. Journal of Applied Psychology, 82, 143–159. doi:10.1037/0021-9010.82.1.143Google ScholarPubMed
*Christian, M.S., Edwards, B.D., & Bradley, J.C. (2010). Situational judgment tests: Constructs assessed and a meta-analysis of their criterion-related validities. Personnel Psychology, 63, 83–117. doi:10.1111/j.1744-6570.2009.01163.xCrossRefGoogle Scholar
DuVernet, A.M., & Popp, E. (2014). Gamification of workplace practices. The Industrial-Organizational Psychologist, 52, 39–42.Google Scholar
Grant, A.M., & Pollock, T.G. (2011). Publishing in AMJ — Part 3: Setting the hook. Academy of Management Journal, 54 (5), 873–879. doi: 10.5465/amj.2011.4000CrossRefGoogle Scholar
Griffin, D.W., Dunning, D., & Ross, L. (1990). The role of construal processes in overconfident predictions about the self and others. Journal of Personality and Social Psychology, 59, 1128–1139. doi:10.1037/0022-3514.59.6.1128CrossRefGoogle ScholarPubMed
Hofstede, G. (1999). Problems remain, but theories will change: The universal and the specific in 21st-century global management. Organizational Dynamics, 28, 34–44. doi:10.1016/S0090-2616(00)80005-4CrossRefGoogle Scholar
*Lievens, F., Peeters, H., & Schollaert, E. (2008). Situational judgment tests: A review of recent research. Personnel Review, 37, 426–441. doi:10.1108/00483480810877598CrossRefGoogle Scholar
*Lievens, F., & Sackett, P.R. (2006). Video-based versus written situational judgment tests: A comparison in terms of predictive validity. Journal of Applied Psychology, 91, 1181–1188. doi:10.1037/0021-9010.91.5.1181CrossRefGoogle ScholarPubMed
*Lievens, F., Sackett, P.R., & Buyse, T. (2009). The effects of response instructions on situational judgment test performance and validity in a high-stakes context. Journal of Applied Psychology, 94, 1095–1101. doi:10.1037/a0014628CrossRefGoogle Scholar
Loewenstein, G. (1996). Out of control: Visceral influences on behavior. Organizational Behavior and Human Decision Processes, 116, 75–98. doi:10.1006/obhd.1996.0028Google Scholar
Loewenstein, G., & Schkade, D. (1999). Wouldn't it be nice? Predicting future feelings. In Kahneman, D., Diener, E., & Schwarz, N. (Eds.), Well-being: The foundations of hedonic psychology (pp. 85–105). New York: Sage.Google Scholar
MacCann, C., & Roberts, R.D. (2008). New paradigms for assessing emotional intelligence: Theory and data. Emotion, 8, 540–551. doi:10.1037/a0012746CrossRefGoogle ScholarPubMed
*McDaniel, M.A., Hartman, N.S., Whetzel, D.L., & Grubb, W.L. (2007). Situational judgment tests, response instructions, and validity: A meta-analysis. Personnel Psychology, 60, 63–91. doi:10.1111/j.1744-6570.2007.00065CrossRefGoogle Scholar
McDaniel, M.A., Morgeson, F.P., Finnegan, E.B., & Campion, M.A. (2001). Use of situational judgment tests to predict job performance: A clarification of the literature. Journal of Applied Psychology, 86, 730–740. doi:10.1037//0021-9010.86.4.730CrossRefGoogle ScholarPubMed
McDaniel, M.A., & Nguyen, N.T. (2001). Situational judgment tests: A review of practice and constructs assessed. International Journal of Selection and Assessment, 9, 103–113. doi:10.1111/1468-2389.0016Google Scholar
Messick, S. (1995). Validity of psychological assessment: validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741–749. doi:10.1037/0003- 066X.50.9.741.CrossRefGoogle Scholar
Motowidlo, S.J., Crook, A.E., Kell, H.J., & Naemi, B. (2009). Measuring procedural knowledge more simply with a single-response situational judgment test. Journal of Business and Psychology, 24, 281–288. doi:10.1007/s10869-009-9106-4CrossRefGoogle Scholar
*Nguyen, N.T., Biderman, M.D., & McDaniel, M.A. (2005). Effects of response instructions on faking a situational judgment test. International Journal of Selection and Assessment, 13, 250–260. doi:10.1111/j.1468-2389.2005.00322.xGoogle Scholar
*Nguyen, N.T., & McDaniel, M.A. (2003). Response instructions and racial differences in a situational judgment test. Applied H.R.M. Research, 8, 33–44.Google Scholar
Ogden, J.A., & MacFarlane-Nathan, G. (1997). Cultural bias in the neuropsychological assessment of young Māori men. New Zealand Journal of Psychology, 26, 2–12.Google Scholar
*Olson-Buchanan, J.B., & Drasgow, F. (2006). Multimedia situational judgment tests: The medium creates the message. In Weekley, J.A. & Ployhart, R.E. (Eds.), Situational judgment tests: Theory, measurement, and application (pp. 253–278). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
*Ployhart, R.E., & Ehrhart, M.G. (2003). Be careful what you ask for: Effects of response instructions on the construct validity and reliability of situational judgment tests. International Journal of Selection and Assessment, 11, 1–16. doi:10.1111/1468-2389.00222CrossRefGoogle Scholar
Privacy Act of 1993. §§ 1–12. Retrieved from http://www.legislation.govt.nz/act/public/1993/0028/latest/DLM296639.htmlGoogle Scholar
*Richman-Hirsh, W.L., Olson-Buchanan, J.B., & Drasgow, F. (2000). Examining the impact of administration medium on examinee perceptions and attitudes. Journal of Applied Psychology, 85, 880–887. doi:10.1037/0021-9010.85.6.880CrossRefGoogle Scholar
Smither, J.W., Reilly, R.R., Millsap, R.E., Pearlman, K., & Stoffey, R.W. (1993). Applicant reactions to selection procedures. Personnel Psychology, 46, 49–76. doi:10.1111/j.1744-6570.1993.tb00867.xCrossRefGoogle Scholar
St-Sauveur, C., Girouard, S., & Goyette, V. (2014). Use of situational judgment tests in personnel selection: Are the different methods for scoring the response options equivalent? International Journal of Selection and Assessment, 22, 225–239. doi:10.1111/ijsa.12072CrossRefGoogle Scholar
*Weekley, J.A., & Jones, C. (1997). Video-based situational testing. Personnel Psychology, 50, 25–49. doi:10.1111/j.1744-6570.1997.tb00899.xCrossRefGoogle Scholar
*Weekley, J.A., Ployhart, R.E., & Holtz, B.C. (2006). On the development of situational judgment tests: Issues in item development, scaling, and scoring. In Weekley, J.A. & Ployhart, R.E. (Eds.), Situational judgment tests: Theory, measurement, and application (pp. 157–182). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar