Correcting Misconceptions About Gamification of Assessment: More Than SJTs and Badges

Describing the current state of gamification, Chamorro-Premuzic, Winsborough, Sherman, and Hogan (2016) provide a troubling contradiction: They offer examples of a broad spectrum of gamification interventions, but they then summarize the entirety of gamification as “the digital equivalent of situational judgment tests.” This mischaracterization grossly oversimplifies a rapidly growing area of research and practice both within and outside of industrial–organizational (I-O) psychology. We agree that situational judgment tests (SJTs) can be considered a type of gamified assessment, and gamification provides a toolkit to make SJTs even more gameful. However, the term gamification refers to a much broader and potentially more impactful set of tools than just SJTs, which are incremental, versatile, and especially valuable to practitioners in an era moving toward business-to-consumer (B2C) assessment models. In this commentary, we contend that gamification is commonly misunderstood and misapplied by I-O psychologists, and our goals are to remedy such misconceptions and to provide a research agenda designed to improve both the science and the practice surrounding gamification of human resource processes.

Gamification is a complex concept, which likely leads to its misunderstanding among many I-O psychologists. Outside of I-O, gamification is often misused as an umbrella term encompassing anything game related in a nongame context (Walz & Deterding, 2015). This impression includes a range of products including educational video games (i.e., serious games), customer loyalty programs awarding points for purchases, and interactive website designs. In contrast, gamification researchers generally define gamification as the use of game elements in nongame contexts (Deterding, Dixon, Khaled, & Nacke, 2011). Gamification and games are two tools used under the broader heading of gameful design (Walz & Deterding, 2015), itself an application of game thinking (Armstrong, Landers, & Collmus, 2015). In the more familiar context of the predictor versus method debate (Arthur & Villado, 2008), gamification is a meta-method; it includes a family of techniques inspired by research in game design used to improve the effectiveness of existing methods. Meta-methods like gamification are much broader in application domain than assessment or even I-O psychology—they are general-purpose toolkits.

Within a testing context, gamified (or game-like) assessments can be further defined by the differences between them and assessment games (Popp, 2014). An assessment game can be presented as a stand-alone experience, and assessments of constructs are made via gameplay behaviors. For example, Chamorro-Premuzic and colleagues briefly discussed Knack, a company that measures constructs from games played on mobile phones. While users play these games, Knack captures data from behaviors displayed within the game to determine standings on constructs. A gamified assessment is not a stand-alone game, but it is instead an existing form of assessment that has been enhanced with the addition of game elements. As Chamorro-Premuzic and colleagues correctly point out, SJTs can be enhanced with game elements such as graphics, storylines, feedback mechanisms, or others (Armstrong et al., 2015). However, any traditional assessment method may potentially be gamified. For example, two of the present authors have investigated the gamification of personality surveys by adding narrative elements (Collmus & Landers, 2015). The content from items assessing conscientiousness and openness was rewritten to form a progressive short story consisting of first-person scenarios, to which the participant responded with a set of behavioral items mapped onto a Likert scale. Another of the present authors has gamified various computer-based simulation-type exercises through the inclusion of animation, sound effects, instantaneous feedback, varying levels of difficulty, progress bars, and narrative context in order to make a more immersive and engaging assessment experience (Ferrell, Carpenter, Vaughn, Dudley, & Goodman, 2015). These examples demonstrate how different types of assessments may be gamified, as well as the incremental nature of gamification.

Another area where the focal article misses a major benefit of gamification pertains to implementation cost, which is generally low due to its incremental and versatile nature despite potentially high returns. First, gamification is incremental in that it can be as simple as the addition of individual game elements, such as competition or a narrative structure, and thus inexpensive relative to many other common assessment modifications. There is also no requirement for gamification to be as complex and in-depth as a video game. Many game elements can be added to an existing assessment with little cost, such as a simple progress bar to provide feedback to the test-taker on how many more items remain in the assessment (Landers, Bauer, Callan, & Armstrong, 2015; Yan, Conrad, Tourangeau, & Couper, 2011). Second, gamification is versatile in that many different combinations of elements might be added to an assessment in order to make it more engaging depending on assessment goals and budget. For example, the use of a leaderboard, a common gamification technique, encompasses the use of goal-driven game elements, a challenge/conflict element, an interpersonal interaction element among others on the leaderboard, and an element of feedback in the form of points or ranking. Assessment designers might add specific game elements to an assessment to maximize outcomes like test motivation while maintaining or improving predictive validity (Yan et al., 2011). Incrementally adding a few targeted game elements to an existing practice in human resources is likely to produce a greater return on investment for a lone organization than producing an entire digital game for the same purpose (Landers, 2014), which requires the development of both a game and a gaming platform. Even the development of new SJTs is generally quite costly (Lievens, Peeters, & Schollaert, 2008) in comparison with many forms of assessment gamification.

Despite the incremental and versatile nature of gamification, I-Os should be wary of the technological requirements associated with some forms of gamification. In its most inexpensive form, gamification is implemented within preexisting assessment technology. For example, narrative elements might be added to an online survey to make it more engaging (Collmus & Landers, 2015). This type of gamification only requires generating additional text to an already text-based survey. Alternatively, an online survey could utilize more game elements and more technology for a more engaging experience; for example, Downs-Le Guin, Baker, Mechling, and Ruylea (2012) incorporated avatars, narrative, and digital rewards within an online survey for the same general purpose. This type of gamification requires either an alternative gamified assessment platform or programming expertise in order to incorporate these game elements. The technological requirements needed for gamification to enhance assessments thus vary depending on the context of the assessment and desired outcomes, and some elements are more expensive than others. Further research is needed to determine whether particular elements or their combination necessarily improves outcomes of interest over others.

In terms of broader benefits, we agree with Chamorro-Premuzic and colleagues that the assessment landscape continues to shift from a business-to-business (B2B) to a B2C focus, but we differ in the extent to which we feel gamification helps accommodate this shift. The focal authors predict a future of firms searching for various talent badges among the applicant pool. This may gamify the talent search from the perspective of the firm, but we believe there is more potential in gamifying the assessment process from the perspective of the applicant. Some of the most critical outcomes of interest in gamified assessment research are the reactions of applicants who complete these new forms of assessments (Collmus & Landers, 2015). As the Internet continues to evolve, applicants increasingly expect assessments that are interesting, engaging, and delivered and evaluated instantly. As such, researchers and practitioners should focus on how gamification can impact test-taking motivation, reduce actual or perceived assessment length, and/or improve the validity of assessments. For example, consider goal-setting theory as a framework to understand why applicants might want to obtain badges for completing assessments. For an applicant to be motivated to pursue that goal, the badge must be difficult to obtain yet still realistically attainable, while being relevant to and valued by the applicant to ensure commitment to the goal (Landers, Bauer, & Callan, in press; Locke & Latham, 2013). Applicants may be more committed, and therefore more motivated, to earn a badge by taking an assessment if the badge will be displayed to the applicant's peers on social media as a sign of status or reputation (Antin & Churchill, 2011; Collmus, Armstrong, & Landers, 2016). By focusing on the impact of gamification on applicant reactions such as test-taking motivation, I-O can thus better accommodate the shift to B2C. Further, we predict this shift to be relatively long term.

Thus, gamification is commonly misapplied and misunderstood by I-O psychologists in assessment contexts; we contend this is likely due to the lack of theoretical guidelines and empirical evidence in the I-O literature despite such recommendations beginning to appear in the broader interdisciplinary games literature. In order to remedy this issue, an I-O literature on gamification in assessment must be established. To begin this effort, we provide recommendations here for a research agenda. We have contributed to the interdisciplinary literature ourselves by providing theoretical guidance for designing gamification and best practices for implementing gamification in a human resource management setting (Armstrong et al., 2015; Ferrell et al., 2015). Beyond providing a theoretical context for gamification in assessment, the next step is to establish a taxonomy of gamification elements for use in assessment contexts along with the likely effects of elements on key criteria. This requires identifying and categorizing the different ways game elements can theoretically impact different outcomes relevant to assessment. An example of a taxonomy of game elements for affecting learning outcomes is provided by Bedwell, Pavlas, Heyne, Lazzara, and Salas (2012), themselves a research team including several I-O psychologists. The Bedwell taxonomy is not inclusive of all possible game elements but is exhaustive for all game elements that learning game designers find likely to impact learning outcomes. The assessment research literature must take a similar approach to identify which elements impact assessment effectiveness, link these elements to existing theory when possible, and develop new theory where justified. For example, Landers et al. (in press) found that the presence of a leaderboard can increase task performance similarly to that of difficult or impossible goals in the context of goal-setting interventions, and this effect was moderated by goal commitment such that higher commitment was associated with higher performance. However, scoring at the bottom of a leaderboard is not a very fun position to be in, and within an assessment context, a leaderboard might disengage some people from an assessment, decreasing their test-taking motivation and their likelihood of completing the assessment (Ferrell et al., 2015). If this were to occur in a high-stakes assessment context, the hiring organization might end up with a smaller talent pool (i.e., losing talented individuals because of the gamified assessment). Future research on leaderboards and related elements in assessment needs to determine in which situations competitive elements are appropriate for implementation. Once a taxonomy of assessment-relevant game elements is available, we recommend I-O psychologists develop a theory or theories of gamified assessment in order to systematically understand the processes involved. Theories of gamified assessment should address which elements impact assessment and in what ways they impact assessment outcomes. An example of theory in gamification is presented by Landers (2014), who proposed that the use of game elements only affect learning outcomes by creating attitudinal and behavioral change. This idea of game elements affecting outcomes through mediators has been proposed elsewhere in the games and gamification literature (Garris, Ahlers, & Driskell, 2002; Hamari, Koivisto, & Sarsa, 2014), making the development of a similar model a promising place to start in the development of gamified assessment theory.

Gamification of assessment will not disappear from practice, just as people will not stop using the Internet, mobile devices, or video-based interviews. Business practices follow consumer demands, and I-O risks becoming irrelevant to business practices if it does not work at least somewhat in parallel. I-O can keep pace with business practices in gamification first by unifying in an understanding of the concept of gamification, which we have presented in this commentary. By first understanding gamification, I-Os can then apply theory to gamification in order to improve applicant and employee assessment in ways that matter to firms and test takers.