Adler et al. (2016) effectively discuss the shortcomings of performance ratings as well as performance management solutions. However, they leave out one important performance management solution from their review and discussion, the Productivity Measurement and Enhancement System (ProMES; Pritchard, 1990; Pritchard, Harrell, DiazGranados, & Guzman, 2008; Scaduto, Hunt, & Schmerling, 2015). We intend to respond to Adler et al. by explaining how ProMES could make up for some of the shortcomings described in performance ratings and performance management, as well as challenge the field to further consider this well-researched and established evidence based solution as a viable alternative.

Lievens and Motowidlo (2016) present a convincing case for why situational judgment tests (SJTs) should be developed specifically to measure general domain knowledge, but I have two concerns regarding the use of SJTs in psychological research and practical settings if the reconceptualization of SJTs offered by the authors is adopted to the exclusion of other current approaches. The first concerns abandoning SJTs as a helpful job analysis tool if we encourage intentional conflation of trait expression with job effectiveness in the development of SJT items. Second, diluting the situational component of SJTs may reduce their acceptance as selection tools by job applicants and practitioners in organizations.

This commentary elaborates on the position taken in the focal article, “Baltimore Is Burning” (Ruggs et al., 2016), that partnerships between industrial–organizational (I-O) psychologists and law enforcement agencies could offer valuable insight and practical tools that help to alleviate long-standing and ongoing conflict between police officers and communities of color. I fully support this stance as well as many of the recommendations proposed within the focal article. I-O psychologists indeed have knowledge and resources that could prove useful for supporting police reform efforts. However, although I-O psychologists have much to contribute to the conversation, a recent review by Hall, Hall, and Perry (2016) draws attention to several ways in which law enforcement as a context and police officers as a population are distinct from many of the workplace environments and civilian employee populations that have informed current I-O knowledge and best practices. As such, our traditional methods may be wrought with unique challenges when implemented in law enforcement contexts. If we fail to give serious consideration to this, the role of I-O psychologists in preventing future fires in Baltimore and throughout the United States may be accompanied by unintended consequences. As I-O psychologists prepare to help law enforcement extinguish the flames between police and communities of color, we must anticipate and prepare to safely combat these dangers so we do not add fuel to the fire or get burned in the process of rendering aid.

Through learning about and doing job analysis, industrial–organizational (I-O) psychologists likely already possess skills and knowledge relevant to doing and understanding qualitative research. We'll illustrate this by showing similarities between common job analysis practices and one particular qualitative research approach likely to be relevant to organizational research: grounded theory. Grounded theory was “discovered” in 1967 by Barney Glaser and Anselm Strauss (Glaser & Strauss, 1967). Though Glaser and Strauss later split in their methodologies (an occurrence not unlike the varied approaches to job analysis), the core idea of grounded theory is to develop a new theory of some process or phenomenon from the “ground” up. In the grounded theory approach, researchers typically collect mostly qualitative data—often including interviews (Creswell, 2007)—and simultaneously develop increasingly abstract codes, concepts, and categories from the data. In the final step of analysis, researchers develop a theory that subsumes all categories from the data. If researchers follow the Straussian tradition, categories can be fit into a theoretical framework that details a central phenomenon underlying the process of interest and the conditions that precede it, result from it, and shape the resulting categories (Creswell, 2007). We illustrate this framework in Figure 1. Grounded theory is particularly useful for developing an accurate understanding of many organizational processes and phenomena that I-O psychologists study.

Although the authors make strong arguments for both sides of the debate in “Getting Rid of Performance Ratings: Genius or Folly? A Debate,” we argue that performance appraisal reactions were largely overlooked beyond a few exceptions, where the authors either alluded to or explicitly mentioned reactions. For example, the authors explain that one reason organizations have eliminated the forced distribution approach is negative employee reactions. The authors also highlight the importance of managers using appropriate language when delivering performance appraisal ratings in order to improve employee reactions. Despite these exceptions, we believe it is necessary to call more attention to the critical role of ratee reactions in the performance appraisal process. Therefore, our commentary expands on the conversation sparked by Adler et al. (2016) by incorporating ratee reactions.

As discussed in the focal article, numerous research studies have supported the existence of automatic or implicit racial bias (Ruggs et al., 2016). In this commentary, we argue that examining implicit bias through the perspective of the motivation and opportunity as determinants (MODE) model (see Fazio & Olson, 2014, for a review) offers a framework for industrial–organizational (I-O) psychologists to design and implement strategies that reduce the number of violent interactions between police and communities. The MODE model has been applied to areas such as interpersonal relationships (McNulty, Olson, Meltzer, & Shaffer, 2013), effective treatment of mental disorders (Vasey, Harbaugh, Buffington, Jones, & Fazio, 2012), and crafting of media messages (Ewoldsen, Rhodes, & Fazio, 2015), as well as racial prejudice (Olson & Fazio, 2004). Below, we elaborate on how the I-O-related strategies and interventions described in the focal article can be captured by the components of the MODE model and highlight which interventions may be most efficacious in reducing discriminatory police officer behavior.

In the 25+ years that the practice of 360° Feedback has been formally labeled and implemented, it has undergone many changes. Some of these have been positive (evolution) in advancing theory, research, and practice, and others less so (devolution). In this article we offer a new definition of 360° Feedback, summarize its history, discuss significant research and practice trends, and offer suggestions for all user communities (i.e., researchers, practitioners, and end users in organizations) moving forward. Our purpose is to bring new structure, discussion, and some degree of closure to key open issues in this important and enduring area of practice.

Lievens and Motowidlo (2016) addressed three of the most important unanswered questions regarding situational judgment tests (SJTs): (a) Should we view them as tests that can assess relatively generic constructs that predict performance across settings, (b) what constructs can they assess, and (c) how should they be scored? They suggested fundamentally changing the SJT development process by targeting the specific constructs we measure, using scoring systems that address both the targeted traits and their situational effectiveness, examining construct validity, and evaluating the criterion-related validity of SJT traits (Lievens & Motowidlo, pp. 11–12). These recommendations are highly significant on both practical and conceptual grounds.

Situational judgment tests (SJTs) occasionally fail to predict job performance in criterion-related validation studies, often despite much effort to follow scholarly recipes for their development. This commentary provides some plausible explanations for why this may occur as well as some tips for SJT development. In most cases, we frame the issue from an implicit trait policy (ITP) perspective (Motowidlo, Hooper, & Jackson, 2006a, 2006b) and the measurement of general domain knowledge. In other instances, we believe that the issue does not have a direct tie to the ITP concept, but our experience suggests that the issue is of sufficient importance to include in this response. The first two issues involve challenges gathering validity evidence to support the use of SJTs, and the remaining issues deal more directly with SJT design considerations.

Recent commentary has suggested that performance management (PM) is fundamentally “broken,” with negative feelings from managers and employees toward the process at an all-time high (Pulakos, Hanson, Arad, & Moye, 2015; Pulakos & O'Leary, 2011). In response, some high-profile organizations have decided to eliminate performance ratings altogether as a solution to the growing disenchantment. Adler et al. (2016) offer arguments both in support of and against eliminating performance ratings in organizations. Although both sides of the debate in the focal article make some strong arguments both for and against utilizing performance ratings in organizations, we believe there continue to be misunderstandings, mischaracterizations, and misinformation with respect to some of the measurement issues in PM. We offer the following commentary not to persuade readers to adopt one particular side over another but as a call to critically reconsider and reevaluate some of the assumptions underlying measurement issues in PM and to dispel some of the pervasive beliefs throughout the performance rating literature.

Bracken, Rose, and Church (2016) call for less self- versus other research. They say that self-ratings versus non-self-ratings research (often referred to as self–other agreement research, or SOA) is not particularly helpful and that self-ratings’ primary purpose is to serve as a “reflection point for individuals receiving feedback” (Bracken, Rose, & Church, p. 785). It is true that SOA research has played a dominant role with 360° feedback for many years (for a review, see Fleenor, Smither, Atwater, Braddy, & Sturm, 2010), but the value we can draw from the self through 360° feedback remains terrain largely unexplored.

It is hard to argue with the central thesis of the focal article (Ruggs et al., 2016) that industrial–organizational (I-O) psychology has much to offer police departments in helping them meet their mission. As an example, the Los Angeles Police Department provides a clear statement that is representative of most police agencies:

Almost 20 years after McKinsey introduced the idea of a war for talent, technology is disrupting the talent identification industry. From smartphone profiling apps to workplace big data, the digital revolution has produced a wide range of new tools for making quick and cheap inferences about human potential and predicting future work performance. However, academic industrial–organizational (I-O) psychologists appear to be mostly spectators. Indeed, there is little scientific research on innovative assessment methods, leaving human resources (HR) practitioners with no credible evidence to evaluate the utility of such tools. To this end, this article provides an overview of new talent identification tools, using traditional workplace assessment methods as the organizing framework for classifying and evaluating new tools, which are largely technologically enhanced versions of traditional methods. We highlight some opportunities and challenges for I-O psychology practitioners interested in exploring and improving these innovations.

Lievens and Motowidlo (2016) present a case for situational judgment tests (SJTs) to be conceptualized as measures of general domain knowledge, which the authors define as knowledge of the effectiveness of general domains such as integrity, conscientiousness, and prosocial behaviors in different jobs. This argument comes from work rooted in the use of SJTs as measures of implicit trait policies (Motowidlo & Beier, 2010; Motowidlo, Hooper, & Jackson, 2006), measured with a format described as a “single response SJT” (Kell, Motowidlo, Martin, Stotts, & Moreno, 2014; Motowidlo, Crook, Kell, & Naemi, 2009). Given evidence that SJTs can be used as measures of general domain knowledge, the focal article concludes with a suggestion that general knowledge can be measured not only by traditional text-based or paper-and-pencil SJTs but also through varying alternate formats, including multimedia SJTs and interactive SJTs.

In this commentary, we build on Bracken, Rose, and Church's (2016) definition stating that 360° feedback should involve “the analysis of meaningful comparisons of rater perceptions across multiple ratees, between specific groups of raters” (p. 764). Bracken et al. expand on this component of the definition later by stressing that “the ability to conduct meaningful comparisons of rater perceptions both between (inter) and within (intra) groups is central and, indeed, unique to any true 360° feedback process” (p. 767; italicized in their focal article). Bracken et al. stress that “This element of our definition acknowledges that 360° feedback data represent rater perceptions that may contradict each other while each being true and valid observations” (p. 767).

Performance evaluation has long been a source of dissatisfaction for practitioners and a focus of research for scholars. The current call for the elimination of performance ratings is not new. This commentary considers the quality perspective as a historical context in which performance ratings were, at best, considered a misguided management tool. Although the current debate doesn't seem to be philosophically based, it may be useful to recognize that serious questions regarding performance ratings have come up before. Potential measurement problems with performance ratings are considered. It is concluded that performance ratings are not the major problem for performance management. Possible sources of problems with performance management are considered. Directions for improvement are discussed.

As organizations become decentralized and work becomes team based, organizations are adopting performance management practices that integrate employees’ performance information from multiple perspectives (e.g., 360-degree performance ratings). Both arguments for and against the use of performance ratings presented in the focal article focused on rater agreement (or lack thereof) as evidence supporting the position that multisource ratings are a useful (or not a useful) approach to performance appraisal. In the argument for the use of multisource ratings, Adler, Campion, and Grubb (Adler et al., 2016) point out that multisource ratings are advantageous because they lead to increased interrater reliability in the ratings. Although Adler and colleagues were not explicit about why this would be true, proponents of multisource ratings often cite the measurement theory assumption that increasing the number of raters will yield more valid and reliable scores to the extent that there is any correlation in the ratings (Shrout & Fleiss, 1979). In the argument against the use of multisource performance ratings, Colquitt, Murphy, and Ollander-Krane argued that because multisource ratings pool together ratings from raters who are systematically different in terms of their roles and perspectives about the target employee's performance, the increased number of raters is not expected to resolve the low level of interrater agreement that is typically observed in performance ratings (Viswesvaran, Ones, & Schmidt, 1996).

In this article, we demonstrate that samples in the industrial and organizational (I-O) psychology literature do not reflect the labor market, overrepresenting core, salaried, managerial, professional, and executive employees while underrepresenting wage earners, low- and medium-skill first-line personnel, and contract workers. We describe how overrepresenting managers, professionals, and executives causes research about these other workers to be suspect. We describe several ways that this underrepresentation reduces the utility of the I-O literature and provide specific examples. We discuss why the I-O literature underrepresents these workers, how it contributes to the academic–practitioner gap, and what researchers can do to remedy the issue.

Chamorro-Premuzic, Winsborough, Sherman, and Hogan (2016) end their focal article with a quote worth remembering from Immanuel Kant: “Theory without data is groundless, but data without theory is just uninterpretable.” We begin with a quote even better known to industrial–organizational (I-O) psychologists in part because it has served for over 65 years as a foundational principle of our field: “There is nothing as practical as a good theory” (Lewin, 1951, p. 169).

Although the authors of the focal article provide a comprehensive definition of 360° feedback, one exclusionary criterion results in an overly narrow definition of 360° feedback. Specifically, Point 3 in their definition described the criticality of strictly using quantitative methods in collecting 360° feedback. The authors provided a brief rationale by stating, “Data generated from truly qualitative interviews would not allow comparisons between rater groups on the same set of behaviors” (Bracken, Rose, & Church, 2016, p. 765). Although there is little doubt about the value in taking a quantitative approach for gathering 360° feedback, it is not clear why this has to be the sole approach. Below, I outline three issues with taking this constricted methodology. That is, first, excluding qualitative methods is not in line with the purpose of 360° feedback, which is directed at minimizing criterion deficiency. Second, qualitative methodologies (in conjunction with quantitative methodologies) are more equipped to provide and inspire a call to action (supporting the change component addressed by the authors). Finally, there are qualitative methods that allow for rigorous quantitative analysis and can provide an additional source of macro organizational-level data.