In spite of numerous attempts over decades to improve performance management (PM) systems, PM is viewed as more broken than ever, with managers and employees seeing it as a burdensome activity that is of little value. Yet, the behaviors that PM is meant to achieve are in fact important drivers of engagement and performance. So where is the disconnect? The problem is that formal PM systems have reduced PM to intermittent steps and processes that are disconnected from day-to-day work and behaviors that actually drive performance: communicating ongoing expectations, providing informal feedback in real time, and developing employees through experience. To deliver on its promise, PM needs to shift from focusing on the formal system to focusing on the PM behaviors that matter every day. We describe a 5-step PM reform process that helps organizations achieve this change and that shows promise for increasing satisfaction and positive outcomes from PM processes. Central to the intervention is that organizational members need to intentionally practice and solidify effective PM behavior through a structured, on-the-job, experiential learning intervention that yields meaningful behavior change. The change-management and training interventions discussed here provide a model for organizational culture and behavior change efforts beyond PM.

The notion of a “Millennial” generation, much like a “Generation X” or the “Baby Boom” generation, with a strong coherence in terms of values and norms that differ from previous cohorts, has been of dependable interest in the popular press. However, given what we know regarding the proportion of trait expression due to sources largely immune to cohort effects (e.g., large genetic contributions), how difficult it is for us to systematically influence their expression (e.g., small long-term parental effects), and the massive variation within groups, the meta-analytic work of Costanza, Fraser, Badger, Severt, and Gade (2012) underscores what should already be known from first principles; generation or cohorts are inevitably a poor predictor of anything. The literature on ingroup/outgroup bias (Hogg & Abrams, 1990), stereotype formation (Mackie, Hamilton, Susskind, & Rosselli, 1996), and reconstructive memory issues (Schacter, 1999) provides ample underlying evidence for how these generational overgeneralizations form.

Here, we expand on Landers and Behrend's (2015) discussion of the external validity of convenience samples. In particular, we note that their focal article failed to mention one important limitation of multi-organization convenience samples (e.g., MTurk samples, student samples): Multi-organization convenience samples tend to confound levels of analysis, which affects the external validity of these samples. Specifically, between-organizations phenomena (i.e., organization-level) and within-organizations phenomena (i.e., individual-level) are distinct and separable (Ostroff, 1993; Robinson, 1950). Unfortunately, multi-organization samples such as those found in MTurk or MBA student samples can confound these two sets of phenomena. The current commentary uses a levels-of-analysis framework to expand on Landers and Behrend's discussion of what external validity is, and then the commentary illustrates how the diversity of convenience samples can actually harm external validity under some common circumstances.

In this response to Guzzo, Fink, King, Tonidandel, and Landis (2015), we suggest industrial–organizational (I-O) psychologists join business analysts, data scientists, statisticians, mathematicians, and economists in creating the vanguard of expertise as we acclimate to the reality of analytics in the world of big data. We enthusiastically accept their invitation to share our perspective that extends the discussion in three key areas of the focal article—that is, big data sources, logistic and analytic challenges, and data privacy and informed consent on a global scale. In the subsequent sections, we share our thoughts on these critical elements for advancing I-O psychology's role in leveraging and adding value from big data.

Costanza and Finkelstein (2015) make a number of important observations in their exploration of how generation-based differences are understood by academics and practitioners. The absence of a unifying theory, the lack of a clear pattern of findings, and the conceptual ambiguity about generational membership have all limited the utility of generational membership to researchers, while the practice of describing large groups of individuals in generational terms has become enormously popular outside of academia. Certainly, there are several topics that are popular among the public and widely used by businesses despite being viewed negatively by academics; however, there is some danger in the widespread perception that group differences between individuals of different ages can be explained by generational membership. Although it might be tempting to consider these generational stereotypes as simply innocuous misperceptions supervisors may hold, they may, in fact, be quite harmful. The purpose of this article is to further expand on the potential for these stereotypes to be harmful for older and younger employees.

In the focal article, Pulakos, Mueller Hanson, Arad, and Moye (2015) have noted that performance management (PM) too often consists of intermittent steps (e.g., end-of-year performance review ratings and meetings) that are not connected to day-to-day work. With regard to performance ratings, Pulakos et al. (2015) have pointed out that it can be “demotivating and disengaging for employees to have their performance boiled down to a single number, with which they are then labeled, unless it is the highest rating or ranking that is available” (p. xx). The authors have also noted that managers sometimes retrofit their ratings to justify the pay increases that managers want to give employees (sometimes for reasons unrelated to the employee's level of performance). Pulakos et al. have also offered a case study of one organization, Cargill, which has abandoned ratings. Although Pulakos and colleagues have pointed out that not all organizations should eliminate ratings, the authors have argued that when there are small differences in pay increases across employees, ratings can be removed with relative ease and little consequence. Pulakos and colleagues have therefore suggested that organizations should consider the impact that ratings have on decision making (and have suggested that this impact is often smaller than one might think).

The focal article (Guzzo, Fink, King, Tonidandel, & Landis, 2015) sought to “raise awareness and provide direction with regard to issues and complications uniquely associated with the advent of big data,” (p. 492), and we commend their success in offering Society for Industrial and Organizational Psychology (SIOP) members a solid foundation and resources on which to draw. Our aim here is to extend their position, particularly to drive the conversation toward concrete recommendations for how industrial and organizational psychologists (I-Os) working in industry can apply the principles set forth in the focal article in our day-to-day work, specifically around the issue of avoiding ethical missteps in this new landscape.

We agree with Landers and Behrend's (2015) proposition that Amazon's Mechanical Turk (MTurk) may provide great opportunities for organizational research samples. However, some groups are characteristically difficult to recruit because they are stigmatized or socially disenfranchised (Birman, 2005; Miller, Forte, Wilson, & Greene, 2006; Sullivan & Cain, 2004; see Campbell, Adams, & Patterson, 2008, for a review). These groups may include individuals who have not previously been the focus of much organizational research, such as those of low socioeconomic status; individuals with disabilities; lesbian, gay, bisexual, or transgender (LGBT) individuals; or victims of workplace harassment. As Landers and Behrend (2015) point out, there is an overrepresentation of research using “Western, educated, industrialized, rich, and democratic” participants. It is important to extend research beyond these samples to examine workplace phenomena that are specific to special populations. We contribute to this argument by noting the particular usefulness that MTurk can provide for sampling from hard-to-reach populations, which we characterize as groups that are in the numerical minority in terms of nationwide representation. To clarify, we focus our discussion on populations that are traditionally hard to reach in the context of contemporary organizational research within the United States.

The scientist–practitioner model of training in industrial and organizational psychology provides the foundation for the education of industrial and organizational psychologists across the world. This approach is important because, as industrial and organizational psychologists, we are responsible for both the creation and discovery of knowledge and the use or application of that knowledge. In multiple articles recently published in this journal, Pulakos and her colleagues (Pulakos, Mueller Hanson, Arad, & Moye, 2015; Pulakos & O’Leary, 2011) have argued that performance management (PM), as applied and implemented in organizations, is broken. This is not a unique take on the state of PM in organizations, as others have been arguing for many years that PM is no longer working in organizations the way that we would like it to work (Banks & Murphy, 1985; Bretz, Milkovich, & Read, 1992). Further, for many years and in many Society for Industrial and Organizational Psychology conference panels and debates in the literature, we have been inundated with discussions and conversations around the science–practice gap and around the gap being especially evident in PM.

Landers and Behrend (2015) call for editors and reviewers to resist using heuristics when evaluating samples in research as well as for researchers to cautiously consider choosing the samples appropriate for their research questions. Whereas we fully agree with the former conclusion, we believe the latter can be extended even further to encourage researchers to embrace the strengths of their samples for understanding their research rather than simply defending their samples. We believe that samples are not inherently better or worse but rather better suited for different research objectives. In this commentary, we identify three continua on which research goals can differ to demonstrate that all samples can inform science. Depending on the position of one's research on these continua, different samples exhibit different strengths; the continua described below can be used to anchor one's sample to demonstrate how it can benefit, rather than limit, research conclusions. As discussed in the focal article, researchers will often apologize for their convenience samples as one of a litany of limitations; we hope that researchers will move sampling issues out of the limitations section and into the main discussion.

The “big data” movement is forcing many fields to establish best practices for the collection, analysis, and application of big data, and the field of industrial–organizational (I-O) psychology is not exempt from this disruptive influence. Over the last several years, I-O scientists and practitioners have grappled with questions related to the definition, application, and interpretation of big data (e.g., Doverspike, 2013; Maurath, 2014; Morrison & Abraham, 2015; Poeppelman, Blacksmith, & Yang, 2013). The focal article by Guzzo, Fink, King, Tonidandel, and Landis (2015) continues this discussion and represents one of the first attempts to establish a formal set of recommendations for working with big data in ways that are consistent with I-O psychology's professional guidelines and ethics requirements.

Although we agree with Costanza and Finkelstein (2015) that current research has failed to find consistent evidence of actual generational differences and that this research is limited methodologically, we suggest that at least some of these limitations could and should be addressed by future research before any firm conclusions are drawn. Further, the authors make an important but empirically untested assumption that generational stereotypes exist. We discuss why it is important to rigorously test this assumption. Finally, the authors assert that generational stereotypes are being “sold” as business strategy to organizations and managers and that that they should be neither sold nor used in the workplace. However, the authors’ recommendation is based on the premise that individuals acquire stereotypes from others and that they can easily suppress the use of their stereotypes. This advice runs counter to what research suggests about stereotypic processes. We more fully discuss each of these three points next.

As a workforce evolves, it is important for employers to consider how trends and changes in the workforce have implications for organizational processes such as recruiting, selecting, training, and rewarding a cohort of workers (Costanza & Finkelstein, 2015). This article examines the practical aspects of workforce changes on organizations’ recruitment efforts. Particularly in the face of recruiting shortages, it is important to understand how to best to develop a large and qualified pool of applicants.

The purpose of this commentary is to complement the lead article by Pulakos, Mueller Hanson, Arad, and Moye (2015) by proposing the incorporation of 360 feedback as another means of improving performance management (PM). A 360 feedback refers to the practice of soliciting anonymous ratings and narrative comments on the job performance and other behaviors of the focal employee from a wide range of others who have worked with the employee. These sources include peers, subordinates, other managers, and often customers, as well as the immediate supervisor and self-ratings. There is an extensive body of research literature on 360s. Almost 300 articles and books have accumulated on the topic over the past 30 years (Campion, Campion, & Campion, 2014), resulting in substantial knowledge about the usefulness and effectiveness of 360s. Although the primary purpose of 360s has been employee development, 360s are being increasingly used within PM systems.

In recent years the concept of mindfulness has become increasingly popular, and with good reason. A growing body of research indicates that mindfulness provides a number of physical, psychological, and even performance benefits. As a result, some organizations have started offering mindfulness programs to their employees. But despite growing interest, mindfulness has received little attention from the industrial–organizational community. In this article, we provide an overview of what mindfulness is, where the concept came from, how it has been utilized and studied to date, and what its application in the work setting is. We also propose new directions for researchers and practitioners.

We agree with the authors of the focal article that too little attention is paid to sampling in industrial–organizational (I-O) psychology research. Upon reflection and in response to the focal article by Landers and Behrend (2015), we answer three primary questions: (a) What is it about our training, science, and practice as I-O psychologists that has led to less focus on sampling issues? (b) Does it matter? (c) If so, then what should we do about it?

For industrial and organizational psychologists who are unfamiliar with the mindfulness literature, Hyland, Lee, and Mills (2015) nicely introduce the concept by highlighting key findings from prior studies. Although their review focuses on the many benefits of mindfulness, we believe that mindfulness research should address certain questions that will help us understand whether mindfulness interventions result in a cost-effective positive return on investment. In alignment with the perspective of evidence-based practice (Briner & Rousseau, 2011; Pfeffer & Sutton, 2006), we call for a holistic evaluation of mindfulness, including a consideration of when or how unintended side effects emerge. Importantly, we discuss the potential mechanisms by which mindfulness generates valued outcomes (e.g., performance and collective psychological climate) and the need for more sophisticated research to isolate these causal effects. We also consider how the judicious use of utility analytics (e.g., cost effectiveness and return on investment) might help demonstrate the value of mindfulness interventions while also acknowledging questions of causality that must be addressed for such value to be experienced. We close by clarifying that we have the intention of promoting research to further evidence-based practices. There are organizations that have already begun providing mindfulness meditation interventions, and it is our hope that our commentary will help practitioners in these settings to consider the evidence suggesting that there may be unknown nuances regarding mindfulness practice. Ultimately, we believe that mindfulness is an important burgeoning area of research deserving of more scholarly attention.

In this article, we highlight why and how industrial and organizational psychologists can take advantage of research on 21st century skills and their assessment. We present vital theoretical perspectives, a suitable framework for assessment, and exemplary instruments with a focus on advances in the assessment of human capital. Specifically, complex problem solving (CPS) and collaborative problem solving (ColPS) are two transversal skills (i.e., skills that span multiple domains) that are generally considered critical in the 21st century workplace. The assessment of these skills in education has linked fundamental research with practical applicability and has provided a useful template for workplace assessment. Both CPS and ColPS capture the interaction of individuals with problems that require the active acquisition and application of knowledge in individual or group settings. To ignite a discussion in industrial and organizational psychology, we discuss advances in the assessment of CPS and ColPS and propose ways to move beyond the current state of the art in assessing job-related skills.

Pulakos, Mueller Hanson, Arad, and Moye (2015) have stated that performance management (PM) best practices have resulted in the dissatisfaction of employees and managers with PM reviews and systems. The authors have suggested that the focus should be on driving critical PM behaviors that have been shown to increase engagement and performance. Corporate Leadership Council (2004) research demonstrated that increased engagement takes place when PM behaviors such as setting clear expectations, providing regular feedback, and helping employees develop and succeed occur. In addition, effective PM happens when employees are contributing to the process as well.

Costanza and Finkelstein (2015) are correct to highlight the dangers of using generationally based stereotypes in organizations. Although popular, these stereotypes are related to a stigmatization based on group membership that can be pernicious and discriminatory. Costanza and Finkelstein are also correct in their assessment of the state of the literature on generational effects: theory and research is woefully lacking. Indeed, a recent review of research on generations at work characterized this research as descriptive and neither theoretical nor empirical (Lyons & Kuron, 2014). Yet, as pointed out by Costanza and Finkelstein, the idea of a generational identity is salient and even appealing to many people. Why would this be if it were completely devoid of psychological import? People seem to resonate with the idea that, to some extent at least, they are a product of their generation.