Expanding the I-O psychology mindset to organizational success

Abstract

The paper proposes that industrial-organizational (I-O) psychology will benefit greatly from expanding our research focus from predominantly individual differences to studying organizational differences. We argue here that an increased organizational frame of reference on variables of interest to I-O psychology (e.g., selection, job design, performance management (PM), work motivation) is important because it will enhance our understanding of organizational behavior and make I-O research more effective in practice. After noting some organizational-level research already being done, several examples are provided for how an organizational mindset and methods can provide new insights into traditional areas of I-O effort. Also discussed is how methodological issues that may have constrained the study of organizational differences in the past and the potential new issues such research may yield can be addressed. We conclude that the future maintenance and enhancement of the I-O psychology brand as a science–practice profession requires enhanced attention to the organization level of analysis as our frame of reference for research.

Industrial-organizational (I-O) psychologists have made many contributions to research that have significantly influenced how companies operate, one of the most noteworthy being our contributions to personnel selection. Although teams, work units, and organizations have certainly been of interest, our predominant focus as a field has been on individual differences. This means that we tend to collect data on individual capabilities, attitudes, behaviors, personality, and perceptions, and relate these to criteria, such as individual performance and individual turnover. Beyond selection, we have applied this individual difference approach to studying many other domains. For example, many studies of leadership have gathered data from individuals about their supervisors and then related these to their individual performance. Similar individual-level work has been conducted in our studies of job characteristics, goal setting, employee engagement, and even organizational climate. In fact, early studies of organizational climate were so focused on the individual level of analysis (Schneider & Bartlett, 1968, 1970) that we have meta-analyses of climate studied at this level (Carr etal., 2003). Fortunately, more recent meta-analytic studies have been conducted at the organizational level of analysis (Beus etal., 2020), showing evidence for the usefulness of climate as a correlate of organizational performance.

In this paper, we present an argument for increased attention to organizational-level research on topics that are central to I-O psychology. We feel this is important for two reasons. First, if an I-O variable of interest has a positive effect on individual performance, we tend to assume that it will likewise be reflected in higher levels of organizational success but lack evidence to support these inferences. Second, organizational leaders who have been supportive of our work to date are increasingly asking questions about the extent to which the foci of our tools and processes improve organizational performance, not just individual performance.

Making the shift to focus more research at the organizational level requires aligning on a definition of what we mean by organization and adopting an organizational mindset. We define organization here as any work entity that has been chartered as a formal entity with an acknowledged purpose. Citibank is an organization and Citibank branches are also organizations, the latter of which we refer to as the work-unit level of analysis. However, work teams—aggregates with no formal designations—are not considered organizations. So, formal work units (divisions, departments) are suborganizations we refer to as work units in our way of thinking about levels of analysis. More research has been conducted on work units and less on umbrella organizations; it is the latter for which we are proposing that more focus is required.

The idea of adopting an organizational mindset is not meant to infer that we fail to think about organizational effectiveness. However, the majority of our conclusions about what drives organizational performance stem from individual differences research and to a lesser extent team research. You might be ready to challenge us: For example, what about the large research literature on organizational change? Even a cursory reading of that literature reveals that the studies were done within one organization at a time; thus, we have produced case studies, with little research that examines the effect of our interventions on organizational performance across organizations (Ford & Foster-Fishman, 2012). Similarly, we have produced ample evidence that the use of our selection procedures improves the performance of individuals in an organization, but there is scant evidence that organizations that use our selection methods outperform those that do not. The problem lies in our tendency to assume that the characteristics that produce high-performing individuals and teams also yield high-performing organizations, without testing this as often as we could or should.

Organizational studies would evaluate the effect of I-O interventions on company-level outcomes, such as customer satisfaction as in Schneider etal. (2009) and financial measures like profitability that business leaders care about as in Pulakos etal. (2019). In essence, adopting an organizational mindset would entail conducting more research to study organizational differences using the same basic approach that we have used so successfully to study individual differences. We believe this shift to the organizational level is needed to close gaps that exist today in our understanding of how I-O variables affect organizational performance. It is noteworthy that other disciplines frequently conduct research like we are advocating here. For example, the strategic management field (Freeman, 2010) and organizational sociology (Aldrich & Ruef, 2006), respectively, have studied how organizations deal with each other in turbulent environments and how organizations emerge and change over time as a function of larger societal issues. I-O psychologists have also conducted some organizational differences research, as we discuss below, but not nearly enough to develop a robust body of knowledge about the organizational effects of I-O interventions like using formal selection assessments, improving leadership, job design characteristics, goal setting, or managing performance through formal PM processes. However, this can be remedied by developing a stronger organizational mindset to prompt more examination of the I-O phenomena we study on organizational performance outcomes.

The research we are advocating here will provide unique insights into the human factors that contribute most to organizational success, thus clarifying for organizational leaders how to prioritize investments in various talent management systems, processes, and other interventions for maximum effect. This work has the potential to elevate the visibility and brand of I-O psychology, as organizational leaders continue to be challenged with disruption, complexity, and unprecedented change that requires evidence-based insights to enable organizational success and in some cases survival. Just think: We have made great strides in understanding individual performance by studying individuals at work; what would we learn if we focused as heavily on understanding organizational performance?

In what follows, we briefly review the history of I-O research that has yielded our large body of knowledge on individual differences. We then discuss more concretely what adopting an organizational mindset might look like and explore the status of such work in various subparts of our field: selection, leadership, job design, work motivation, PM, and organizational climate. We offer thoughts on why relatively few of our studies have been conducted at the organizational level of analysis and briefly discuss solutions to methodological challenges that may have deterred I-O psychologists from pursuing such research. We conclude that more attention to organizational-level research by I-O psychologists is likely to yield important insights into organizational success that can elevate the effectiveness and visibility of I-O research among organizational leaders.

Our tradition of focusing on individual differences

Our field has historically been grounded in individual differences research. Almost 100 years ago, Morris Viteles (1932) produced the major text on industrial psychology, largely devoted to the study of individual differences and the development and validation of tests for hiring and promotion. As Viteles put it, “Industrial psychology is based on the study of individual differences—of human variability” (1939, p. 29). By 1932, procedures had been developed for job analysis, criterion development, performance appraisal, and selection. Tests already existed for skilled (bricklayers and lathe operators) and semiskilled workers, different types of transportation workers (trolley and railway motormen), and office and clerical occupations. Indeed, in 1928, Hull—the famous learning scholar—summarized the then-known validity of a vast number of aptitude tests, demonstrating how effective they had proven to be. At the same time, a literature on personality tests (“temperament and character”) was well underway (May et. al., 1927, first in a series of four reviews), as was development of the Strong Vocational Interest Blank (Strong, 1929). And we never slowed down.

Our emphasis on individual differences has stood the test of time, as companies have found our evidence-based approach to selection to be persuasive and useful. Comprehensive volumes have been written that demonstrate our accomplishments in this area, for example, Schmitt’s (2012) and Farr and Tippins’s (2017) edited volumes. We have produced a vast research literature that has proven remarkably useful in practice, showing that our selection procedures help companies hire more effective employees and yield substantial cost savings (Scott & Cascio, 2017). What we have studied much less is whether companies that use validated selection procedures outperform those that don’t in terms of their financial performance, meeting business objectives, and creating value (see Ployhart & Schneider, 2012, as cited in Schmitt, 2012, and Ployhart & Weekley, 2017, as cited in in Farr and Tippins, 2017). An example of the type of research we are advocating here was conducted by Huselid (1995), who showed that the use of bundled “high-performance work practices,” including formal selection procedures, yielded higher organizational financial outcomes, a finding that has since been replicated by others (Combs etal., 2006; Jiang etal., 2012; Wright etal., 2000).

Toward an expanded focus on organizational differences

The field of I-O psychology has grown to appreciate that important phenomena occur at higher levels, such as the team, work unit, and organizational levels. This shift began in the mid-20th century with books titled “Organizational Psychology” (Bass, 1965; Schein, 1965), although these did not yet focus on the organizational level of analysis. Over time, interest in adopting an organizational focus has grown, with Rousseau (1985) and others promoting multilevel research and well-regarded books on this topic such as Klein and Kozlowski (2000). The development of hierarchical linear modeling (HLM) procedures (Mathieu & Chen, 2011) has clarified the usefulness of attention to levels above the individual. However, it is noteworthy that HLM has been used by I-O psychologists mostly to find levels of analysis above the individual level that contribute to predicting individual behavior (Mathieu & Chen, 2011).

I-O psychology handbooks, one edited by Kozlowski (2012) and the other edited by Zedeck (2011), reinforce the larger point we are making here about the relatively small amount of attention that has been given to studying organizational effectiveness. Only three chapters in the Kozlowski volume discuss the type of research we are suggesting here. First, Ployhart (2012) notes that selection research needs to be done differently to show its effects on organizational effectiveness, citing studies that have shown how effective selection practices produce organizational success (e.g., Ployhart etal., 2009; Van Iddekinge etal., 2009). The second chapter, by Snow and Snell (2012), concerns work done by strategic human resources (HR) researchers to study the effect of high-performance work practices on organizational financial and operational outcomes—findings that were confirmed in subsequent meta-analyses (e.g., Combs etal., 2006; Jiang etal., 2012). The third chapter examines the effect of organizational climate and culture on effectiveness (Zohar & Hofmann, 2012) across organizations and units within organizations, showing that organizational service climate and safety climate predict customer satisfaction (see Hong etal.’s, 2013 meta-analysis) and accidents/injuries (see Zohar, 2014), respectively. Indeed, organizational climate research is a good example of what is possible when we adopt an organizational mindset and conduct sufficient research across different outcomes to establish the importance of a phenomenon for organizational performance (Beus etal., 2020).

In the Zedeck (2011) handbook, only one chapter, by Schneider etal. (2011)—also on organizational climate—reflects organizational-level research. An important feature of this chapter—discussed in more detail below—is the evidence it provides showing that data from individuals can be aggregated to create organizational-level measures that have significant relationships with organizational outcomes. This chapter thus provides a useful model that can be applied to studying numerous organization-level phenomena based on the kinds of data we collect on individuals. Beyond this example and those above, however, relatively little research has adopted an organizational mindset to study the effect on organizational performance of focal I-O variables like leadership, work motivation, job design, and PM practices.

Shifting to an organizational mindset

I-O research, begun over 100 years ago, as summarized by Viteles (1932), focused on solving problems faced by companies, initially concerning the hiring and appraisal of workers. Much of the work was in the form of exploratory research that focused on developing measures of individual differences that might be reflected in worker job performance and not on testing theories. Although interest in theory has increased over time, I-O psychologists still perform considerable exploratory research in organizations to inform I-O theory and practice. In what follows, we discuss ideas for exploratory research that can be conducted to improve our understanding in several I-O topic areas.

Selection

We have already noted that relatively little research has examined the effect of selection practices on organizational performance. An important question that we posed above is this: Do companies that hire a larger proportion of people using formal assessments—thus having higher levels of relevant knowledge, skills, abilities, and other characteristics (KSAOs)—outperform companies that use proportionately less of these? Utility analyses tell us that part of the answer to this question is yes; companies that use formal assessments do better by using them than they did before. What we do not know is whether they outperform other companies. But there are a few hints that they do because (a) we also know that high-performance work practices, including selection, have been associated with higher organizational performance (Combs etal., 2006) and (b) Ployhart etal. (2011) showed that aggregate KSAOs in companies get reflected in the kinds of training offered, the service performance of employees, and organizational performance differences. Indeed, Kim and Ployhart (2018) showed that firms using more selective hiring practices outperformed their competitors both before and especially after the Great Recession (Ployhart etal., 2017). But this is about all that we know to date about organizational performance based on selection kinds of practices. Can organizational analyses modeled after job analyses produce people who fit better and thus improve organizational performance? We actually also have hints on this latter question.

Attraction-selection-attrition theory (Schneider, 1987) suggests that the personality attributes of people differ across companies due to self-selection in and out of companies and companies deselecting those who do not fit. Considerable evidence now exists that there is significant differentiation across companies in the personalities of the people in them (Oh etal., 2015; Oh etal., 2018). Recent research indicates that the selection phase of the attraction-selection-attrition model is most responsible for the relative homogeneity that exists in companies (Oh etal., 2018) and that the better the fit within a company the more likely it is to be financially successful (Schneider & Bartram, 2017).

The above studies reveal organizational-level effects stemming from aggregate individual KSAOs. Given the current state of data analytics and availability of large data bases, we could provide additional data-based insights that address the effects of formal selection assessments on organizational performance. There are many questions that have yet to be addressed, for example: Do assessment centers or the use of structured interviews yield comparative success? Do companies with higher levels of cognitive ability outperform those with lower levels of cognitive ability? For what jobs and organizational sectors do formal selection procedures matter most? And, when human capital resources are considered simultaneously with differences in leadership and work design, what is the relative contributions of these different correlates to organizational performance?

Leadership

We have produced a very large number of leadership studies (Day, 2014), only a fraction of which have examined leadership effects on organizational performance outcomes. An example of one such study, conducted by Hiller and Beauchene (2014), showed relationships between CEO/executive leadership and organizational performance by charting the effects of leadership changes, which were associated with changes in organizational performance. Other research has been more granular in studying the effects of specific leader behavior on organizational performance, yielding more actionable insights. One such study examined the effects of CEO transformational leadership on organizational climate and organizational identity across 150 companies, indicating that transformational leadership was shown to relate positively to firm performance (Boehm etal., 2015). In another study across 147 companies, Shin etal. (2015) showed that ethical leadership among top management was predictive of ethical climate, which in turn predicts a procedural justice climate that mediates the effects of ethical leadership on organizational-level citizenship behaviors and organizational-level financial performance. Similarly, Huang etal. (2016) examined data across 90 hotels and showed significant relationships between CEOs behaving as servant leaders producing more positive service climates and higher performance (see also Peterson etal., 2012 and Wang etal., 2011 for similar studies).

It is noteworthy that in a meta-analysis of 305 studies on ethical, authentic, and servant leadership (Lemoine etal., 2019), only 17 (5%) appear to have examined effects at the organizational (called “firm” in the paper) level of analysis (see Table 2), and two thirds of those concerned the effects of servant leadership on customer outcomes in service settings. Arepresentative study was conducted by Hmieleski etal. (2012), who studied the effects of authentic leadership on new venture performance. The effect of authentic leadership on new venture performance was mediated by the affective tone leaders created. Again, we see the idea that leaders create climates that significantly affect organizational performance. This research demonstrates the important insights we can gain by studying the effect of I-O variables on organizational success. Unfortunately, the few studies we have cited relevant to the organizational level of analysis for research on leadership represent a very small piece of the body of evidence we have created pertaining to leadership.

We have numerous conceptualizations of leadership behavior that can be explored from the organizational level of analysis. For example, consider the vast research literature on leader–member exchange (LMX; Dulebohn etal., 2012; Gottfredson & Aguinis, 2017; Graen & Uhl-Bien, 1995). Ameta-analysis of that research across 102 samples (Martin etal., 2016) shows that LMX is significantly related to positive outcomes for employees (e.g., trust, motivation, job satisfaction, and task performance). But this research was all conducted at the individual level of analysis. An earlier meta-analysis of LMX reached similar conclusion in analyzing 247 studies of LMX antecedents and consequences and encouraged future research on organizational effects, which we could not find.

LMX-relevant questions that can be explored at the organizational level of analysis include the following. Across companies, do the mean and variance of company-level LMX ratings correlate significantly with organizational performance? That is, when leaders are seen as having positive relationships with a higher proportion of their subordinates (high means), is that reflected in higher company competitive performance? Moreover, does the variability or strength of the within company reports serve as a moderator of the LMX mean–organizational performance relationship? Interestingly, one might hypothesize, based on team-level LMX research that high LMX variances would be associated with low organizational performance due to the negative influence high differentiation can have on job attitudes, coworker relationships, and turnover in teams (Erdogan & Bauer, 2014). For example, if servant leadership has its effects mostly through the climate that gets created by leaders, what kind of climate does high LMX differentiation create in organizations? Knowing that differentiation has negative consequences on individuals and group processes (Yu etal., 2018), a question is this: Does high LMX differentiation produce subclimates in organizations that leave some feeling they are in the in-group and others feeling they are in the out-group, and what effects do these subclimates have on organizational success? The main point here is that relatively few studies of leadership (mostly transformational and servant leadership) have examined effects at the organizational level of analysis and that substantial contributions to theory and practice are likely by studying the direct and indirect effects of leadership behaviors on organizational climate and performance directly (Schein, 2010).

Job characteristics and work motivation

The Hackman and Oldham (1980) research paradigm to study job characteristics has been the subject of hundreds of studies. An organizational-level question we can ask here is this: Do companies with higher aggregate scores on the Job Diagnostic Survey (JDS) have higher organizational performance? In an era of crafting jobs for flattened hierarchies, increased work automation and feedback from work (Oldham & Hackman, 2010; Petrou etal., 2018), it is likely that some companies are making the same basic work different in terms of its motivating potential and meaningfulness. Thinking at the organizational level, we might ask: Are companies that design work to involve the whole person and meaningfully align to the firm’s goals creating higher engagement, financial performance, and lower turnover at the company level? Indeed, we found a study by Wood etal. (2012) that did just that. Wood etal. had access to the UK Workplace Employment Relations Survey 2004 and studied enriched jobs (task variety and job autonomy) across 1,177 workplaces in the UK. They examined the enriched job index in relation to various performance indicators including financial performance, labor productivity, quality, and absenteeism rates and found that the enriched job index, as mediated by job satisfaction, was a significant predictor of all performance outcomes. Interestingly, this study also included an index of high involvement management that was not significantly related to any of the outcomes. Job characteristics seem to matter.

There are additional hints of this possibility in the organizational-level studies that have been done on a variety of high-performance human resource management characteristics. In particular, the meta-analysis by Jiang etal. (2012) categorized such characteristics into three major variables: skill enhancing (e.g., selection, training), motivation enhancing (e.g., PM, rewards), and opportunity enhancing (e.g., flexible job designs, employee involvement). They showed across 120 organizations that higher levels of these characteristics were associated with lower turnover and superior operational (quality, labor productivity) and financial outcomes. The variable concerning job design was especially strong regarding turnover and operational outcomes.

Additional hints come from several recent studies of organizational-level work engagement with validity against organizational performance, including financial outcomes (Barrick etal., 2015; Schneider, Yost etal., 2017). In Schneider etal., a new measure of job characteristics was shown to be a significant correlate of workforce engagement that mediated its relationship to organizational financial performance and customer satisfaction outcomes. Thus, if the JDS (or another well-researched measure of work design; see Parker etal., 2017) was used to tap into the underlying antecedents of workforce engagement across companies, such studies may also yield workforce engagement (Schneider, Yost etal., 2017) as the mediator between job characteristics and organizational performance.

We want to raise the related issue of work motivation, given that the JDS factors are seen as stimulating work motivation and sources of job satisfaction (Hackman & Oldham, 1980). Aquestion that comes to mind is this: How do overall levels of work motivation influence organizational performance? That is, do companies whose workers are more motivated outperform those whose workers are less motivated? Today, we have few evidence-based answers, and those that exist are of the Jiang etal. (2012) or work engagement sort just reviewed. That is, the motivation is either a composite of motivation-enhancing organizational practices or an index of work engagement.

A fine example of what is possible in this respect is the research program called ProMeas (see Pritchard etal., 2008, for a meta-analysis). ProMeas is a work unit- or department-based productivity intervention technique developed by Pritchard (Pritchard, 1995) that is grounded in a combination of valence-instrumentality-expectancy (VIE) theory (Naylor etal., 1980) and goal setting (Locke & Latham, 2019). ProMeas first identifies productivity indicators that reflect goal attainment. Unit members are then provided with frequent feedback on their accomplishments relative to their goals and taught how to alter processes to achieve goals. ProMeas has been shown to drive 80% to 90% greater productivity in units that receive this intervention, and the intervention usually lasts 6 months or longer. It is noteworthy that this successful work-motivation-based productivity intervention—directed at unit performance—has received scant (Kanfer, 2012) or no (Kanfer etal., 2017) attention in reviews of the work motivation literature.

In summary, although many of us in I-O psychology grew up in an era where job characteristics and work motivation were prominent topics of study, both then and now the way these play out at the work unit and organizational levels of analysis has received scant attention. As a field that has as its base a fundamental interest in the motivation people experience at work, it is surprising to find a lack of attention to how these motivation variables affect organizational success. Perhaps the recent interest in work engagement at the organizational level of analysis will generate renewed interest in the pursuit of foundational theoretical approaches, like the JDS (Hackman & Oldham, 1980; Oldham & Hackman, 2010), VIE theory (Naylor etal., 1980), and goal-setting theory (Locke & Latham, 2019), the latter of which also has not been studied in field research conducted at the organizational level of analysis (see Smith etal., 1990 for a lab study of 12 organizations).

Performance management

I-O researchers and practitioners have assumed (hoped?) that the presence of well-developed, formal systems through which people receive behavioral and developmental feedback will elevate their individual job performance and, in turn, the organization’s performance. However, organizational-level research conducted by the Corporate Leadership Council (Corporate Executive Board, 2004) raised significant questions about our assumptions regarding the value of formal PM systems in organizations. Looking across hundreds of organizations, the Corporate Leadership Council evaluated the influence on organizational performance of formal PM system features (things we believed were essential for driving high performance) and informal leader behaviors related to managing performance day to day. The results surprisingly showed that the formal PM system features we have obsessed about for decades (e.g., what performance to evaluate, what rating scale to use, how many formal reviews should occur during the year) have no influence whatsoever on achieving organizational performance outcomes. However, the extent to which leaders engaged in informal PM behaviors on a regular basis had the strongest effects on organizational performance. These behaviors included providing real-time feedback, adjusting goals as situations change, and helping employees solve problems—not unlike what ProMeas (Pritchard etal., 2008) had shown to be useful. Asimilar organizational-level study conducted by Ledford and Schneider (2018) likewise showed that when it comes to driving organizational financial success, companies should invest in developing a strong feedback climate and worry much less about the specific performance appraisal and management techniques used.

In the case of PM, it is easy to see the value and in fact necessity of conducting organizational-level research. It was only through moving to the organizational level that we were able to see the relative influence of the formal versus informal PM characteristics in driving organizational performance outcomes. Furthermore, the results of this research had a profound influence on both research and practices, creating a sea-change in what organizations focus on in developing their PM processes and prompting massive experimentation with new tools and PM transformation efforts starting about 10 years ago and continuing today (Pulakos & Battista, 2020). Imagine what other insights and substantial shifts may result from increased research attention at the organizational level of analysis.

Organizational climate

Climate is an area for which organizational and unit differences have been studied more extensively than most others (Schneider, González-Romá, etal., 2017). Although organizational climate research started at the individual level of analysis, climate researchers elevated their work to studying the unit and organizational levels of analysis in the 1980s by relating climate characteristics to outcomes such as service quality and accidents (Schneider, 1980; Zohar, 1980). This research has consistently shown that focused climates (e.g., climate for safety and climate for service) are reflected in appropriate organizational performance outcomes like accidents and injuries (Zohar, 2014) and customer satisfaction (Yagil, 2008); these relationships have been replicated with considerable reliability over time (Beus etal., 2020; Schneider, González-Romá, etal., 2017). As we suggested in discussing leadership, it may be that climate can be conceptualized as a mediating variable in exploring job design, work motivation, and PM when conducting such research at the organizational level of analysis. What kinds of climate are created when jobs are inherently more meaningful or more involving or offer more opportunities for job crafting in a company? Maybe people do act more engaged and find that their coworkers behave in ways that create a positive engagement climate (Barrick & Mount, 2015)? Pritchard etal. (2008) do not use the term climate when describing how goal setting tied to feedback yields significant interdepartmental differences in companies, but perhaps such tactics produce a climate for learning and for accomplishment, precisely the kinds of climate some have suggested are necessary for organizational change to happen (Argote, 2012). Perhaps the more focused a PM process is on critical company-level outcomes, the more likely it is that employees will see that system as one producing appropriate targeted effort and accomplishment, and then, the system might get reflected in actual organizational performance.

Organizational agility

We leave this discussion with one final example of organizational differences research—this one directed at understanding organizational agility. Although this topic is not a traditional I-O research topic, it has been of great interest to business leaders facing exponential change and disruption from advancements in automation and now a global pandemic that threatens their survival. The prevailing wisdom from thought leaders and reflected in “best practices” was that that hiring and training agile leaders, and standing up agile teams, was the path to building organizational agility. However, like so many of the examples we have presented, no empirical research was ever conducted to investigate whether these strategies would actually create organizational agility that leads to competitive success.

To address this gap, Pulakos etal. (2019) collected data from over 300 organizations globally on several factors that they hypothesized may affect organizational agility (e.g., strategy, climate, teamwork, leader behaviors, and so forth). The results of this research showed that three surprising factors had the most influence on organizational agility, which in turn had very substantial relationships with corporate financial performance. Paradoxically, the factor that had the most influence on agility was organizational stability—a condition that leaders create by engaging in several specific stabilizing behaviors (e.g., clarifying focus, building optimism, harmonizing resources, and learning from failure, among others). This research provides an example of how I-O psychologists can contribute important evidence-based insights by conducting organizational differences research in areas that are front of mind for organizational leaders. In the case of organizational agility, without evidence-based research confirming the factors that matter most in creating agility, companies were misguided in their efforts and investments to build agility and in fact were likely undermining it (see Pulakos etal., 2019, for a complete discussion).

We began this paper proposing that an important advantage of doing organizational differences research is that it has high potential for uncovering unique insights on topics that are important to business leaders and thus can provide a platform for gaining broader attention, visibility, and influence for our I-O work—things we continually try to achieve. The organizational agility research described here is an example of this idea coming to fruition. Two spin-off articles were recently published in Harvard Business Review, an accessible and visible outlet, that provided plain-language evidence-based tips to guide leaders in how to create agile organizations and teams (Pulakos & Kaiser, 2020a, 2020b).

Why is there less research at the organizational level of analysis?

We think there are five major reasons I-O researchers may shy away from organizational-level research: (a) the assumption that our work at the individual level of analysis generalizes to the organizational level, (b) the individual differences mindset, (c) methodological issues (obtaining samples and data aggregation), (d) development of valid criterion data, and (e) less familiarity with new methods and tools relevant to organizational-level work.

Implicit assumption of generalizability

As we noted earlier, we tend to implicitly assume that if individual differences predict individual performance, they will also positively affect organizational performance. This is an ecological fallacy—in other words, an attribution of behavior from one level of analysis for which one has data (e.g., individual) to another level of analysis (e.g., the organization). But our assumptions that high individual or team performance translate to high organizational performance may be incorrect. For example, DeNisi and Murphy (2017) reported a lack of evidence for linkages between changes in job performance (as a result of training, changes in technology, PM, etc.) and organizational effectiveness. The problem is that we simply do not have enough such research to draw conclusions about what organizational characteristics have the most influence on organizational success metrics like customer satisfaction, growth, and profitability.

The individual mindset

The second reason for relative lack of organizational-level research stems from the fact that we view the variables we study with an individual rather than organizational mindset. If we do not think we have an organizational variables—something obvious like organizational climate—then we think we cannot study them as organizational-level phenomena. The reality is that every individual-level variable can be aggregated to other levels of analysis; psychological variables exist in aggregate as well as across individuals. As an example, after his extensive research on individuals’ need for achievement (nAch), McClelland (1961) studied it at the country level of analysis and showed important between country differences in economic progress as a function of country-level nAch.

Methodological issues

There are two possible major methodological constraints that can complicate studying organizational differences. First is the most obvious issue that this research requires studying phenomena systematically across many organizations. The availability of new data sources (e.g., big data; Fink, & Macey, 2021) and new data analytic techniques (e.g., partitioning techniques; Strobl etal., 2009) may somewhat alleviate such difficulties. It can be hard enough to gather data within one organization, letalone across multiple organizations. In spite of these complications, it is important to remember that other fields are conducting organizational-level research and producing insights that gain attention in media outlets and from top organizational leaders. We would likewise benefit from an external-visibility perspective by conducting more organizational-level studies so that I-O psychology is at the forefront of providing unique and important insights about what leads to successful business outcomes that our evidence-based research has the potential to provide. As just one example, consider the recent studies of employee engagement done at the company level of analysis against company financial outcomes revealing how useful the variable may be at this level of analysis (Barrick etal., 2015; Schneider, Yost, etal., 2017). Or consider the efforts at Gallup over the years to reveal how their employee attitude survey (referred to as the G12) correlates with not only individual performance but also organizational performance (Harter etal., 2020).

For researchers at large, however, there are several viable paths for conducting organizational-level research such as studying small organizations like family-owned businesses, local grocery stores, restaurants, start-ups, and so forth. Or, as climate researchers have done, we can study the many semiautonomous organizations that exist in large, networked chains—branches of banks, hotels in hotel chains, auto dealerships, and so forth. Another option is to seek samples of organizations from research and/or consulting firms that study or work with many organizations like the Mayflower Group (Schneider etal., 2003) or the Center for Creative Leadership (Schneider etal., 1998) that have access to multiorganizational data sets. Relatedly, Murphy (2020) offered the idea of leveraging organizations like the Society for Industrial and Organizational Psychology (SIOP) to spearhead consortium studies across organizations that will enable developing insights about what matters most in driving organizational performance.

Fortunately, some consulting firms with access to many organizations have also been publishing their efforts, so we have access to their findings as well. For example, the researchers at Gallup have produced several articles of note across units and organizations over the years, a recent example being Harter etal. (2020) in which they showed that, over time, aggregate global satisfaction in companies predicts the financial performance, customer satisfaction, and turnover in and of organizations—especially in times of economic recession.¹ Jack Wiley while at Kenexa (e.g., Dietz etal., 2004; Wiley, 2010) also served researchers well by making public his research linking organizational practices to important organizational consequences. It would of course be useful to the research base of the field if more consulting groups that have data across multiple companies actually published results of their work—which we know exists.

Another potentially interesting tack, used extensively by our HR colleagues, is using a very small sample of respondents—even only one—from many organizations. Such studies would never have been deemed acceptable in the field of I-O psychology when this was first done (Huselid, 1995), but findings from work of this kind have proven to be highly replicable (Combs etal., 2006). The idea here is to sample a key informant (like the chief human resources officer or a knowledgeable senior leader with insight into how the organization operates) to report on what they see in their organization. Reports from carefully chosen single respondents have been shown to be sufficiently robust and replicable that they can be used on their own if additional respondents are not available. One such study, by Ledford and Schneider (2018), showed that reports on a climate for feedback from key informants across 90 organizations correlated significantly with corporate financial performance.

The Pulakos etal. (2019) study also found significant relationships between organizational factors and financial performance outcomes using a sample that comprised a large number of single respondents. In this study, there were multiple respondents from some companies and only one from others, but when the analyses across all companies were done, using only one respondent, the results were the same as using the maximum sample per company. Interestingly, the agreement (r _wg) observed among multiple raters of the organizational phenomena included in the study was surprisingly high (>.95)—in fact, so high that adding respondents did not show the typical increases in relationships we would expect to see. These findings suggest that items worded to capture organizational features provide a common frame of reference that helps raters agree on their reports. Perhaps it is time to adopt a new mindset regarding organizational research that we can realistically do this instead of thinking it is too complicated to be within reach.

The second methodological constraints concern the issue of data aggregation. We suspect that avoidance of organizational-level research has been fueled by questions about how to aggregate data to higher levels of analysis. Issues associated with item wording and data aggregation took about a decade of research attention to work out, from approximately 1990 to 2000. Klein and Kozlowski (2000) provide an extensive treatment of both the conceptual and methodological issues associated with data aggregation, and anyone venturing into organizational differences research must become familiar with these. However, we have been at a point for some time now that we can dispense with data-aggregation concerns—we now know the issues and how to do it—and proceed with conducting the type of organizational-level research that is important for us to perform. We need not go into great detail here on the issues of measurement issues vis a vis aggregation because such information can be found in Bliese (2000), Chan (1998), and LeBreton and Senter (2008).

But one issue is really necessary to discuss and that issue is item wording in survey research. Chan (1998) has persuasively argued that survey items need to be written to depict the targeted level of analysis being studied. For organizational-level research, this entails writing items that ask people to describe what they are seeing regarding the phenomena of interest in their companies overall rather than asking them to reflect their own personal attitudes and experience. Using an example from our earlier discussion of LMX, below are the two items from the LMX survey that is most often used to collect data from individuals about their dyadic relationships with their managers (Erdogan & Bauer, 2014):

1. 1. Do you know where you stand with your leader … do you usually know how satisfied your leader is with what you do?

2. 2. How well does your leader understand your job problems and needs?

Transforming these items to capture organizational-level differences across companies instead of each individual’s own experience, per Chan (1998), might produce rewording as follows:

1. 1. Do people in your company know where they stand with their immediate supervisors … do people usually know how satisfied their immediate supervisor is with what they do?

2. 2. How well do immediate supervisors in your company understand their employees’ job problems and needs?

Using another example from our discussion above, items from the JDS (Hackman & Oldham, 1980) can likewise be rewritten to shift respondents from describing their own jobs (as is usually done) to describing jobs in their companies, in general. Looking at the first two items from the JDS:

1. 1. I have almost complete responsibility for deciding how and when the work is to be done.

2. 2. I have a chance to do a number of different tasks, using a wide variety of different skills and talents.

Rewording these to describe the nature of the jobs in the company overall could be done as follows:

1. 1. People who work here have almost complete responsibility for deciding how and when their work is to be done.

2. 2. People here have a chance to do a number of different tasks, using a wide variety of different skills and talents.

Recent research has shown that Chan’s (1998) method of item wording to reflect the organizational level of interest yields more reliable data than using survey wording not matched to the target level (Wallace etal., 2016). For more examples of item-wording changes see Schneider (2020).

Organizational-level criteria

The fourth challenge we face, also discussed by Murphy (2020), is being able to develop and collect reliable, consistent criterion measures against which organizational differences can be evaluated. Developing accurate criterion measures with solid measurement properties can be challenging at any level of analysis, but in the case of studying organizational differences, we are aided by the availability of publicly reported financial metrics that provide solid criterion measures of a certain type for the subset of companies to which these apply (e.g., Compustat for financial data and the American Customer Satisfaction Index or ACSI for customer satisfaction evidence). In cases in which these are not available or other types of criterion measures are desirable, Murphy (2020) argues for building a body of evidence with available measures. Even with imperfect measures, consistent results and identification of trends across these can provide a basis for reaching solid organizational-level conclusions (for a recent such example see Harter etal., 2020).

New analytical methods and tools

The type of organization-level research we are advocating will likely yield more exploratory research with larger sets of variables than our research studies typically involve, generating the need to expand our statistical/methodological tool kit. Strobl etal. (2009) discuss techniques such as recursive partitioning that would be helpful to apply in conducting organizational difference studies. We can look to fields that have long histories of conducting organizational-differences research, such as our strategy colleagues, for lessons learned. For example, decisions have to be made about how many control variables to include, and there has been a tendency among researchers who do this type of work to include excessively large numbers of these, which cloud the meaning of organization-level findings. The techniques discussed by Strobl etal. (2009) provide useful strategies for addressing this (also see Therneau & Atkinson, 2017, for a more recent discussion).² In short, more organizational differences will bring new methodological challenges and tools to I-O research, but like the other potential deterrents discussed here, they can be resolved.

Summary and conclusions

As we were thinking about and writing this paper, we became excited by the thought that it is possible to conceptually integrate across the various topics we discussed to produce a relatively comprehensive view of the human organization: who gets hired and how, the jobs they work at and the motivation they experience, the leadership they encounter, and the climate in which they work. Such integration is possible and provides the potential for even more valuable insights onto organizational performance. The story may go something like this: The people selected by organizations have a direct effect on the ways leaders lead them (LMX says more competent people get better treatment by their leaders) and also have a direct effect on the kinds of training that will be offered in the companies (more talented people receive more extensive and intensive training; Kim, & Ployhart, 2018). More talented people also get to work at and create more meaningful jobs and thus become more highly motivated and engaged and working in an agile organizational climate that has stability at its base—all of these should yield superior organizational performance. The point is that we have the concepts and methods to do the type of organizational-level research that will yield more robust and evidence-based conclusions about the factors that are most critical in driving organizational performance.

We have offered several questions here that are worthy of exploration and, importantly, will provide insights to expand our knowledge and likely others’ interest in I-O work. We have discussed strategies that enable us to measure organizational-level phenomena, overcome challenges associated with gaining access to organizational-level samples, and deal with new methodological issues. We acknowledge that doing this type of research may require forgoing the highest level of gold standard measurement that I-O psychologists notoriously demand. We may need to accept smaller samples of respondents within each organization, even single respondents at times. We may need to accept imperfect criterion measures—for example, different items to assess customer satisfaction or have variable reliabilities across companies. But we can also leverage the American Customer Satisfaction Index or ACSI (https://www.theacsi.org/) that is publicly available on the largest American companies or Glassdoor employee data (https://www.glassdoor.com/Reviews/index.htm) as new organizational criterion measures.

When we consider the largely untapped area of studying organizational differences and the potential it has for highly effective evidence-based insights that I-O psychologists can uniquely provide, we believe the opportunity outweighs the concessions in research design and measurement that may be needed. Focusing on the I-O topics we have mentioned and others would clearly differentiate our efforts from the human resources community that focuses on high-performance work practices. In sum, we encourage a shift in mindset, paths, and tactics to break the logjam that has kept I-O research focused predominantly on individuals. It is in within our reach to do this and time to expand and enhance the relevance of the evidence-based insights I-O psychologists can provide to organizations.