Hostname: page-component-cd9895bd7-jkksz Total loading time: 0 Render date: 2024-12-27T10:57:49.888Z Has data issue: false hasContentIssue false

Why I-O Journals Do Not Publish Qualitative Work

Published online by Cambridge University Press:  29 December 2016

Mark Fichman*
Affiliation:
Tepper School of Business, Carnegie Mellon University
*
Correspondence concerning this article should be addressed to Mark Fichman, Tepper School of Business, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213. E-mail: mf4f@cmu.edu
Rights & Permissions [Opens in a new window]

Extract

Pratt and Bonaccio (2016) ask why there is very little qualitative research published in “top [industrial–organizational] I-O” journals, such as the Journal of Applied Psychology (JAP), Personnel Psychology (PP), or Organizational Behavior and Human Decision Processes (OBHDP). They propose better training, placing skilled qualitative researchers on editorial boards, and “dispelling” myths related to qualitative research as a way to begin changing this situation. Even if these I-O journals explicitly invite qualitative work for its own sake, I do not think the steps Pratt and Bonaccio propose will have the intended effect, however valuable they may be in other respects.

Type
Commentaries
Copyright
Copyright © Society for Industrial and Organizational Psychology 2016 

Pratt and Bonaccio (Reference Pratt and Bonaccio2016) ask why there is very little qualitative research published in “top [industrial–organizational] I-O” journals, such as the Journal of Applied Psychology (JAP), Personnel Psychology (PP), or Organizational Behavior and Human Decision Processes (OBHDP). They propose better training, placing skilled qualitative researchers on editorial boards, and “dispelling” myths related to qualitative research as a way to begin changing this situation. Even if these I-O journals explicitly invite qualitative work for its own sake, I do not think the steps Pratt and Bonaccio propose will have the intended effect, however valuable they may be in other respects.

Publication Is a Communication Process

While the issues Pratt and Bonaccio raise have merit, I believe the explanation lies elsewhere, in the nature of the publication process itself and the social dynamics this process reflects. The publication process is, in its original conception, intended to facilitate scholarly communication. That is one of the primary purposes of publishing research. It is now considerably influenced by the need for “A pubs” in the race for tenure and the need to score in ranking systems such as those promulgated by the Financial Times or Business Week.Footnote 1 However much the races for tenure and rankings have twisted and distorted the process, in the end, publication is a communication device. Studying this communication process may help us better understand the situation Pratt and Bonaccio identify.

Choosing the Audience for a Publication

When I sit down to start a research project, my colleagues and I discuss what research question(s) we want to answer and where to present those answers if we find them. We discuss who the potential audience or audiences are for these results. This suggests where (i.e., what meetings and what journals) to present the results. One very quick and reliable way to do this analysis is to ask, “What papers have influenced our thinking and helped us define and refine our research question(s)?” In particular, where were these ideas presented? Ideally, if the ideas were presented in JAP, then that is where we would like to publish our results because that is where the research question is being discussed. A more sophisticated examination might also involve asking who is citing and discussing this work and where those citations are appearing. The objective is to find where we can effectively communicate with our audience. If I am studying the impact of personality on job performance, a quick look will tell me that JAP is one frequently used communication outlet and that Administrative Science Quarterly or Organization Science are less useful communication outlets for this work. Publications and citation searches can verify this.Footnote 2

These communication decisions explain why so few qualitative projects appear in JAP or PP. Unless I am studying a wholly new phenomenon never previously observed or discussed, my work will build on the work of others. If I am studying goal setting and job performance, I can look and see that JAP and OBHDP have published in this area, so I will consider those journals as outlets for my work. Much of what will appear in I-O journals is a result of what has come before in the pages of that journal.

If we look at publishing as a communication process, we can ask two questions. I will focus on JAP in my analysis to simplify the discussion, but the idea will apply more broadly to other I-O journals. First, do those who are using qualitative methods study problems that have arisen in an I-O journal like JAP? Second, do those who publish in a journal like JAP use qualitative research results in their own work? If the answer to both questions is no, then we would conclude that those who use qualitative methods are not studying problems that the I-O community discuss with each other when using JAP to communicate. The authors of a qualitative article might then be going to Academy of Management Journal or Administrative Science Quarterly because that is where conversations between qualitative researchers and others do happen. We are all accustomed to thinking about where different issues are discussed; that is fundamental to the identity of almost all scholarly journals. Experienced researchers often have very clear and nuanced understandings of where different work “should” appear given the past history and stated editorial practices of different journals. Occasionally, as Pratt and Bonaccio note, a journal will try to explicitly signal what work they are interested in, but most researchers have a pretty clear impression of where different ideas are being discussed and read. That interest is best identified by the studying publication and citation patterns.

Who Do Qualitative Researchers Listen to and Talk to When They Publish?

I am not going to present a systematic citation and network analysis, but I want to suggest with some informal analysis that the communication choices (i.e., where to publish and who to cite) qualitative and quantitative researchers make create the situation Pratt and Bonaccio observe, rather than the distribution of research skills in the I-O community and the editorial signals sent to the I-O psychology community. My hypothesis is that the lack of qualitative research in a journal like JAP occurs because the research community that uses JAP to communicate does not study the same problems or read the same work as those who are now doing qualitative research. Most qualitative researchers communicate using platforms other than JAP for the same reason. As a result, qualitative researchers will use different publishing platforms for communicating with other researchers (of all stripes, qualitative or quantitative).

A good example to use is the work of Michael Pratt and his colleagues. Pratt has published work on organizational identification that should be of interest to those who publish in JAP. I will consider two papers, Pratt (Reference Pratt, Whetten and Godfrey1998) and Pratt, Rockmann, and Kaufmann (Reference Pratt, Rockmann and Kaufmann2006), to illustrate my point. Both papers are on organizational identification. I used Web of Science or Google Scholar for my search, both of which should include all the citations to and from JAP articles. Pratt (Reference Pratt, Whetten and Godfrey1998) is cited 13 times in JAP. Not all 13 papers citing Pratt (Reference Pratt, Whetten and Godfrey1998) are on organizational identification. If I search Web of Science for papers on “organizational identification” in JAP, I find 19 papers on the topic, six of which cite Pratt (Reference Pratt, Whetten and Godfrey1998). Some people in the JAP community are reading this paper, and it has some influence on about one third of the work appearing in JAP on organizational identification.

To keep these numbers in perspective, Google Scholar reports 1,149 citations to Pratt (Reference Pratt, Whetten and Godfrey1998).Footnote 3 Six of the 1,149 citations to Pratt (Reference Pratt, Whetten and Godfrey1998) are in JAP, so there are people talking about this work in many other places as well. Pratt (Reference Pratt, Whetten and Godfrey1998) has 96 citations, and seven citations are from JAP, so there is some two-way communication here.

Pratt et al. (Reference Pratt, Rockmann and Kaufmann2006) is cited twice in JAP, and Pratt et al. (Reference Pratt, Rockmann and Kaufmann2006) does not cite any work in JAP. This paper is cited 769 times in Google Scholar. Pratt et al. (Reference Pratt, Rockmann and Kaufmann2006) is cited 37 times in the Academy of Management Journal. So although some very small amount of JAP research is influenced by this paper, it is not a primary or major audience for Pratt and his colleagues. Another prominent work noted by Pratt and Bonaccio is Ibarra (Reference Ibarra1999) on provisional selves and identification. The paper has 1,259 citations in Google Scholar.Footnote 4 Of those citations, only seven are in JAP. Ibarra cites 57 papers, not one of which was published in JAP. Here are two highly cited papers using qualitative methods on a problem of interest to those who publish in JAP. But the authors show little to, at best, modest interest in the JAP community, and there is little or no citation of the work in JAP by the I-O researcher community. There are others who are not in the JAP community who are making far more use of this work than is the JAP community itself. To summarize, I looked at three papers on organizational identification that are frequently cited. They are on a problem that researchers have published on in JAP. The papers are occasionally cited in JAP, so the answer to Question 1 above is a weak yes, JAP researchers are looking at some of this qualitative work. The answer to Question 2 is a strong no; the qualitative researchers’ work only modestly intersects with work published in JAP. Neither the qualitative researchers (Pratt and colleagues, and Ibarra) nor the I-O community is communicating with each other using JAP. Qualitative work may not be appearing in JAP because those who use qualitative methods may not be studying problems that are being actively discussed in JAP. Even when they are being discussed, the qualitative research results may be more actively discussed elsewhere rather than in JAP.

Researchers With Quantitative Skills Will Do Quantitative Work

Pratt and Bonaccio comment that “The choice of which method(s) to employ should follow the research question one chooses to pose” (p. 695). This is one reason why I-O psychologists might want to use qualitative methods. Although this is widely accepted in the social sciences and prescribed in research methods courses, this is not how researchers usually behave. Most researchers have certain skills and capabilities in which they specialize. These influence the research question(s) they choose and the methods they choose to study those research questions. McGrath (Reference McGrath1981) does tell us that, as Pratt and Bonaccio rightly note, different methods have different strengths and weaknesses, so researchers should use multiple methods to trade off on those strengths and weaknesses. However, it does not follow that each individual researcher has to use multiple methods, only that the group or community of researchers studying a particular problem will be well served by supporting the use of multiple methods. If multiple methods are employed, this is most likely to happen when different researchers with different skills study the same research question. We are all specialists to some degree with respect to method. So I might study a problem using my best skills (which are quantitative), and others (Ibarra, Pratt, or Bonaccio) might choose qualitative methods because that is where they have a comparative advantage. The researchers who do publish and communicate through JAP have used quantitative methods (in part, I assume, because that is in their skill set), and we can expect them to continue to use quantitative methods in their research because that is where their comparative advantage lies. The argument that they (as individuals) should use other methods when appropriate is reminiscent of the argument against the “drunkard's search” often attributed to Kaplan (Reference Kaplan1964). Kaplan (Reference Kaplan1964) tells the “story of the drunkard searching under a street lamp for his house key, which he had dropped some distance away. Asked why he didn't look where he had dropped it, he replied, ‘It's lighter here’” (p. 11). This certainly is the paradigmatic argument against always using the same method. Yet that is what each of us (with rare and notable exceptions) does. People often omit a later comment by Kaplan wherein he recognizes this monomethod preference of any individual researcher. Kaplan (Reference Kaplan1964) later suggests, “The drunkard's search is relevant here; the pattern of search, we feel, should be closely related to the probability of the thing sought being in the place where the seeker is looking. But the joke may be on us. It may be sensible to look first in an unlikely place just because ‘it's light there.’ . . . But we entertain the hypothesis, that is, we look in a certain place, for additional reasons: We happen to be in the place already, others are looking elsewhere, and so on” (pp. 17–18). So while ideally we choose the “right” method for a problem, the criterion for “right” is a bit fuzzier on closer examination. It is not unreasonable for us to use the methods with which we are most skilled, even if that is not the “optimal” or “ideal” method. It becomes less surprising that JAP is dominated by quantitative researchers, who, one hopes, are not drunk when they do their research. In effect, we have a community of quantitative researchers talking to each other through JAP, and we should not be surprised they continue to use quantitative methods, given that those methods are their strongest set of skills. Even if a problem merits qualitative methods, the current community of I-O scholars communicating through JAP would tend to choose those methods where they are most skillful, even if those methods might not be ideally suited to the question.

JAP Often Focuses on Essentially Quantitative Questions

Another reason that JAP may not publish much qualitative research is that the journal is concerned with a number of issues that are essentially quantitative. If I look at the most recent description of the journal's interests, I find an itemized list of 18 items. Of those 18, several are essentially quantitative by design. In particular, (a) individual differences in abilities, personality, and other characteristics; (b) testing and personnel selection; and (c) performance measurement and management are quantitative by definition, though one can surely study these phenomena usefully using qualitative methods. But “performance measurement” looks quantitative; it is hard to conceive of it as other than that. Individual differences in the I-O world have been quantitative for over a century, and testing has been quantitative for nearly as long. In a list that was not sorted alphabetically, these were the first three items on the JAP description of research topics of interest to JAP. Next on the list is “training, learning, and skill acquisition,” which psychologists have treated quantitatively for a century. Very near the end of the list is organizational culture and climate, which has been studied qualitatively in the larger organizational behavior literature. JAP also has been one of the primary outlets for meta-analysis, which is quantitative by definition. So the competition for space for qualitative research is a challenge as well.

My observations are not unique to organizational identification or the I-O field as exemplified by JAP. Consider Sutton and Hargadon's (Reference Sutton and Hargadon1996) excellent paper on brainstorming. Sutton and Hargadon report a wonderful qualitative study at IDEO, a company that uses brainstorming often and successfully. They make a very persuasive case that the issue social psychologists worry about, whether brainstorming leads to more and better ideas, is essentially irrelevant in a particular organizational context—a product design firm—that highly values brainstorming and practices it extensively. Sutton and Hargadon conclude that brainstorming researchers in social psychology miss what really happens when companies do brainstorming because social psychologists study the phenomenon in the laboratory rather than in the field. Several of Sutton and Hargadon's critical research citations are to articles in social psychology journals like the Journal of Personality and Social Psychology. Even though the authors criticize social psychology research for inattention to context, the paper appears in Administrative Science Quarterly, a journal rarely, if ever, read (or to be more precise, cited) by social psychologists but frequently cited by qualitative and quantitative organizational researchers and organizational sociologists. These two veteran qualitative researchers go to others in their community rather than the social psychologists they criticize.Footnote 5 Social psychologists are unlikely to see this work that so dramatically illustrates a critical shortcoming in their brainstorming research. The article, published in 1996, is cited 897 times in Google Scholar and 312 times in Web of Science. It is cited two times in the Journal of Personality and Social Psychology as of June 2016 (20 years later). These are different networks of scholars who, for a host of reasons (some of which are touched on by Pratt and Bonaccio), are not talking (and not listening) to each other.

Pratt and Bonaccio have identified a real difference between the I-O community and other research communities with respect to qualitative research. The I-O community and the qualitative researchers in organizational behavior are in different communities that rarely talk to each other. As a result, there is little qualitative research in I-O journals. I think the two communities do not talk to each other or read each other's work. This is the problem. The problem is not due to prejudices, poor signaling of interest, and a lack of skill in qualitative research in the I-O community. Those problems do exist, but they are not the cause of the infrequent publication of qualitative research in I-O journals. When qualitative and quantitative I-O researchers want to talk to each other, we will see qualitative work in journals like JAP.

Footnotes

1 I reside in a business school where discussions of these issues dominate many conversations at lunch tables and tenure reviews.

2 I understand that although we can choose where to submit our work, the journal's editors decide whether to accept our work for publication. I may be responding to work in JAP, but JAP may decide my work does not merit publication, and I may seek another similar communication outlet (Personnel Psychology, maybe). As a result, location of publication is a noisy signal given that much work was not accepted where it was first submitted. Nevertheless, we do still try to submit to journals where we think we can reach our intended audience. Our citation practices are more informative because, with rare exceptions, we do choose whom we will cite.

3 This and all other citation searches were done in June 2016.

4 The search was run on June 6, 2016.

5 To clarify the use of Administrative Science Quarterly as the publication outlet, it was the authors’ first choice. The authors considered going to a social psychology outlet but were persuaded from personal experience that the work would not be well received in a social psychology journal (Sutton, personal communication, June 16, 2016).

References

Ibarra, H. (1999). Provisional selves: Experimenting with image and identity in professional adaptation. Administrative Science Quarterly, 44 (4), 764791.Google Scholar
Kaplan, A. (1964). The conduct of inquiry: Methodology for behavioral sciences. San Francisco, CA: Chandler.Google Scholar
McGrath, J. (1981). Dilemmatics: The study of research choices and dilemmas. American Behavioral Scientist, 25 (2), 179210.CrossRefGoogle Scholar
Pratt, M. G. (1998). To be or not to be: Central questions in organizational identification. In Whetten, D. A. & Godfrey, P. (Eds.), Identity in organizations: Building theory through conversations (pp. 171207). Thousand Oaks, CA: Sage.Google Scholar
Pratt, M. G., & Bonaccio, S. (2016). Qualitative research in I-O psychology: Maps, myths, and moving forward. Industrial and Organizational Psychology: Perspectives on Science and Practice, 9 (4), 693715.Google Scholar
Pratt, M. G., Rockmann, K. W., & Kaufmann, J. B. (2006). Constructing professional identity: The role of work and identity learning cycles in the customization of identity among medical residents. Academy of Management Journal, 49 (2), 235262.Google Scholar
Sutton, R. I., & Hargadon, A. (1996). Brainstorming groups in context: Effectiveness in a product design firm. Administrative Science Quarterly, 41 (4), 685718.Google Scholar