Toward Better Hiring Practices

Introduction

Observers of gender dynamics in the academy have long characterized academic careers as a “leaky pipeline,” which refers to the tendency for women to occupy a steadily decreasing proportion of academic positions as the rank and status of those positions increase. Among the many loci of such leaks, implicit and explicit biases against women have been shown to affect the hiring process across the entire range of STEM fields (Moss-Racusin et al. 2012; Storage et al. 2020), including political science.

This article describes the impacts of several hiring practices that offer the potential for reducing gender-related biases in that process. Our description takes the form of a case study, focusing a faculty search at a Carnegie “Doctoral—Very High Research Activity” (“R1”) university in a political science subfield that has been and remains overwhelmingly male dominated: political methodology. The innovations include establishing clear ex ante criteria for evaluating applicants, emphasizing “fit” to the position as advertised and postponing reading letters of recommendation until candidates were ranked in a “long list” based on other evaluation criteria. Although many of these strategies increasingly are being adopted as best practices, our case study provides evidence of the immediate effects of these strategies on gender balance, both on the search in question and (briefly) in subsequent searches. Our experience suggests that such procedural changes offer the potential to increase gender diversity in the hiring process. Moreover, in most instances, these innovations are simple and largely costless to adopt.

Our experience suggests that such procedural changes offer the potential to increase gender diversity in the hiring process. Moreover, in most instances, these innovations are simple and largely costless to adopt.

The Search

In the two decades since political methodology was famously (if hilariously) described as a “welcoming discipline” (Beck 2000), numerous studies have documented the pervasive, persistent gender imbalance in scholars seeking, making, and continuing careers in political methodology (Breuning and Sanders 2007; Dion, Sumner, and Mitchell 2018; Roberts 2018; Shames and Wise 2017; Teele and Thelen 2017). As recently as 2018, a report by the Diversity Committee of the American Political Science Association (APSA) Society for Political Methodology noted that “the political methodology field faces severe diversity challenges” (Hidalgo et al. 2018, 7).

The political science department at Pennsylvania State University recently received authorization to conduct an open-rank search for a tenure-track professor with a specialization in quantitative methodology. ¹ The authors were appointed to the five-member search committee, with Zorn—a senior male faculty member—acting as committee chair. Carlson, then a junior faculty member in the department, was the committee’s only female member.

At the time of the search, the department’s climate committee had created a draft memorandum outlining a series of best practices for departmental search committees, in keeping with practices being increasingly adopted by R1 universities. Those practices included establishing clear criteria for a successful applicant and then independently rating candidates on each of those criteria. This practice is intended to prevent unstated, subjective—and potentially biased—criteria from eliminating objectively qualified candidates. In adopting these recommendations, the search committee agreed on the following four criteria on which candidates would be scored:

• • research and teaching that fit the specific needs identified in the advertisement

• • a high-quality publication and grant record (or the promise thereof)

• • a compelling political science research agenda

• • record (or intention) of commitment or contribution to improving departmental climate and diversity in political science

Members of the search committee were instructed to review each candidate’s file and to rate them on each of these four criteria using an 11-point scale from 0 (worst possible) to 10 (best possible), with 3 being the minimum “acceptable” rating. ²

In preliminary meetings of the search committee, Carlson noted the demonstrated tendency of women to apply for positions for which they meet all stated criteria, whereas men are more likely to apply broadly, including for positions for which they do not meet one or more of the qualifications stated in the advertisement (Ceci et al. 2014). She suggested that to combat this, “fit” be interpreted strictly and given particular weight in the rubric. ³ By emphasizing fit, the committee thus screened out what historically have been called “best-athlete” candidates: individuals with impressive credentials who nonetheless were poor matches with a specific position. This category of applicants is likely to be disproportionately male and may exclude a number of equally strong female “athletes” who did not apply for the job because they did not believe they were a good fit for the advertised position. ⁴ Noting these dynamics, the committee agreed to weight applicants’ scores on the four criteria, with the fit and publication criteria receiving greater weight (0.4 each) and the agenda and climate criteria weighted less (0.1 each). ⁵

Carlson also noted the tendency for letters of recommendation to display bias (conscious or implicit) against female job applicants, typically without the awareness of the applicant, ⁶ and she suggested that letters not be considered in the hiring process. However, recognizing that letters of recommendation also provide information that can benefit a candidate, the committee instead decided to read the letters—but only after a long list of eight to 10 candidates was generated based on other application materials. This meant that the materials considered by the search committee at the initial evaluation stage—including cover letters, curricula vitae, examples of published and unpublished research, and teaching materials—consisted entirely of materials over which the applicant had complete control. ⁷

The committee received and reviewed a total of 53 complete application files for the position. Of those applicants, 15 (28.3%) were female. This percentage was substantially lower than the percentage of women holding tenure-track positions in political science but higher than (for example) the share of female members of the Society for Political Methodology. ⁸ Committee members scored each of the 53 candidates on each of the four criteria; the committee chair then gathered and analyzed those assessments before the decision-making meeting. Figure 1 summarizes results of the committee’s candidate-rating process.⁹ The left-hand panel reports the (unweighted, standardized) mean ratings on each of the four search criteria for male (in blue) and female (in orange) candidates, along with t-statistics for differences of means between male and female applicants. As expected, female candidates generally scored higher in terms of fit to the position. Indeed, categorized dichotomously into “good” (≥6) and “poor (<6) fits, half of those deemed to be a good fit were women, whereas more than three quarters of those deemed to be a poor fit were men. Therefore, weighting fit more heavily increased the average scores of women in the applicant pool. Conversely, male candidates scored higher, on average, on the criterion related to publications and grants, a result consistent with recent research on publication bias in the discipline (Teele and Thelen 2017). Although there were no substantial differences between men and women regarding their research agenda, female candidates scored notably higher, on average, with respect to their contribution to diversity and departmental climate.

Figure 1 Standardized Score Means by Gender and Candidate-Criteria Biplot

The left panel plots the means of the unweighted standardized scores for the candidates, by gender. The right panel is the biplot of candidates and search criteria. In both plots, symbols for female candidates are in orange, male candidates are in blue. See the main text for details.

The right-hand panel of figure 1 is a biplot summarizing the data regarding the candidates’ ratings on the search criteria. The horizontal axis—representing the first extracted principal component—shows that the fit and research-agenda criteria load similarly to one another, with publications also loading strongly on that axis. The criterion related to climate loads strongly on the vertical axis and is mostly orthogonal to the other three criteria. In addition, although both male and female candidates are arrayed across a range of values on both axes, female candidates are notably absent from the lower range on the component mapping most strongly onto fit/agenda while also consistently loading strongly on the axis relating to climate. This pattern reinforces the general finding that placing greater emphasis on fit to the position works to mitigate at least some of the biases against female candidates and to remove any advantage for male “best athletes.” Upweighting climate, which we did not do, might have increased further the number of women on the short list. These results provide evidence contrary to Stacy et al. (2018), who found that searches using weighted rubrics are not more effective in hiring women.

Following this analysis, the committee chair summed scores from the (standardized, weighted) committee ratings; those summary scores then provided the basis for the committee’s initial winnowing of the candidate pool. Using the weighted-sum scores, two of the top four candidates and four of the top eight candidates were female. Across all candidates, the mean (standardized, weighted) sum score for male candidates was -0.05; for female candidates, the average was 0.12 (t=1.1). After review and discussion, the search committee created a long list of 10 candidates; it then adjourned to read the letters of recommendation for those 10 candidates. In subsequent discussion, committee members noted a number of instances in which bias appeared to shape letters and indicated that, in a few examples, if those letters had been read before the candidate’s own research, they might have led to committee members lowering their evaluation of the candidate’s promise. The committee then reconvened and agreed to recommend that the department interview five candidates, three of whom were female; the department agreed and proceeded with the interview process.

In retrospect, several aspects of this search process are notable. Primary among them was the outcome: in a subfield of political science known for the underrepresentation of women, the search processes led to a high degree of gender balance in the candidate long list and interview pool. Moreover, the department’s experience with adopting similar procedures in subsequent searches has led to similar results. A search for a nearly identical position the following academic year resulted in an interview pool that was two thirds female. In the ensuing years, use of these procedures for searches in American politics and international relations has resulted in the department interviewing and hiring significant numbers of female candidates, as well as a number of faculty from historically underrepresented groups. Notably, in every such search, the committee chair was a male faculty member who applied these protocols to overcome structural bias, thereby increasing the share of women being interviewed and hired. Although it is likely that these documented changes do not explain the successes, we believe that they contributed to them.

Conclusions and Implications

In the introduction to a recent symposium in The Washington Post, Dionne (2019) reviewed the lamentable state of women in academic political science. She also noted that “(T)he political science ecosystem may be on the verge of a big shift” because increasing awareness of gender disparities has begun (it is hoped) to lead to meaningful changes in the institutions and practices of our discipline. Our description of a series of innovations made to the hiring process—designed to counteract institutional and behavioral dynamics that work to decrease the representation of women in the discipline—is an example of these changes. Our review of that process suggests that a few relatively simple changes can contribute to greater gender balance in the field. Moreover, these changes are relatively simple and costless to adopt.

These are practices that can (and, we think, should) be implemented by anyone conducting a search in political science. However, given the historical and (resulting) demographic composition of most political science departments, it is likely that most department chairs/heads and most search committee chairs are male (Mitchell and Hesli 2013). Of course, this is particularly likely to be the case in fields that historically have been male dominated (Charlesworth and Banaji 2019). Within political science, this includes the subfields of quantitative methodology, formal theory, normative political theory, and international relations. Indeed, the Society for Political Methodology’s diversity report recently noted that “the majority of the positions of (formal and informal) power in our field are occupied by non-minority men” (Hidalgo et al. 2018, 12). To the extent that such changes are implemented by primarily male faculty and administrators, their effectiveness offers the potential for multiplier effects because higher numbers of female faculty in turn may be empowered to assume these roles.